Why Adam works in optimizing language models: heavy-tailed data imbalance

Adam and related methods are the default optimizers in training language models because gradient descent is so slow it is thought it simply does not work. Many methods have been proposed to improve on Adam, but we have a poor understanding of why it works better in the first place. This talk will be about understanding what makes language models hard for gradient descent, and how Adam fixes it. We give evidence that the problem is due to imbalance in the data: in language, some words are very frequent but the majority are rare, following Zipf’s law; the ?th most frequent word has frequency proportional to 1/?. This heavy-tailed imbalance leads to poor performance with gradient descent, while Adam and other sign-like methods do not suffer from the same slowdown. We show empirical evidence on language models, and prove a separation in the performance of gradient and a simplified form of Adam, sign descent, on a linear bigram model as we the vocabulary size grows.