DNN optimizers

Training a network is minimising a non-convex function of hundreds of millions of variables using noisy gradient estimates. Nobody computes a Hessian at that scale, so every practical optimizer is a cheap approximation to second-order information — built from running averages of the gradient and its square. This page derives each one in the order the ideas arrived, explains what problem each was invented to solve, and makes the case for why AdamW is the right default and when it is worth leaving.

Symbols used on this page are defined in the Nomenclature → deep learning & training.

Learning-rate schedules, batch-size coupling and the surrounding training configuration are on the hyperparameter tuning page; this one is specifically about the update rule.

1 · The common template

1.1 One form, several instances

Almost every optimizer in use is a special case of this: maintain a smoothed direction, maintain a per-coordinate scale, and step.

$$ \mathbf m_t = \beta_1 \mathbf m_{t-1} + (1-\beta_1)\,\mathbf g_t \qquad\text{(first moment — where to go)} $$ $$ \mathbf v_t = \beta_2 \mathbf v_{t-1} + (1-\beta_2)\,\mathbf g_t^{\odot 2} \qquad\text{(second moment — how far)} $$ $$ \boldsymbol\theta_{t+1} = \boldsymbol\theta_t - \eta\,\frac{\hat{\mathbf m}_t}{\sqrt{\hat{\mathbf v}_t} + \epsilon} - \eta\,\lambda\,\boldsymbol\theta_t $$

Reading the table this way makes the lineage obvious: the first moment is about averaging away gradient noise, the second is about normalising for scale, and the decay column is a correctness detail that took until 2017 to get right.

1.2 Why not use the Hessian

Newton's method, $\boldsymbol\theta \leftarrow \boldsymbol\theta - \mathbf H^{-1}\mathbf g$, converges quadratically and is immune to bad conditioning. It is also unusable here: $\mathbf H$ has $N^2$ entries, so a 1-billion-parameter model would need $10^{18}$ numbers, and the loss is non-convex so $\mathbf H$ is not even positive definite. Every optimizer below is a diagonal, cheaply-estimated stand-in — the $\sqrt{\mathbf v_t}$ term is a crude approximation to the diagonal of the Fisher information or of $\mathbf H$, which is exactly why it helps with ill-conditioning.

2 · SGD and momentum

2.1 Plain SGD

$$ \boldsymbol\theta_{t+1} = \boldsymbol\theta_t - \eta\,\mathbf g_t $$

One step size for every parameter in the network. That is the entire problem: $\eta$ must be small enough for the sharpest direction of curvature (or training diverges, see hyperparameters §2.1) and is therefore far too small for the flat directions. In a loss with condition number $\kappa = L/\mu$, convergence takes $O(\kappa \log(1/\varepsilon))$ steps — and for real networks $\kappa$ is enormous.

2.2 Momentum

Momentum accumulates a velocity instead of stepping along the instantaneous gradient:

$$ \mathbf v_{t+1} = \gamma\,\mathbf v_t + \mathbf g_t, \qquad \boldsymbol\theta_{t+1} = \boldsymbol\theta_t - \eta\,\mathbf v_{t+1}, \qquad \gamma = 0.9 . $$

Two effects, and both matter. Oscillating components (across a narrow valley) alternate sign and cancel in the average, while consistent components (along the valley) accumulate — so the effective step along a consistent direction is amplified by $1/(1-\gamma)$, a factor of 10 at $\gamma = 0.9$. And a point of near-zero gradient — a saddle or a plateau — no longer stalls the optimizer, because the velocity carries it through. Momentum improves the convergence rate from $O(\kappa)$ to $O(\sqrt\kappa)$, which is a large asymptotic gain.

PyTorch's momentum is not the classical formula. It uses $v \leftarrow \gamma v + g$ then $\theta \leftarrow \theta - \eta v$, whereas many papers write $v \leftarrow \gamma v + \eta g$ then $\theta \leftarrow \theta - v$. The difference matters when $\eta$ changes during training: in PyTorch's form a schedule takes effect immediately, in the other form the velocity still carries the old learning rate. This is a real source of confusion when porting a recipe between frameworks.

Nesterov momentum evaluates the gradient at the look-ahead point $\boldsymbol\theta_t - \eta\gamma\mathbf v_t$ rather than at $\boldsymbol\theta_t$, which lets it start decelerating before overshooting. It is a small, nearly free improvement — turn it on.

2.3 The sim

Five optimizers, implemented exactly as PyTorch implements them, on the same surface from the same start. Things worth doing: raise the condition number and watch SGD's zig-zag grow while Adam barely changes; switch to the saddle surface and watch plain SGD stall where momentum carries through; switch to the ravine and see why adaptivity matters on curved valleys.

3 · Adaptive methods

3.1 Adagrad — and why it dies

$$ \mathbf s_t = \mathbf s_{t-1} + \mathbf g_t^{\odot 2}, \qquad \boldsymbol\theta_{t+1} = \boldsymbol\theta_t - \frac{\eta}{\sqrt{\mathbf s_t} + \epsilon}\odot \mathbf g_t $$

The first per-parameter learning rate: each coordinate is divided by the accumulated magnitude of its own gradients, so rarely-updated parameters take large steps and frequently-updated ones take small steps. This was a genuine advance for sparse problems — and it has a fatal flaw. The sum $\mathbf s_t$ is monotonically increasing, so the effective learning rate decays to zero whether or not the model has converged. Adagrad reliably stops learning partway through a long training run.

3.2 RMSProp — the fix

$$ \mathbf s_t = \beta\,\mathbf s_{t-1} + (1-\beta)\,\mathbf g_t^{\odot 2} $$

Replace the cumulative sum with an exponential moving average. Now $\mathbf s_t$ tracks the recent gradient magnitude and can decrease as well as increase, so the effective learning rate does not collapse. This one change — a sum becoming an EMA — is the single most important step from Adagrad to everything modern.

3.3 Adam

Adam is RMSProp plus momentum plus a bias correction:

$$ \mathbf m_t = \beta_1\mathbf m_{t-1} + (1-\beta_1)\mathbf g_t, \qquad \mathbf v_t = \beta_2\mathbf v_{t-1} + (1-\beta_2)\mathbf g_t^{\odot 2} $$ $$ \hat{\mathbf m}_t = \frac{\mathbf m_t}{1-\beta_1^t}, \qquad \hat{\mathbf v}_t = \frac{\mathbf v_t}{1-\beta_2^t} $$ $$ \boldsymbol\theta_{t+1} = \boldsymbol\theta_t - \eta\,\frac{\hat{\mathbf m}_t}{\sqrt{\hat{\mathbf v}_t}+\epsilon} $$

The bias correction is not cosmetic. Both moments are initialised at zero, so early estimates are biased toward zero by a factor $(1-\beta^t)$ — at $t=1$ with $\beta_2 = 0.999$ the raw $\mathbf v_1$ is a thousand times too small, which would make the first step enormous. Dividing by $1-\beta^t$ corrects exactly that, and the factor decays to 1 within a few hundred steps.

3.4 Why Adam is the right default

The update has a property worth stating explicitly. Because the numerator and denominator have the same units, the step is approximately

$$ \left|\Delta\theta_i\right| \approx \eta \cdot \frac{\left|\hat m_i\right|}{\sqrt{\hat v_i}} \;\lesssim\; \eta , $$

i.e. Adam's step is scale-invariant and bounded by roughly $\eta$ regardless of the gradient magnitude. Multiply the whole loss by 1000 and Adam takes the same steps; SGD would diverge. This is why one global learning rate works across layers whose gradients differ by orders of magnitude, and it is the practical reason Adam "just works" where SGD needs per-layer tuning.

It also explains why Adam's learning rate is ~100× smaller than SGD's: $\eta$ in Adam is close to the actual per-step displacement, whereas in SGD it multiplies a gradient that may be tiny. The two numbers mean different things, and confusing them is the most common optimizer mistake (see hyperparameters §6.2).

3.5 AdamW — decoupled weight decay

Classical L2 regularisation adds $\lambda\boldsymbol\theta$ to the gradient. Inside Adam that term then gets divided by $\sqrt{\hat{\mathbf v}_t}$ along with everything else — so a parameter with a large gradient history receives less regularisation. That is precisely backwards. AdamW applies the decay directly to the weights, outside the adaptive rescaling:

$$ \boldsymbol\theta_{t+1} = \boldsymbol\theta_t - \eta\,\frac{\hat{\mathbf m}_t}{\sqrt{\hat{\mathbf v}_t}+\epsilon} \;-\; \eta\,\lambda\,\boldsymbol\theta_t . $$

The consequence is that decay strength becomes independent of gradient scale, which makes $\lambda$ transferable across layers and across problems. This is why AdamW's weight decay values (0.01–0.3) are so much larger than Adam's and still work. The simulation below runs both on two parameters with very different gradient scales:

4 · Convergence and stability

4.1 What is actually guaranteed

The Reddi result is worth understanding rather than just citing. The original Adam convergence proof was flawed, and the counterexample exploits exactly the EMA that fixed Adagrad: because $\mathbf v_t$ can decrease, a large but rare informative gradient can be forgotten, letting the optimizer drift away from the optimum. AMSGrad fixes this by tracking the running maximum, $\hat{\mathbf v}_t = \max(\hat{\mathbf v}_{t-1}, \mathbf v_t)$, which restores the proof.

4.2 Stability in practice

4.3 The generalisation question

A persistent empirical observation: on image classification, a well-tuned SGD with momentum often generalises slightly better than Adam at the same training loss. The usual explanation is that SGD's uniform, noisier steps bias it toward flatter minima, which are argued to generalise better. Much of the original gap, however, turned out to be the weight-decay bug — AdamW closed a large part of it.

The practical reading: for CNN vision models with established recipes, SGD+momentum is still a defensible choice and frequently what published state-of-the-art numbers used. For transformers it is not a real option — attention models trained with SGD underperform badly, and essentially all of them use AdamW.

5 · What LLM and large-model training actually uses

5.1 The memory cost of optimizer state

Adam stores two extra tensors per parameter, which dominates the memory budget at scale:

So training a 7 B model in fp32 needs 28 GB for weights, 28 GB for gradients and 56 GB for Adam state — the optimizer is the single largest consumer. This is the entire motivation for Adafactor (which stores a rank-1 factorisation of the second moment instead of the full matrix), 8-bit Adam, and the ZeRO/FSDP family that shards optimizer state across devices.

6 · Newer optimizers

7 · PyTorch specifics

7.1 Parameter groups

The most important practical detail: normalisation parameters and biases should not be weight-decayed. Decaying a LayerNorm scale toward zero actively damages the network. This requires two parameter groups:

7.2 Options worth knowing

One ordering trap worth repeating from the hyperparameters page §9.1: with AMP, call scaler.unscale_(opt) before clipping. Clipping a scaled gradient clips against the wrong threshold and silently does nothing useful.

8 · Choosing

9 · Practical checklist

• Use AdamW, not Adam with weight decay. They are different algorithms (§3.5).
• Exclude norms and biases from weight decay via parameter groups (§7.1).
• Always use warmup with adaptive optimizers. The early second-moment estimate is unreliable.
• Set $\beta_2 = 0.95$ for large models; 0.999 reacts too slowly and invites loss spikes.
• Raise $\epsilon$ to $10^{-6}$ under fp16, or use bf16 and leave it alone.
• Clip by global norm, and log the pre-clip norm as a health metric.
• Never port a learning rate between optimizers. The LR belongs to the (optimizer, batch size) pair.
• Budget optimizer state explicitly — 2× the model for Adam (§5.1).
• Tune the AdamW baseline before trying anything exotic (§6).
• Checkpoint the optimizer state, not just the weights, or a resumed run restarts its moment estimates from zero and takes a damaging first step.

10 · Further reading

• Kingma & Ba (2015), "Adam: A Method for Stochastic Optimization," ICLR.
• Loshchilov & Hutter (2019), "Decoupled Weight Decay Regularization," ICLR. The AdamW paper, and the clearest statement of why coupled decay is wrong.
• Reddi, Kale & Kumar (2018), "On the Convergence of Adam and Beyond," ICLR. The counterexample and AMSGrad.
• Ruder (2016), "An overview of gradient descent optimization algorithms." Still the best single tour of the lineage in §2–3.
• Goh (2017), "Why Momentum Really Works," Distill. The clearest visual explanation of the $O(\sqrt\kappa)$ acceleration.
• Duchi, Hazan & Singer (2011) (Adagrad); Tieleman & Hinton (2012) (RMSProp).
• Schmidt, Schneider & Hennig (2021), "Descending through a Crowded Valley — Benchmarking Deep Learning Optimizers," ICML. The sceptical empirical study behind §6.
• Chen et al. (2023), "Symbolic Discovery of Optimization Algorithms" (Lion).
• Shazeer & Stern (2018), "Adafactor: Adaptive Learning Rates with Sublinear Memory Cost."
• Dettmers et al. (2022), "8-bit Optimizers via Block-wise Quantization."
• Foret et al. (2021), "Sharpness-Aware Minimization for Efficiently Improving Generalization."
• Zhang et al. (2020), "Why Gradient Clipping Accelerates Training."
• PyTorch torch.optim documentation — the per-optimizer algorithm blocks state the exact update each one implements, including the momentum convention noted in §2.2.

Related pages: Hyperparameter tuning & training · Bundle adjustment (Levenberg–Marquardt, the second-order view) · Deep Learning