ML Concepts & Workflow

Optimization & Optimizers

Difficulty:Advanced

Reading Time:35 min

Track:

Deep Learning

How neural networks actually learn: turning the gradients from backpropagation into parameter updates with SGD, momentum, RMSProp, and Adam — plus the learning-rate schedules and warmup that make training converge.

Prerequisites

Backpropagation Neural Networks & Deep Learning

Deep LearningModule 4 of 19

Optimization & Optimizers

21%

TL;DR

Backprop gives the gradient; the optimizer turns it into a parameter update — choosing step size, memory of past steps, and per-parameter scaling.
Mini-batch SGD is the default: a noisy gradient from a small batch buys many cheap, frequent, GPU-friendly updates.
Momentum accumulates a velocity of past gradients, accelerating consistent directions and damping oscillation across ravines.
Adaptive methods (RMSProp, Adam) rescale each parameter's step by its recent gradient magnitude, so flat directions are not starved — Adam = momentum + per-parameter scaling + bias correction.
The learning rate has a sweet spot: too high diverges/oscillates, too low crawls. Schedules (warmup then decay) move it over training.
Adam/AdamW is the safe default for Transformers; well-tuned SGD+momentum still wins much of computer vision — optimizer choice is task-dependent.

Learning Objectives

Explain the spectrum from full-batch gradient descent to stochastic and mini-batch SGD, and the noise/throughput trade-off between them
Describe how momentum accelerates descent along consistent directions and damps oscillation across ravines
Explain how adaptive methods (RMSProp, Adam) rescale each parameter's step using a running estimate of gradient magnitude
Choose a learning rate and schedule (decay, warmup) and recognize the symptoms of one that is too high or too low

Intuition

How to think conceptually about this topic

Imagine rolling a ball down a hilly landscape to find the lowest valley. The gradient tells the ball which way is downhill right where it sits. Plain gradient descent is a ball with no inertia: at every instant it moves straight downhill by a fixed fraction of the slope. In a long, narrow valley it wastes energy bouncing from wall to wall while barely creeping along the valley floor.

Momentum gives the ball mass. It builds up speed rolling consistently downhill and its side-to-side bounces cancel, so it shoots along the valley floor instead of pinballing. Adaptive methods are cleverer still: they notice that some directions are steep and others shallow and give each its own step size — big strides where the ground is flat, careful steps where it is steep — so no direction is neglected.

And the learning rate is how far the ball is allowed to move each tick. Too large and it leaps over the valley and out the other side (diverging); too small and it inches along forever. Schedules change this allowance over time: start cautious (warmup), then take big confident strides, then shorten the steps to settle gently at the bottom.

Interactive Diagram

Test the intuition above by changing the model parameters

In Depth

Detailed explanations, contexts, and details

Backpropagation hands you the gradient — the direction of steepest increase of the loss. An optimizer decides what to do with it: how big a step to take, whether to remember past steps, and whether to treat every parameter the same. Getting this right is often the difference between a model that trains and one that does not.

The batch spectrum

Full-batch gradient descent uses the entire dataset for one exact gradient per step — stable but slow and memory-hungry.
Stochastic gradient descent (SGD) uses a single example: very noisy, but cheap and frequent.
Mini-batch SGD (the default) uses a small batch — a sweet spot that balances gradient quality, update frequency, and GPU efficiency.

Beyond vanilla SGD

Plain SGD struggles on realistic loss surfaces that are steep in some directions and flat in others. Two ideas fix this:

Momentum keeps a running 'velocity' of past gradients, so progress accelerates in consistent directions and oscillations across a ravine cancel out.
Adaptive methods (RMSProp, Adam) give each parameter its own step size based on the recent magnitude of its gradients, so small-gradient directions are not starved and large-gradient directions do not explode.

Adam combines momentum with per-parameter scaling and is the default for Transformers and many modern models; well-tuned SGD + momentum still rules much of computer vision. On top of the optimizer sits the learning-rate schedule — warmup to stabilize the start, then decay to settle into a minimum — which often matters as much as the optimizer choice itself.

How It Compares

SGD vs SGD+Momentum vs Adam

Dimension	SGD	SGD + Momentum	Adam
Per-parameter adaptivity	None — one global rate	None — one global rate	Yes — per-coordinate scaling
Uses gradient history	No	Yes — velocity	Yes — 1st & 2nd moments
Behavior in ravines	Zig-zags / crawls	Accelerates along the valley	Handles uneven curvature well
Tuning sensitivity	High — learning rate critical	Moderate	Lower — forgiving defaults
Extra optimizer memory	None	1× params (velocity)	2× params (m and v)
Typical home	Simple/convex problems	Computer vision (tuned)	Transformers, sparse gradients

TakeawayReach for Adam/AdamW as a robust default (especially Transformers); invest in tuned SGD+momentum when you want the best generalization on vision tasks. Plain SGD is mostly a teaching baseline.

When to Use It

Reach for this when

Training any neural network — the optimizer and learning-rate schedule are core hyperparameters, not afterthoughts.
Default to Adam/AdamW for Transformers, NLP, and problems with sparse or wildly-scaled gradients where its robustness shines.
Choose well-tuned SGD + momentum when you can afford to tune and want top generalization, especially in computer vision.
Add warmup then decay for large-batch training and Transformers, where a cold start at full learning rate is unstable.

Avoid it when

Don't crank the learning rate to 'train faster' — past the sweet spot it oscillates or diverges; use a schedule instead.
Don't assume Adam is strictly best; on some vision benchmarks it generalizes worse than tuned SGD.
Don't ignore optimizer memory on huge models — Adam's extra per-parameter state can be the difference between fitting and OOM.
Don't expect any optimizer to rescue a badly-conditioned, unnormalized model — fix inputs/normalization first.

Rules of thumb

Start with AdamW (lr ≈ 3e-4) for Transformers, or SGD+momentum (lr ≈ 0.1 with decay) for CNNs, then tune.
If the loss explodes or NaNs, your learning rate is too high — halve it, add warmup, or clip gradients.
Use a schedule (cosine or step decay) and warmup; the schedule often matters as much as the optimizer.

Implementation

Reference code implementation

Python

model_fitting.py

1import torch
2
3# A toy ill-conditioned quadratic: steep in y, flat in x.
4def loss(p):
5    return 0.5 * (0.3 * p[0] ** 2 + 3.0 * p[1] ** 2)
6
7def run(make_opt, steps=40):
8    p = torch.tensor([-2.4, 0.9], requires_grad=True)
9    opt = make_opt([p])
10    for _ in range(steps):
11        opt.zero_grad()
12        loss(p).backward()        # gradient from backprop
13        opt.step()                # optimizer turns it into an update
14    return loss(p).item()
15
16lr = 0.1
17print("SGD:      ", round(run(lambda P: torch.optim.SGD(P, lr=lr)), 4))
18print("Momentum: ", round(run(lambda P: torch.optim.SGD(P, lr=lr, momentum=0.9)), 4))
19print("RMSProp:  ", round(run(lambda P: torch.optim.RMSprop(P, lr=lr)), 4))
20print("Adam:     ", round(run(lambda P: torch.optim.Adam(P, lr=lr)), 4))
21# Momentum and Adam reach a far lower loss in the same number of steps because
22# they make progress along the flat x-direction that plain SGD crawls through.

Strengths & Advantages

Mini-batch SGD turns an intractable full-dataset optimization into many cheap, frequent, hardware-friendly updates.
Momentum and adaptive methods (Adam) handle ill-conditioned loss surfaces that stall plain SGD, often training far faster with less tuning.
Learning-rate schedules (warmup + decay) give a reliable recipe for stable starts and well-settled minima across many architectures.

Limitations & Drawbacks

Performance is highly sensitive to the learning rate and schedule, which usually need tuning and can fail loudly (divergence) or quietly (slow crawl).
Adaptive optimizers store extra per-parameter state (Adam doubles the optimizer memory) and can generalize worse than tuned SGD on some tasks.
Stochastic noise means runs are not exactly reproducible and convergence is to a local/flat region, not a guaranteed global minimum.

Real-World Case Studies

Adam becomes the default optimizer for training deep networks

Deep learning practice

Scenario

By 2014, practitioners faced a confusing menu of optimizers (SGD, momentum, AdaGrad, RMSProp), each needing careful, problem-specific learning-rate tuning. Training deep models with sparse or very unevenly-scaled gradients — common in NLP and recommendation — was especially fiddly.

Approach

Kingma and Ba combined momentum (a first-moment estimate) with RMSProp-style per-parameter scaling (a second-moment estimate) and added bias correction for the zero-initialized averages, yielding Adam — an optimizer with well-behaved default hyperparameters ( $\beta_1=0.9$ , $\beta_2=0.999$ , $\epsilon=10^{-8}$ ) that adapts each parameter's effective step automatically.

Outcome

Adam matched or beat the best-tuned alternatives across MNIST, CIFAR-10, and logistic-regression benchmarks while needing far less learning-rate tuning, and it has since become the most widely used optimizer in deep learning — the standard choice (with its AdamW variant) for training virtually every large Transformer. The Adam paper is among the most-cited works in modern machine learning.

Source: Adam: A Method for Stochastic Optimization — Kingma, D. P. and Ba, J.

Common Misconceptions

MisconceptionStochastic gradient descent computes the true gradient of the loss each step.

CorrectionSGD uses a noisy estimate of the gradient from one example or a mini-batch, not the full-dataset gradient. That noise is a feature: it adds many cheap updates per epoch and can help escape sharp minima and saddle points.

MisconceptionAdam is always the best optimizer, so you never need to tune or consider SGD.

CorrectionAdam converges fast and is forgiving, but well-tuned SGD with momentum often generalizes better on vision tasks and is the standard there. The right optimizer is task-dependent; Adam (or AdamW) is the safe default for Transformers and sparse-gradient problems, not a universal winner.

MisconceptionA higher learning rate always means faster training.

CorrectionPast a point, a larger learning rate overshoots the minimum and the loss oscillates or diverges; too small and training crawls or stalls. There is a sweet spot, and schedules (warmup then decay) move it over the course of training.

References & Further Reading

Adam: A Method for Stochastic Optimizationpaper
By Kingma, D. P. and Ba, J.
View Resource →
Deep Learning (Chapter 8: Optimization for Training Deep Models)textbook
By Goodfellow, I., Bengio, Y. and Courville, A.
View Resource →
Decoupled Weight Decay Regularization (AdamW)paper
By Loshchilov, I. and Hutter, F.
View Resource →

Optimization & Optimizers

Prerequisites

TL;DR

Learning Objectives

Intuition

Interactive Diagram

The Mathematics

In Depth

The batch spectrum

Beyond vanilla SGD

How It Compares

SGD vs SGD+Momentum vs Adam

When to Use It

Reach for this when

Avoid it when

Rules of thumb

Implementation

Strengths & Advantages

Limitations & Drawbacks

Real-World Case Studies

Adam becomes the default optimizer for training deep networks

Common Misconceptions

Self-Check Quiz

References & Further Reading

Related Topics

Backpropagation

Neural Networks & Deep Learning

L1 & L2 Regularization