Backpropagation

Difficulty:Advanced

Reading Time:40 min

Track:

Deep Learning

The foundational algorithm for training neural networks, using the chain rule of calculus to compute loss gradients with respect to model weights.

Prerequisites

Neural Networks & Deep Learning

Deep LearningModule 3 of 19

Backpropagation

16%

TL;DR

Backpropagation computes $\frac{\partial L}{\partial W}$ for every weight in a network by applying the chain rule backward from the loss to the inputs.
It reuses intermediate values from the forward pass and propagates a single error signal $\delta$ layer by layer, so the cost is roughly the same as one extra forward pass — not one pass per weight.
The key recursive step is $\delta^{(l)} = (W^{(l+1)T} \delta^{(l+1)}) \odot \sigma^{\prime}(z^{(l)})$ : each layer’s error is the next layer’s error pulled back through the weights and gated by the local activation derivative.
Sigmoid/tanh derivatives are bounded well below 1, so multiplying many of them together across deep networks causes the vanishing gradient problem — a major motivation for ReLU and residual connections.
Backprop is exact and efficient; it is not the same thing as the optimizer (SGD/Adam) that actually uses the gradients to update weights.
Gradient checking with finite differences is the standard way to verify a hand-written backward pass is correct before trusting it.

Learning Objectives

Construct and interpret a computational graph for arbitrary mathematical expressions.
Formulate and apply the chain rule of calculus in reverse mode to compute local derivatives.
Compute vector-Jacobian products to propagate gradients through multi-dimensional matrix operations.
Debug common gradient issues including vanishing, exploding, and un-zeroed gradients.
Implement gradient checking using finite differences to verify analytical derivative correctness.

Intuition

How to think conceptually about this topic

Think of backpropagation as a game of telephone in reverse. In the forward pass, information flows from the inputs, through layers, to produce a final prediction. We measure how wrong the prediction was using a loss function. In the backward pass, we want to trace the error back to its sources. If a node contributed heavily to the error, it gets a large gradient. We do this step-by-step, starting from the final loss and working backward. Each node computes its local contribution to the rate of change and multiplies it by the incoming gradient from the step ahead. This recursive multiplying is the chain rule in action.

Interactive Diagram

Test the intuition above by changing the model parameters

In Depth

Detailed explanations, contexts, and details

Backpropagation is reverse-mode automatic differentiation applied to compute the gradient of a loss function with respect to the weights of a neural network. It enables gradient descent and other gradient-based optimization algorithms to update weights and train deep representations.

How It Compares

Backpropagation vs Numerical Gradient Checking vs Forward-mode Autodiff

Dimension	Backpropagation	Numerical Gradient Checking	Forward-mode Autodiff
Computational cost (per full gradient)	$O(1)$ extra forward-pass-equivalent — one backward pass computes gradients for all parameters at once.	$O(n)$ forward passes for $n$ parameters (or $O(2n)$ with central differences) — prohibitively slow for networks with millions of weights.	$O(n)$ forward passes for $n$ input directions — efficient only when there are few inputs relative to outputs.
Accuracy	Exact (up to floating-point rounding) — computes the true analytical derivative.	Approximate — has $O(\epsilon^2)$ truncation error from the finite-difference formula, plus floating-point cancellation error for small $\epsilon$ .	Exact (up to floating-point rounding), same as backprop.
Memory usage	Must cache all intermediate activations from the forward pass for reuse in the backward pass.	No caching needed — perturb and re-run forward each time.	Propagates derivative information alongside values in a single forward sweep; no separate backward pass needed.
When it is used	The default for training virtually all deep neural networks (few outputs — one scalar loss — and many parameters).	A debugging tool only: used sparingly to verify a hand-written backward pass is correct, never for actual training.	Preferred when there are far more outputs than inputs, e.g. sensitivity analysis with few parameters and many outputs.

TakeawayUse backpropagation (reverse-mode autodiff) for training — it scales with the number of outputs (typically one loss value) regardless of how many millions of parameters there are. Use numerical gradient checking only as a one-off correctness check on a small subset of parameters. Forward-mode autodiff is the right tool only in the opposite regime: few inputs, many outputs.

When to Use It

Reach for this when

You need to train any differentiable model — neural networks, but also any parameterized computational graph — via gradient-based optimization.
The network has far more parameters than outputs (the common case: millions of weights, one scalar loss), where reverse-mode is asymptotically cheapest.
You are implementing or debugging a custom layer/operator and need to verify your hand-derived gradient formula is correct.

Avoid it when

You only need a rough sensitivity estimate for a tiny number of parameters and want to avoid implementing a backward pass at all — finite differences may be simpler.
The function has many more outputs than inputs (e.g. a few parameters mapping to a huge output vector) — forward-mode autodiff is cheaper there.
The computational graph contains non-differentiable operations (hard thresholds, discrete sampling without a reparameterization trick) — plain backprop does not apply without modification (e.g. straight-through estimators).

Rules of thumb

Always validate a new backward-pass implementation with numerical gradient checking on a small example before trusting it at scale.
Watch activation statistics during training — saturating sigmoids/tanh (outputs near 0 or 1) are an early warning sign of vanishing gradients.
Remember backprop only computes gradients; an optimizer (SGD, Adam, etc.) is still required to turn gradients into weight updates.

Implementation

Reference code implementation

TypeScript

example.ts

1/**
2 * Simple Computational Graph Engine for Scalar Operations (Micrograd-like)
3 */
4export class Value {
5  public grad: number = 0;
6  private _backward: () => void = () => {};
7
8  constructor(
9    public data: number,
10    public prev: Value[] = [],
11    public op: string = ""
12  ) {}
13
14  add(other: Value): Value {
15    const out = new Value(this.data + other.data, [this, other], "+");
16    out._backward = () => {
17      this.grad += out.grad;
18      other.grad += out.grad;
19    };
20    return out;
21  }
22
23  mul(other: Value): Value {
24    const out = new Value(this.data * other.data, [this, other], "*");
25    out._backward = () => {
26      this.grad += other.data * out.grad;
27      other.grad += this.data * out.grad;
28    };
29    return out;
30  }
31
32  backward() {
33    const topo: Value[] = [];
34    const visited = new Set<Value>();
35
36    function buildTopo(v: Value) {
37      if (!visited.has(v)) {
38        visited.add(v);
39        for (const child of v.prev) {
40          buildTopo(child);
41        }
42        topo.push(v);
43      }
44    }
45
46    buildTopo(this);
47    this.grad = 1;
48
49    for (let i = topo.length - 1; i >= 0; i--) {
50      topo[i]._backward();
51    }
52  }
53}

Strengths & Advantages

Computational complexity scales linearly with the number of operations in the graph.
Enables end-to-end training of deeply nested neural network architectures.
Highly parallelizable and efficiently mapped to modern GPU hardware.

Limitations & Drawbacks

Can suffer from numerical instability in very deep or recurrent networks.
Requires storing intermediate activations in memory for the backward pass.
Hard to debug when intermediate gradients overflow or underflow.

Real-World Case Studies

The 1986 Rumelhart, Hinton & Williams paper that popularized backpropagation

Neural network training / connectionist AI

Scenario

In the mid-1980s, multi-layer "connectionist" networks were known to be theoretically more powerful than single-layer perceptrons, but there was no efficient, general algorithm to train their hidden-layer weights — researchers lacked a practical way to assign credit/blame for errors to internal units several layers removed from the output.

Approach

Rumelhart, Hinton, and Williams described training multi-layer networks by propagating the output error backward through the layers using the chain rule, computing a local error term $\delta$ at each layer and using it to derive weight updates — efficiently reusing computation from the forward pass instead of perturbing each weight individually.

Outcome

The paper ("Learning representations by back-propagating errors", Nature, 1986) showed hidden units could automatically learn useful internal representations (e.g. solving the XOR problem and learning symmetry-detection tasks) without those representations being hand-designed. It became the foundational training algorithm behind essentially all subsequent deep learning, turning what was previously an $O(n)$ -forward-pass-per-weight numerical procedure into a single backward pass costing about the same as one forward pass — a complexity reduction that made training networks with thousands and later billions of weights practical.

Source: Learning representations by back-propagating errors — Rumelhart, D. E., Hinton, G. E., and Williams, R. J.

Common Misconceptions

MisconceptionBackpropagation is a unique optimization algorithm that updates neural network weights.

CorrectionBackpropagation is only the method used to compute the gradients of the loss function with respect to weights. The actual updating of the weights is performed by separate optimization algorithms like Stochastic Gradient Descent or Adam, using those computed gradients.

MisconceptionGradients must always be calculated analytically and implemented by hand.

CorrectionModern deep learning libraries use automatic differentiation (autograd) to build the computational graph dynamically and compute gradients automatically, though manual implementation remains crucial for custom operators and verification.

References & Further Reading

Calculus on Computational Graphs: Backpropagationtutorial
By Christopher Olah
View Resource →
Deep Learning (Chapter 6.5)textbook
By Ian Goodfellow, Yoshua Bengio, and Aaron Courville
View Resource →

Backpropagation

Prerequisites

TL;DR

Learning Objectives

Intuition

Interactive Diagram

The Mathematics

In Depth

How It Compares

Backpropagation vs Numerical Gradient Checking vs Forward-mode Autodiff

When to Use It

Reach for this when

Avoid it when

Rules of thumb

Implementation

Strengths & Advantages

Limitations & Drawbacks

Real-World Case Studies

The 1986 Rumelhart, Hinton & Williams paper that popularized backpropagation

Common Misconceptions

Self-Check Quiz

References & Further Reading

Related Topics

Neural Networks & Deep Learning

Transformers