Computer Vision

Diffusion Models

Difficulty:Expert

Reading Time:35 min

Track:

Computer Vision

Generate images by learning to reverse a gradual noising process: a fixed forward chain destroys an image into pure Gaussian noise, and a trained network denoises step by step to turn fresh noise back into a sample.

Prerequisites

Generative Adversarial Networks Neural Networks & Deep Learning

Computer VisionModule 8 of 8

Diffusion Models

100%

TL;DR

A diffusion model learns to generate by reversing a gradual noising process — destroy an image into noise, then learn to undo it one small step at a time.
The forward process is fixed (no learning) and has a closed form: $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$ , so you can sample any noise level in one shot.
The reverse process is a network trained with a simple MSE loss to predict the added noise $\epsilon$ (equivalently, the score); the clean estimate is derived from it.
Generation starts from pure Gaussian noise and iteratively denoises — there is no hidden real image being recovered.
DDPM is high-quality but slow (hundreds of steps); DDIM makes sampling deterministic in 10–50 steps; latent diffusion runs the whole thing in a compressed autoencoder space to make text-to-image practical.
Diffusion replaced GANs as the default for high-fidelity image generation thanks to stable training (no adversarial game) and strong mode coverage.

Learning Objectives

Describe the fixed forward (noising) process and the learned reverse (denoising) process that define a diffusion model
Derive and use the closed-form forward sampler $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$ and explain the role of the noise schedule
Explain why the network is trained to predict the added noise (or score) with a simple mean-squared-error objective
Contrast DDPM and DDIM sampling and explain how latent diffusion makes text-to-image generation tractable

Intuition

How to think conceptually about this topic

Imagine a clear photograph sitting in a snow globe. Shake the globe a tiny bit and a faint dusting of snow settles over the image; shake again and a little more; after enough shakes the picture is buried under uniform white noise. That is the forward process — and crucially, each shake is small and random in a known, simple way.

Now run the film backward. If you had a machine that, shown any snowed-over frame, could guess "this much snow was just added here," you could brush off that snow one layer at a time and watch the photograph reappear. The neural network is that machine. The astonishing part: once it has learned to brush snow off real photos, you can hand it a globe that is nothing but snow — fresh noise that never had a picture in it — and by removing snow step after step, a brand-new, never-before-seen photograph emerges. The model is not recovering a hidden image; it is hallucinating a plausible one consistent with everything it learned about how images get buried.

Interactive Diagram

Test the intuition above by changing the model parameters

In Depth

Detailed explanations, contexts, and details

A diffusion model learns to create data by reversing a process of gradual destruction. The idea is disarmingly simple: if you can take any image and slowly turn it into static, then learning to undo each tiny step lets you start from static and grow an image.

There are two processes:

Forward (diffusion) process — fixed, no learning. Over $T$ steps it adds a small amount of Gaussian noise at a time until, by step $T$ , the image is indistinguishable from pure noise. Because each step is a simple Gaussian, the whole thing has a convenient closed form: you can jump to the noise level of any step $t$ in one shot.
Reverse (denoising) process — learned. A neural network looks at a noisy image $x_t$ and the step index $t$ and predicts the noise that was added. Subtracting a calibrated amount of that predicted noise takes you from $x_t$ toward $x_{t-1}$ . Chain these reverse steps from pure noise at $t=T$ down to $t=0$ and a clean sample appears.

Why diffusion took over generative vision

Diffusion models displaced GANs as the default for high-fidelity image generation because they are stable to train (a plain regression loss — no adversarial min–max, no mode collapse) and offer excellent sample quality and coverage. Their one weakness — slow, many-step sampling — is largely tamed by faster samplers (DDIM) and by running the process in a compact latent space (latent diffusion), which is what makes modern text-to-image systems like Stable Diffusion practical.

How It Compares

Diffusion vs GANs vs VAEs

Dimension	Diffusion	GAN	VAE
Training objective	Denoising regression (stable MSE)	Adversarial min–max (can be unstable)	Evidence lower bound (reconstruction + KL)
Sample quality	State of the art	High, but prone to artifacts	Lower — samples often blurry
Mode coverage	Broad	Can collapse to a few modes	Broad
Sampling speed	Slow (many steps) unless accelerated	Fast (single pass)	Fast (single pass)
Latent structure	No compact latent by default (latent diffusion adds one)	Low-dimensional latent	Explicit, regularized latent

TakeawayDiffusion buys top quality and stable, mode-covering training at the cost of slow sampling — and faster samplers plus latent-space diffusion close most of that gap, which is why it is the default for modern high-fidelity image generation.

When to Use It

Reach for this when

You need high-fidelity, diverse image generation and can afford (or accelerate) multi-step sampling.
Training stability and mode coverage matter — diffusion avoids the adversarial instability and mode collapse of GANs.
You want flexible conditioning: text-to-image, inpainting, super-resolution, or image-to-image, all of which fit naturally via guidance and cross-attention.
You can run in a latent space (via a pretrained autoencoder) to keep high-resolution generation affordable.

Avoid it when

You have a hard real-time, single-pass latency budget and cannot use few-step samplers or distillation — a GAN or feed-forward model may fit better.
Compute is severely constrained for both training and sampling and a simpler generator suffices.
You need exact, tractable log-likelihoods as a primary output rather than samples.
The task is discrete/structured in a way that does not map cleanly to continuous Gaussian noising without extra machinery.

Rules of thumb

Prototype with DDIM at 20–50 steps; only increase steps if quality is visibly limited.
For high resolution, do diffusion in latent space (Stable Diffusion-style) rather than on raw pixels.
Predict noise ( $\epsilon$ ) rather than $x_0$ for stable training, and tune the classifier-free guidance scale (≈5–8) for text-to-image.

Implementation

Reference code implementation

Python

model_fitting.py

1import torch
2
3T = 1000
4betas = torch.linspace(1e-4, 2e-2, T)        # linear noise schedule
5alphas = 1.0 - betas
6abar = torch.cumprod(alphas, dim=0)           # cumulative product -> bar-alpha_t
7
8def q_sample(x0, t, eps):
9    """Forward: jump straight to noise level t in one step (closed form)."""
10    a = abar[t].sqrt().view(-1, 1, 1, 1)
11    b = (1 - abar[t]).sqrt().view(-1, 1, 1, 1)
12    return a * x0 + b * eps                    # x_t = sqrt(abar) x0 + sqrt(1-abar) eps
13
14def training_loss(model, x0):
15    """The whole DDPM objective: predict the noise you added."""
16    n = x0.size(0)
17    t = torch.randint(0, T, (n,), device=x0.device)
18    eps = torch.randn_like(x0)                 # the target
19    xt = q_sample(x0, t, eps)
20    eps_pred = model(xt, t)                    # network guesses the noise
21    return ((eps - eps_pred) ** 2).mean()      # simple MSE loss
22
23# Sampling (sketch): start x_T ~ N(0, I), then for t = T-1 .. 0 subtract the
24# model's predicted noise (DDPM) or take a deterministic DDIM step.
25x0 = torch.rand(4, 3, 32, 32)
26print("Loss:", training_loss(lambda xt, t: torch.zeros_like(xt), x0).item())

Strengths & Advantages

Stable training: a single regression (denoising) loss with no adversarial min–max, so none of the instability or mode collapse that plagues GANs.
State-of-the-art sample quality and broad mode coverage, with a natural way to trade compute for fidelity by changing the number of sampling steps.
Flexible conditioning (text, class, image, inpainting masks) via cross-attention and classifier-free guidance, enabling controllable text-to-image generation.

Limitations & Drawbacks

Sampling is slow by default — hundreds of sequential network passes per image unless accelerated by DDIM, latent diffusion, or distillation.
Training and high-resolution sampling are compute- and memory-intensive (mitigated, but not eliminated, by latent-space diffusion).
Quality is sensitive to the noise schedule and guidance scale, and exact log-likelihood is less direct than in some other generative families.

Real-World Case Studies

DDPM sets a new bar for image generation on CIFAR-10

Generative image modeling

Scenario

Before 2020, GANs dominated high-fidelity image synthesis but were notoriously unstable to train and prone to mode collapse. It was unclear whether a likelihood-style model trained with a simple regression loss could compete on raw sample quality.

Approach

Ho, Jain and Abbeel trained a U-Net to predict the noise added by a fixed 1000-step Gaussian forward process, optimizing the simplified denoising MSE objective, then generated samples by running the learned reverse chain from pure noise back to a clean image.

Outcome

On unconditional CIFAR-10, DDPM reached an FID of 3.17 and an Inception score of 9.46, beating the strong GAN baselines of the time — and did so with a stable, non-adversarial training procedure. The result kicked off the wave of work (DDIM, score-based SDEs, latent diffusion) that made diffusion the default approach for image, audio, and video generation.

Source: Denoising Diffusion Probabilistic Models — Ho, J., Jain, A. and Abbeel, P.

Common Misconceptions

MisconceptionA diffusion model denoises a real, slightly-noisy photograph to produce its output.

CorrectionGeneration starts from a sample of pure Gaussian noise — there is no underlying clean photo. The image emerges purely from the network's learned reverse steps; the forward (noising) process is only used during training to create supervised noisy/clean pairs.

MisconceptionThe model removes all the noise in a single forward pass.

CorrectionSampling is iterative: DDPM uses hundreds to a thousand small reverse steps. A network can predict

x_0

in one step, but the high quality comes from many gradual refinements. DDIM and distillation reduce the step count, but the process is still fundamentally iterative.

MisconceptionThe network's job is to output the clean image directly.

CorrectionIn the standard formulation the network predicts the noise

\epsilon

that was added (equivalently, the score

\nabla_{x}\log p(x_t)

). The clean estimate

\hat x_0

is then derived algebraically. Noise/score/

x_0

parameterizations are mathematically interchangeable, and

\epsilon

-prediction trains most stably.

References & Further Reading

Denoising Diffusion Probabilistic Models (DDPM)paper
By Ho, J., Jain, A. and Abbeel, P.
View Resource →
Denoising Diffusion Implicit Models (DDIM)paper
By Song, J., Meng, C. and Ermon, S.
View Resource →
High-Resolution Image Synthesis with Latent Diffusion Modelspaper
By Rombach, R., Blattmann, A., Lorenz, D., Esser, P. and Ommer, B.
View Resource →

Diffusion Models

Prerequisites

TL;DR

Learning Objectives

Intuition

Interactive Diagram

The Mathematics

In Depth

Why diffusion took over generative vision

How It Compares

Diffusion vs GANs vs VAEs

When to Use It

Reach for this when

Avoid it when

Rules of thumb

Implementation

Strengths & Advantages

Limitations & Drawbacks

Real-World Case Studies

DDPM sets a new bar for image generation on CIFAR-10

Common Misconceptions

Self-Check Quiz

References & Further Reading

Related Topics

Generative Adversarial Networks

Autoencoders

Vision Transformers (ViT)

Convolutional Neural Networks