Autoencoders

Autoencoders are self-supervised neural networks designed for feature learning and dimensionality reduction. The network architecture is split into two halves:

The Encoder: Compresses high-dimensional inputs ( $x$ ) into a low-dimensional bottleneck representation ( $z$ , the latent space).
The Decoder: Takes the latent code ( $z$ ) and attempts to reconstruct the original input ( $\hat{x}$ ) as closely as possible.

Because the bottleneck has a very limited capacity, the network cannot simply copy the input. It is forced to discard noise and redundancy, learning only the most essential, underlying structural coordinates of the dataset.

Best use

Completely self-supervised: Requires no manual labels; uses the input dataset itself as the target.

Watch out for

Overfitting vulnerability: If the bottleneck is too wide, the network simply learns identity mapping and memorizes data.

Intuition

How to think about this algorithm

Imagine trying to describe a complex movie to a friend using a limit of only five words (the bottleneck). You cannot tell them about individual camera angles, minor characters, or costumes. You must choose five highly descriptive words that capture the exact essence of the film.

Your friend then takes those five words and tries to write the full plot summary. If their reconstruction matches the actual movie, you successfully compressed the movie's core signal. An autoencoder does this with numbers.

Interactive Diagram

Manifold Compression Autoencoders

Click anywhere on the coordinate plot to position the Input data (cyan). Observe how it compresses onto the training manifold bottleneck (pink).

Input coordinate

Decoded reconstruction

Bottleneck Manifold

Reconstruction Loss

[Click plot space to compress coordinates]

Bottleneck scalar (z): 3.350

Reconstruction Loss: 3.073

Key InsightAutoencoders bottleneck information by mapping inputs to a low-dimensional space. The decoder reconstructs outputs, confining prediction spaces onto the target manifold.

∑

The Logic

Mathematical core for autoencoders

1. Feed-Forward Reconstruction

The encoder maps input $x$ to latent code $z$ :

$z = g_{\phi}(x) = \sigma(W_e x + b_e)$

The decoder maps latent code $z$ to output estimate $\hat{x}$ :

$\hat{x} = f_{\theta}(z) = \sigma(W_d z + b_d)$

We train the weights by minimizing the Reconstruction Loss:

$\mathcal{L}_{recon}(x, \hat{x}) = \|x - \hat{x}\|^2 \quad \text{(Mean Squared Error)}$

2. Variational Autoencoders (VAEs)

Instead of outputting a fixed point, VAEs output probability distribution parameters (mean $\mu$ and variance $\sigma^2$ ). They optimize the Evidence Lower Bound (ELBO):

$\mathcal{L}_{VAE}(\theta, \phi; x) = \mathbb{E}_{q_{\phi}(z|x)}[\log p_{\theta}(x|z)] - D_{\text{KL}}(q_{\phi}(z|x) \parallel p(z))$

Where the KL Divergence ( $D_{\text{KL}}$ ) forces the latent space to be a smooth, continuous Gaussian, allowing us to generate new realistic samples by picking random points.

Code Example

autoencoders.py · pytorch example

Python

model_fitting.py

1import torch
2import torch.nn as nn
3import torch.optim as optim
4
5# Simple Autoencoder in PyTorch
6class Autoencoder(nn.Module):
7    def __init__(self, input_dim=784, latent_dim=32):
8        super().__init__()
9        # Encoder: compresses from 784 down to 32 dimensions
10        self.encoder = nn.Sequential(
11            nn.Linear(input_dim, 128),
12            nn.ReLU(),
13            nn.Linear(128, latent_dim)
14        )
15        # Decoder: reconstructs 784 back from 32 dimensions
16        self.decoder = nn.Sequential(
17            nn.Linear(latent_dim, 128),
18            nn.ReLU(),
19            nn.Linear(128, input_dim),
20            nn.Sigmoid() # Squish outputs between 0 and 1 (like normalized pixels)
21        )
22
23    def forward(self, x):
24        latent = self.encoder(x)
25        reconstruction = self.decoder(latent)
26        return reconstruction
27
28# Define model, loss, and run training step
29model = Autoencoder()
30criterion = nn.MSELoss()
31optimizer = optim.Adam(model.parameters(), lr=0.005)
32
33# Fake inputs representing 4 flat image vectors (e.g. MNIST)
34fake_inputs = torch.rand(4, 784)
35
36optimizer.zero_grad()
37outputs = model(fake_inputs)
38loss = criterion(outputs, fake_inputs) # Target is the same as the input!
39loss.backward()
40optimizer.step()
41
42print(f"Reconstruction Loss: {loss.item():.4f}")

✓

Strengths

Completely self-supervised: Requires no manual labels; uses the input dataset itself as the target.
Powerful denoising: Denoising autoencoders learn to strip out corruption and restore clean signals.
Smooth latent coordinates: VAEs provide continuous latent spaces, perfect for generating new data samples.

Limitations

Overfitting vulnerability: If the bottleneck is too wide, the network simply learns identity mapping and memorizes data.
Blurry reconstructions: Because they use MSE loss, reconstructions tend to be smooth averages, losing fine details.
Disconnected latent spaces: Standard autoencoders can have massive empty regions in latent space, where output is garbage.

Key Assumptions

Scope conditions and interpretation notes

1
The input data resides on a lower-dimensional manifold in high-dimensional space.
2
A bottleneck capacity restricts the network from copying inputs.

References

Books and papers for deeper study

Hinton, G. E. and Salakhutdinov, R. R. (2006) 'Reducing the dimensionality of data with neural networks', Science, 313(5786), pp. 504-507.