Computer Vision

Vision Transformers (ViT)

Difficulty:Expert

Reading Time:35 min

Track:

Computer Vision

Apply the standard Transformer encoder directly to images by splitting the picture into a sequence of patches — trading the convolution's built-in locality for global, content-based attention that, with enough data, matches or beats CNNs.

Prerequisites

Transformers Computer Vision Foundations

Computer VisionModule 5 of 8

Vision Transformers (ViT)

63%

TL;DR

A Vision Transformer feeds an image to the standard Transformer encoder by first cutting it into patches and treating each as a token.
Pipeline: patchify → linear patch embedding → add position embeddings + a [CLS] token → Transformer encoder → classify from the final [CLS].
Self-attention gives every patch a global receptive field in layer 1, unlike a convolution whose receptive field grows slowly with depth.
ViT discards the CNN's locality and translation-equivariance inductive biases, so it is data-hungry: it needs large-scale pre-training (or distillation) to beat CNNs, and loses to them on small datasets.
Cost is quadratic in patch count ( $N = HW/P^2$ ), which is why patches are coarse and hierarchical/windowed variants (Swin) exist.
The patch-token formulation underpins modern vision: DETR (detection as set prediction) and CLIP (contrastive image–text alignment for zero-shot recognition).

Learning Objectives

Explain how an image becomes a sequence of tokens via patch embedding, position embeddings, and a prepended [CLS] token
Contrast the inductive biases of CNNs (locality, translation equivariance) with the Vision Transformer's near-absence of them, and why that makes ViT data-hungry
Describe why self-attention gives every patch a global receptive field from the very first layer, unlike a convolution
Connect the patch-token idea to downstream vision systems — DETR for detection and CLIP for image–text alignment

Intuition

How to think conceptually about this topic

A CNN reads an image like someone studying it through a small magnifying glass: it can only see a little neighborhood at a time and slides the glass around, slowly building up context layer by layer. Locality is wired in — distant corners of the image cannot directly influence each other until many layers deep.

A Vision Transformer instead chops the picture into a deck of cards (the patches), lays them all on the table, and lets every card look at every other card at once. A patch showing a wheel can immediately attend to a patch showing a car door on the opposite side of the image and conclude "vehicle", with no notion that they happen to be near or far. Nothing tells it that adjacent patches are special — it must learn that from examples. Give it few examples and it flails; give it hundreds of millions and it discovers spatial structure on its own, plus long-range relationships a small magnifying glass would take many layers to assemble.

Interactive Diagram

Test the intuition above by changing the model parameters

In Depth

Detailed explanations, contexts, and details

The Vision Transformer (ViT) asks a deliberately blunt question: what if we drop the convolution entirely and feed an image to the same Transformer encoder used for language? The answer, from the 2020 ViT paper, is that it works remarkably well — provided you train on enough data.

The recipe is short:

Patchify. Cut the image into a grid of fixed-size, non-overlapping patches (commonly 16×16 pixels). A 224×224 image becomes a 14×14 grid of 196 patches.
Embed. Flatten each patch and linearly project it to a $D$ -dimensional vector — a patch embedding, the visual counterpart of a word embedding.
Add position + [CLS]. Prepend a learnable [CLS] token and add learned position embeddings so the model knows where each patch sits.
Encode. Run the sequence through a standard Transformer encoder (multi-head self-attention + MLP blocks).
Read out. Feed the final [CLS] representation to a small classification head.

Why this matters

A convolution hard-codes two assumptions: nearby pixels matter most (locality) and a feature detector should behave the same everywhere (translation equivariance). These priors make CNNs superb when data is scarce. ViT throws them away and instead lets global, content-based attention decide which patches should talk to each other — from the very first layer. That flexibility is a liability with little data and an advantage at scale, and the patch-token formulation has become the substrate for detection (DETR), open-vocabulary recognition (CLIP), and segmentation (SAM, Mask2Former).

How It Compares

Vision Transformer vs Convolutional Neural Network

Dimension	Vision Transformer (ViT)	CNN
Built-in inductive bias	Almost none — must learn spatial structure from data	Locality + translation equivariance hard-coded
Receptive field in layer 1	Global — every patch attends to every patch	Local — limited to the kernel's neighborhood
Data efficiency	Low — needs large-scale pre-training or distillation	High — trains well on modest datasets
Compute scaling with input size	Quadratic in patch count ( $O(N^2)$ )	Roughly linear in pixels
Where it shines	Large data, multimodal (CLIP), long-range relations	Small/medium data, edge/real-time, dense local features

TakeawayViT trades the convolution's hand-coded priors for flexibility: that costs data but pays off at scale and unlocks a shared architecture for vision and language. Hybrids (conv stem + attention) try to get both.

When to Use It

Reach for this when

You have access to a large pre-trained backbone (or can pre-train at scale) and want state-of-the-art transfer performance.
The task needs long-range or global reasoning across the image that a small-kernel CNN reaches only deep in the stack.
You want a multimodal system — e.g. image–text retrieval, zero-shot classification, or captioning — where sharing the Transformer/embedding space with text is valuable (CLIP-style).
You are building modern detection or segmentation heads (DETR, Mask2Former, SAM) that assume a token/attention backbone.

Avoid it when

You must train from scratch on a small dataset with no pre-training — a CNN's inductive biases will usually win.
You face tight real-time or low-power constraints at high resolution, where quadratic attention is too costly relative to an efficient CNN.
Your problem is dominated by local texture with little long-range structure, where convolutions are already a near-perfect fit.
You lack the engineering budget for the heavy augmentation/regularization schedules ViTs typically need to train stably.

Rules of thumb

Default to fine-tuning a pre-trained ViT rather than training one from scratch.
If data is limited and you still want attention, reach for a distilled (DeiT) or hybrid/windowed (Swin) variant.
Pick patch size to balance accuracy and cost: smaller patches = more tokens = more accuracy but quadratically more compute.

Implementation

Reference code implementation

Python

model_fitting.py

1import torch
2import torch.nn as nn
3
4class PatchEmbed(nn.Module):
5    """Turn an image into a sequence of patch tokens (the heart of ViT)."""
6    def __init__(self, img=224, patch=16, in_ch=3, dim=768):
7        super().__init__()
8        self.n_patches = (img // patch) ** 2          # 224/16 = 14 -> 196 patches
9        # A stride-patch conv is an efficient way to flatten + project each patch.
10        self.proj = nn.Conv2d(in_ch, dim, kernel_size=patch, stride=patch)
11        self.cls = nn.Parameter(torch.zeros(1, 1, dim))
12        self.pos = nn.Parameter(torch.zeros(1, self.n_patches + 1, dim))
13
14    def forward(self, x):                              # x: [B, 3, 224, 224]
15        x = self.proj(x)                               # [B, 768, 14, 14]
16        x = x.flatten(2).transpose(1, 2)               # [B, 196, 768] patch tokens
17        cls = self.cls.expand(x.size(0), -1, -1)       # [B, 1, 768]
18        x = torch.cat([cls, x], dim=1) + self.pos      # [B, 197, 768] (+CLS, +pos)
19        return x
20
21tokens = PatchEmbed()(torch.rand(2, 3, 224, 224))
22print("Sequence shape:", tuple(tokens.shape))          # (2, 197, 768)
23# Feed 'tokens' to a standard nn.TransformerEncoder; read out tokens[:, 0] (the CLS).

Strengths & Advantages

Global receptive field from the first layer: any patch can directly attend to any other, capturing long-range relationships a CNN reaches only after many layers.
Scales smoothly with data and model size — given large-scale pre-training, ViTs match or surpass the best CNNs at lower pre-training compute.
Reuses the exact Transformer stack from language, so vision and text can share one architecture and even one embedding space (CLIP), enabling powerful multimodal systems.

Limitations & Drawbacks

Data-hungry: without large-scale pre-training or distillation, a from-scratch ViT is beaten by a comparable CNN because it lacks locality and translation-equivariance priors.
Self-attention is quadratic in patch count, so high-resolution or dense-prediction use cases need windowed/hierarchical attention to stay affordable.
Less interpretable spatially than convolutions and sensitive to patch size and position-embedding choices, which interact with input resolution.

Real-World Case Studies

ViT overtakes CNNs on ImageNet — but only after pre-training on 300M images

Image classification

Scenario

By 2020, convolutional networks had defined image recognition for nearly a decade. The ViT authors wanted to know whether a near-pure Transformer, stripped of convolutional priors, could compete — and crucially, how that answer depends on the amount of pre-training data.

Approach

They trained the same ViT architecture under three pre-training regimes — ImageNet-1k (~1.3M images), ImageNet-21k (~14M), and the in-house JFT-300M (~300M) — then fine-tuned and compared against strong ResNet-based Big Transfer (BiT) baselines, holding the transfer protocol fixed.

Outcome

The data dependence was stark. Pre-trained only on ImageNet-1k, ViT underperformed comparable ResNets; on ImageNet-21k it drew level; pre-trained on JFT-300M, ViT-H/14 reached 88.55% ImageNet top-1, beating the best BiT CNN while using substantially less pre-training compute. The experiment crystallized the rule that flexible, low-bias models overtake hand-biased ones once data is abundant.

Source: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., et al.

Common Misconceptions

MisconceptionA Vision Transformer runs self-attention over individual pixels.

CorrectionPixel-level attention would cost

O((HW)^2)

and is infeasible for real images. ViT first groups pixels into patches (e.g. 16×16) and attends over the few hundred resulting patch tokens, not the tens of thousands of pixels.

MisconceptionVision Transformers are simply better than CNNs, so CNNs are obsolete.

CorrectionViT only matches or beats CNNs when pre-trained on very large datasets. With limited data, a CNN's locality and translation-equivariance biases win decisively. Distillation (DeiT), convolutional stems, and hybrid models exist precisely to give ViTs some of that bias back.

MisconceptionSelf-attention is translation equivariant, just like a convolution.

CorrectionAdding (typically learned) position embeddings breaks permutation/translation invariance on purpose so the model can reason about location. ViT therefore does not inherit the convolution's translation equivariance; it must learn any such regularity from data.

References & Further Reading

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalepaper
By Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., et al.
View Resource →
End-to-End Object Detection with Transformers (DETR)paper
By Carion, N., Massa, F., Synnaeve, G., Usunier, N., et al.
View Resource →
Learning Transferable Visual Models From Natural Language Supervision (CLIP)paper
By Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., et al.
View Resource →

Vision Transformers (ViT)

Prerequisites

TL;DR

Learning Objectives

Intuition

Interactive Diagram

The Mathematics

In Depth

Why this matters

How It Compares

Vision Transformer vs Convolutional Neural Network

When to Use It

Reach for this when

Avoid it when

Rules of thumb

Implementation

Strengths & Advantages

Limitations & Drawbacks

Real-World Case Studies

ViT overtakes CNNs on ImageNet — but only after pre-training on 300M images

Common Misconceptions

Self-Check Quiz

References & Further Reading

Related Topics

Transformers

Convolutional Neural Networks

Computer Vision Foundations

Image Segmentation