ML Learn

Learning AI & ML

Computer Vision

Vision-Language Models (CLIP)

Difficulty:Expert
Reading Time:35 min
Track:
Computer Vision
Align images and text in one shared embedding space by contrastive training on web-scale image–caption pairs — so a model can classify, retrieve, and reason about images using natural-language prompts, zero-shot.

Prerequisites

Vision Transformers (ViT)Transformers
Computer VisionModule 7 of 8

Vision-Language Models (CLIP)

TL;DR

  • Vision-language models (CLIP) train a dual encoder — image + text — to share one embedding space, using contrastive learning on web-scale (image, caption) pairs.
  • The objective: in a batch, matching image–text pairs get high cosine similarity, mismatched pairs low — a symmetric InfoNCE over the similarity matrix, with no class labels.
  • Zero-shot classification: embed text prompts ('a photo of a {label}') and pick the label whose embedding is nearest the image — open-vocabulary, defined at inference.
  • Natural-language supervision generalizes broadly and transfers robustly — CLIP matched a supervised ResNet-50 on ImageNet without ImageNet labels.
  • The shared space also powers retrieval, captioning, VQA, and text-to-image conditioning (the text encoder in diffusion models).
  • Limits: web bias, and weakness at fine-grained, counting, and compositional reasoning — strong gist, weak fine print.

Learning Objectives

  1. Explain CLIP's dual-encoder design and how contrastive training aligns image and text embeddings in a shared space
  2. Describe zero-shot classification by embedding text prompts and matching the nearest image embedding
  3. Reason about why web-scale natural-language supervision generalizes and transfers more robustly than fixed-label training
  4. Identify the limitations of vision-language models — bias, fine-grained and compositional weaknesses, and prompt sensitivity

Intuition

How to think conceptually about this topic

Imagine teaching someone about the world not with flashcards that say "this is exactly a Labrador," but by showing them millions of photos each paired with whatever caption a human wrote on the internet. They never get a tidy list of categories — just images and the words people naturally used. Over time they build a single mental space where a picture of a beach and the phrase "a sunny shoreline" sit in the same place.

Now you can quiz them in plain language. Show a new photo and ask "is this closer to 'a cat' or 'a car'?" — they just check which phrase feels nearest to the picture in that shared mental space. You never trained them on a "cat-vs-car" task; you simply described the options in words, and they matched. That is zero-shot classification, and it works for any labels you can phrase. The catch is that learning from messy web captions, they pick up the internet's blind spots and biases, and they grasp the gist of a scene better than its fine print — who is holding what, how many there are, exactly which breed.

Interactive Diagram

Test the intuition above by changing the model parameters

In Depth

Detailed explanations, contexts, and details

A vision-language model learns images and text together, in one shared representation. CLIP (Contrastive Language–Image Pretraining) is the canonical example, and its recipe is strikingly simple.

The recipe

  1. Collect hundreds of millions of (image, caption) pairs from the web — no manual labels.

  2. Run each image through an image encoder (a ViT or CNN) and each caption through a text encoder (a Transformer), projecting both into the same embedding space.

  3. Train contrastively: in a batch of NN pairs, the NN matching image–text pairs should have high similarity and the N2−NN^2 - N mismatched pairs low. That is it — no class labels anywhere.

What you get for free

Because supervision comes from language, the model is not boxed into a fixed label set. To classify an image zero-shot, you write the candidate classes as text — "a photo of a cat", "a photo of a dog" — embed them, and pick the one closest to the image embedding. The same shared space enables image–text retrieval (search images by sentence, or vice versa) and serves as the backbone for captioning and visual question answering (BLIP, Flamingo) and for steering text-to-image generation (the text encoder in diffusion models).

Why it generalizes

Natural-language supervision is broad and open-ended, so CLIP learns concepts far beyond any curated label list and transfers robustly — famously matching a supervised ResNet-50 on ImageNet without ever training on ImageNet's labels, while being markedly more robust to distribution shift. The trade-offs are real: web-scale data carries bias, and the global contrastive objective leaves the models weak at fine-grained and compositional reasoning.

How It Compares

Supervised Classifier vs CLIP (Vision-Language)

DimensionSupervised CNN/ViTCLIP (Vision-Language)
SupervisionFixed integer labelsNatural-language captions
Output spaceClosed set of K classesOpen vocabulary (any text prompt)
New classesRequire labeled data + retrainingJust write a new prompt (zero-shot)
Robustness to shiftOften brittleMarkedly more robust
Beyond classificationNeeds new heads/trainingRetrieval, captioning, VQA, gen. conditioning
TakeawayLanguage supervision trades a clean fixed taxonomy for an open, transferable, multi-purpose representation — powerful and flexible, but carrying web bias and weaker fine-grained precision.

When to Use It

Reach for this when

  • You need open-vocabulary classification, tagging, or moderation where the categories change or are not known in advance.
  • You want cross-modal retrieval — search images with text (or text with images) in a shared space.
  • You have no labeled data for the target classes but can describe them in words (zero-shot).
  • You need a text-conditioning backbone for generation or a strong general image encoder to build on.

Avoid it when

  • The task needs fine-grained, counting, or spatial/compositional precision that bag-of-concepts models handle poorly.
  • The application is bias-sensitive or safety-critical without auditing and mitigation of web-inherited biases.
  • You have ample labeled data for a fixed taxonomy and need maximum accuracy — a fine-tuned supervised model may win.
  • Latency or memory budgets cannot fit running two encoders, or prompts cannot be reliably engineered.

Rules of thumb

  • Use prompt templates and ensembling ('a photo of a {label}', plus variants) — wording materially affects zero-shot accuracy.
  • Treat zero-shot as a strong baseline; fine-tune (or linear-probe) on the target domain when you need the last few points.
  • Audit for bias and test on a compositional benchmark before trusting the model beyond coarse recognition.

Implementation

Reference code implementation

Python
model_fitting.py
1import torch
2import torch.nn.functional as F
3
4# Pretend encoders already produced L2-normalized embeddings (dim 512).
5image_emb = F.normalize(torch.randn(1, 512), dim=1)            # one image
6labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
7text_emb = F.normalize(torch.randn(len(labels), 512), dim=1)  # one row per prompt
8
9# Zero-shot classification = nearest text prompt in the shared space.
10logits = (image_emb @ text_emb.t()) / 0.07     # cosine similarity / temperature
11probs = logits.softmax(dim=1)
12pred = labels[int(probs.argmax())]
13print("Prediction:", pred, "  probs:", [round(p, 3) for p in probs[0].tolist()])
14
15# Training (sketch): build the NxN image-text similarity matrix for a batch and
16# apply symmetric cross-entropy so the diagonal (true pairs) scores highest.

Strengths & Advantages

  • Open-vocabulary: classify, retrieve, or filter by any natural-language prompt without task-specific labels or retraining.
  • Strong zero-shot transfer and notable robustness to distribution shift, learned from cheap web-scale image–text pairs.
  • A reusable shared embedding that powers retrieval, captioning, VQA, and text-conditioning for image generation (e.g. diffusion).

Limitations & Drawbacks

  • Inherits web-scale social biases and unsafe associations from uncurated training data.
  • Weak at fine-grained recognition, counting, spatial relations, and compositional reasoning — often behaving like a bag of concepts.
  • Sensitive to prompt wording, and requires enormous data and compute to train from scratch.

Real-World Case Studies

CLIP classifies ImageNet zero-shot — without ImageNet labels

Vision-language representation learning
Scenario

Supervised models defined a closed taxonomy and were brittle under distribution shift, while collecting labels was the bottleneck. The question was whether learning purely from naturally-occurring image–text pairs could yield a general, transferable visual model that needed no task-specific labels.

Approach

Radford et al. trained CLIP on ~400M image–text pairs scraped from the web with a symmetric contrastive objective over dual ViT/Transformer encoders. To classify, they embedded prompts like 'a photo of a {label}' for the target classes and chose the nearest text embedding to each image — no fine-tuning on the target dataset.

Outcome

CLIP reached about 76.2% zero-shot top-1 on ImageNet, matching a fully-supervised ResNet-50 trained on ImageNet's 1.28M labeled images — while using none of those labels — and it degraded far less on shifted test sets (ImageNet-R, ImageNet-Sketch, ImageNet-A). The shared embedding space went on to power retrieval, captioning/VQA systems, and the text conditioning in modern text-to-image diffusion models.

Source: Learning Transferable Visual Models From Natural Language Supervision — Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., et al.

Common Misconceptions

MisconceptionCLIP is trained to classify images into a fixed set of categories.
CorrectionCLIP is never trained on a fixed label set. It is trained contrastively to align images with their captions. Classification is an emergent zero-shot use: embed candidate label prompts and pick the nearest — so the 'classes' are open-vocabulary, defined at inference by the text you provide.
Misconception'Zero-shot' means the model did no training.
CorrectionIt means no task-specific labeled training for the new classes. CLIP was heavily pretrained on hundreds of millions of image–text pairs; zero-shot refers to transferring that knowledge to a new task with only natural-language prompts, not fine-tuning on labeled examples of it.
MisconceptionA vision-language model understands images the way humans do.
CorrectionThese models inherit web biases and are weak at fine-grained distinctions, counting, spatial relations, and compositional reasoning — often behaving like a bag of concepts. Strong zero-shot accuracy on common objects does not imply robust, human-like understanding.

References & Further Reading

  1. Learning Transferable Visual Models From Natural Language Supervision (CLIP)paper

    By Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., et al.

    View Resource →
  2. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (ALIGN)paper

    By Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., et al.

    View Resource →
  3. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationpaper

    By Li, J., Li, D., Xiong, C. and Hoi, S.

    View Resource →

Related Topics

Vision Transformers (ViT)

Apply the standard Transformer encoder directly to images by splitting the picture into a sequence of patches — trading the convolution's built-in locality for global, content-based attention that, with enough data, matches or beats CNNs.

Transformers

Sequence models built around attention, letting each token form a context-dependent weighted mixture of other token representations.

Self-Supervised Visual Pretraining

Learn powerful visual representations from images alone — no labels — by inventing a pretext task: contrast augmented views (SimCLR), reconstruct masked patches (MAE), or match a teacher (DINO). The labels come later, for a small fine-tuning step.

Diffusion Models

Generate images by learning to reverse a gradual noising process: a fixed forward chain destroys an image into pure Gaussian noise, and a trained network denoises step by step to turn fresh noise back into a sample.
← Previous Topic
Self-Supervised Visual Pretraining
Next Topic →
Diffusion Models