Computer Vision

Image Segmentation

Difficulty:Advanced

Reading Time:30 min

Track:

Computer Vision

Dense, per-pixel prediction that assigns every pixel in an image to a class — the most spatially detailed level of visual understanding, scored by how well predicted masks overlap the ground truth.

Prerequisites

Convolutional Neural Networks Computer Vision Foundations

Computer VisionModule 4 of 8

Image Segmentation

50%

TL;DR

Segmentation predicts a class for every pixel, so the output is a mask the same size as the input — the most spatially detailed vision task.
Semantic merges all pixels of a class; instance separates individual objects; panoptic does both.
It is trained with a per-pixel loss (pixel-wise cross-entropy) because a single image-level label cannot say where an object's boundary is.
Quality is measured by mask overlap — Dice $= 2TP/(2TP+FP+FN)$ and IoU $= TP/(TP+FP+FN)$ — which ignore the huge background and so survive class imbalance.
Encoder–decoder networks (U-Net) downsample for context then upsample for detail, with skip connections restoring the sharp boundaries lost during downsampling.
For small or rare structures, swap or augment cross-entropy with soft Dice / focal loss so the background doesn't dominate training.

Learning Objectives

Distinguish semantic, instance, and panoptic segmentation by what each output represents
Explain why segmentation uses a per-pixel loss and how encoder–decoder (U-Net) architectures recover spatial resolution
Compute and contrast the Dice coefficient and Intersection over Union (IoU) from true/false positives and negatives
Choose an appropriate loss (pixel-wise cross-entropy vs Dice/focal) for class-imbalanced segmentation problems

Intuition

How to think conceptually about this topic

Think of handing an artist a photo and three different instructions.

"Tell me what's in it" — they shout "a street scene!" (classification). "Box the cars" — they draw rectangles (detection). "Color it in by hand" — they take colored pencils and shade every pixel: road grey, cars blue, people red, sky pale (semantic segmentation). If you further ask them to give each car its own shade of blue, that's instance segmentation.

The catch: to color in the picture well, the artist must reason about both the big picture (where the road is) and the fine detail (the exact silhouette of each car). Segmentation networks mirror this — a "contracting" path that zooms out to understand context, and an "expanding" path that zooms back in to draw crisp boundaries.

Interactive Diagram

Test the intuition above by changing the model parameters

In Depth

Detailed explanations, contexts, and details

Image segmentation is the most spatially detailed task in computer vision: instead of one label per image (classification) or a box per object (detection), it predicts a class label for every pixel. The output is a mask the same height and width as the input.

There are three common flavors, in increasing richness:

Semantic segmentation — every pixel gets a class, but all instances of a class are merged (all "road" pixels are one region).
Instance segmentation — every pixel of every object instance is labeled separately (car #1's mask is distinct from car #2's), but "stuff" classes like sky or road are usually ignored.
Panoptic segmentation — unifies the two: every pixel gets both a class and, for countable "things", an instance id.

Where is it used?

Segmentation powers medical imaging (delineating tumors and organs), autonomous driving (free-space and lane estimation), satellite and aerial mapping (land-use classification), photo editing (background removal, "portrait mode"), and robotics (graspable-region detection). Anywhere a system needs the exact shape of things, not just their presence or rough location, segmentation is the tool.

How It Compares

Semantic vs Instance vs Panoptic Segmentation

Dimension	Semantic	Instance	Panoptic
What each pixel gets	A class label only	A class label + an instance id (for objects)	A class label + an instance id for every pixel
Separates individual objects?	No — all instances of a class merge	Yes — car #1 vs car #2	Yes, for countable 'things'
Handles 'stuff' (sky, road)?	Yes	Usually no — focuses on countable objects	Yes — unifies 'stuff' and 'things'
Typical architecture	FCN, U-Net, DeepLab	Mask R-CNN (detect then segment per box)	Panoptic FPN, Mask2Former
Primary metric	mean IoU (mIoU)	mask Average Precision (AP)	Panoptic Quality (PQ)

TakeawayPick the least expensive task that answers your question: semantic if you only need the class map, instance if you must count or separate objects, panoptic if you need a single complete labeling of the whole scene.

When to Use It

Reach for this when

You need the exact shape of objects, not just their presence or a bounding box (e.g. tumor delineation, free-space estimation, background removal).
Downstream geometry depends on pixel-precise boundaries (measuring areas, cutting masks, 3D reconstruction).
Objects have irregular or overlapping outlines that a rectangle would misrepresent.
You can obtain (or bootstrap from a foundation model like SAM) pixel-level masks for training or fine-tuning.

Avoid it when

A bounding box or a single image label already answers the question — segmentation adds annotation and compute cost for no benefit.
You cannot afford pixel-level annotation and no usable pre-trained/zero-shot model exists for your domain.
Hard real-time, low-power constraints make a full-resolution decoder too slow — a lighter detector may be the better trade-off.
Your masks are extremely imbalanced and you can only use plain pixel accuracy/cross-entropy — without overlap-based losses and metrics the task will silently fail.

Rules of thumb

Report Dice or mIoU, never pixel accuracy, on imbalanced masks.
Start from a U-Net or DeepLab backbone pre-trained on a large dataset and fine-tune; training a decoder from scratch is rarely worth it.
Combine cross-entropy with Dice (or focal) loss when the foreground is small.

Implementation

Reference code implementation

Python

model_fitting.py

1import torch
2from torchvision.models.segmentation import deeplabv3_resnet50
3
4# Pre-trained semantic segmentation model (21 Pascal VOC classes)
5model = deeplabv3_resnet50(weights="DEFAULT").eval()
6
7img = torch.rand(1, 3, 256, 256)          # [batch, channels, H, W]
8
9with torch.no_grad():
10    logits = model(img)["out"]            # [1, 21, 256, 256] per-pixel class logits
11
12# Hard mask: the argmax class at every pixel
13mask = logits.argmax(dim=1)               # [1, 256, 256]
14print("Mask shape:", tuple(mask.shape))
15print("Classes present:", torch.unique(mask).tolist())
16
17def dice_score(pred, target, eps=1e-6):
18    pred, target = pred.float(), target.float()
19    inter = (pred * target).sum()
20    return ((2 * inter + eps) / (pred.sum() + target.sum() + eps)).item()
21
22# Dice of the 'person' class (id 15) against a stand-in ground truth
23person = (mask == 15).float()
24print("Dice(person):", dice_score(person, person))   # 1.0 against itself

Strengths & Advantages

Produces exact object shapes and boundaries — the richest spatial output in vision.
Overlap metrics (Dice/IoU) give a principled, imbalance-robust way to measure quality.
Strong pre-trained backbones and architectures (U-Net, DeepLab, Mask R-CNN, SAM) transfer well to new domains with little data.

Limitations & Drawbacks

Pixel-level annotation is the most expensive labeling there is — every boundary must be traced.
Heavy compute and memory: full-resolution feature maps are large, especially for high-resolution or 3D (volumetric) data.
Sensitive to class imbalance and fuzzy boundaries; needs careful loss design to segment small or thin structures.

Real-World Case Studies

U-Net wins the ISBI cell-segmentation challenge with only 30 training images

Biomedical imaging

Scenario

Microscopy segmentation datasets are tiny — the 2015 ISBI cell tracking challenge provided only a few dozen annotated images — yet pixel-accurate cell boundaries were required. Conventional deep networks of the time were data-hungry and produced blurry boundaries that merged touching cells.

Approach

U-Net paired a contracting encoder with a symmetric expanding decoder linked by skip connections, trained with heavy elastic deformation augmentation and a weighted loss that emphasized the thin borders separating touching cells, so very few images yielded many effective training examples.

Outcome

U-Net achieved an IoU of about 0.92 on the PhC-U373 dataset and 0.78 on DIC-HeLa, beating the next-best methods (roughly 0.83 and 0.46 respectively) by a wide margin — and did so fast enough to segment a 512×512 image in well under a second on a GPU. The architecture became the default backbone for biomedical and many general segmentation tasks.

Source: U-Net: Convolutional Networks for Biomedical Image Segmentation — Ronneberger, O., Fischer, P. and Brox, T.

Common Misconceptions

MisconceptionSemantic segmentation can count how many separate objects of a class are present.

CorrectionSemantic segmentation merges all pixels of a class into one region, so it cannot tell two touching cars apart. Counting separate instances requires instance (or panoptic) segmentation.

MisconceptionSegmentation is just image classification run independently on each pixel.

CorrectionA pixel's label depends on its spatial context (neighbors, object shape, scene layout). Segmentation networks use large receptive fields and encoder–decoder structure with skip connections to combine global context with local detail — far more than per-pixel classification in isolation.

References & Further Reading

U-Net: Convolutional Networks for Biomedical Image Segmentationpaper
By Ronneberger, O., Fischer, P. and Brox, T.
View Resource →
Fully Convolutional Networks for Semantic Segmentationpaper
By Long, J., Shelhamer, E. and Darrell, T.
View Resource →
Mask R-CNNpaper
By He, K., Gkioxari, G., Dollár, P. and Girshick, R.
View Resource →

Image Segmentation

Prerequisites

TL;DR

Learning Objectives

Intuition

Interactive Diagram

The Mathematics

In Depth

Where is it used?

How It Compares

Semantic vs Instance vs Panoptic Segmentation

When to Use It

Reach for this when

Avoid it when

Rules of thumb

Implementation

Strengths & Advantages

Limitations & Drawbacks

Real-World Case Studies

U-Net wins the ISBI cell-segmentation challenge with only 30 training images

Common Misconceptions

Self-Check Quiz

References & Further Reading

Related Topics

Computer Vision Foundations

Convolutional Neural Networks

Autoencoders