ML Learn

Learning AI & ML

Computer Vision

Object Detection

Difficulty:Advanced
Reading Time:35 min
Track:
Computer Vision
Find and classify every object in an image by predicting a labeled bounding box for each — the localization step between whole-image classification and per-pixel segmentation, scored by box overlap and mean average precision.

Prerequisites

Convolutional Neural NetworksComputer Vision Foundations
Computer VisionModule 3 of 8

Object Detection

TL;DR

  • Detection = localization + classification of a variable number of objects, each output as a labeled, scored bounding box.
  • Two-stage detectors (R-CNN → Fast → Faster R-CNN) propose regions then classify/refine them; one-stage detectors (YOLO, SSD, RetinaNet) predict boxes directly in a single pass — faster, historically less accurate.
  • Detectors predict offsets from anchor boxes, not raw coordinates, and emit many redundant boxes that non-maximum suppression collapses to one per object.
  • Quality is matched by IoU and summarized by mean Average Precision (mAP) — the area under the precision–recall curve, averaged over classes and IoU thresholds.
  • Focal loss (RetinaNet) fixes the foreground–background imbalance of dense one-stage detectors, closing the accuracy gap with two-stage methods.
  • DETR recasts detection as direct set prediction with bipartite matching, removing anchors and NMS entirely.

Learning Objectives

  1. Explain why detection needs both localization and classification for a variable, unknown number of objects
  2. Contrast two-stage (R-CNN family) and one-stage (YOLO, SSD, RetinaNet) detectors and their speed/accuracy trade-off
  3. Describe anchors, bounding-box regression, and how non-maximum suppression removes duplicate detections
  4. Interpret IoU, precision–recall, and mean average precision (mAP), and explain focal loss's role in fixing foreground–background imbalance

Intuition

How to think conceptually about this topic

Imagine asking someone to mark up a busy street photo. Classification is them saying "it's a street scene." Detection is handing them a stack of sticky rectangles and saying "put one labeled box around each car, person, and sign — tightly."

Two strategies emerge. One person first squints and jots down "objects probably live here, here, and here" (region proposals), then carefully examines and labels each spot — thorough but slow. That is the two-stage approach. Another person, trained by lots of practice, just glances once and slaps down all the boxes in a single sweep — faster, occasionally messier. That is the one-stage approach.

Both run into the same nuisance: they instinctively stick several overlapping rectangles on each car. So afterward they tidy up — for each cluster of overlapping boxes on the same object, keep the most confident one and peel the rest off. That cleanup is non-maximum suppression, and it is why the raw, redundant output becomes one clean box per object.

Interactive Diagram

Test the intuition above by changing the model parameters

In Depth

Detailed explanations, contexts, and details

Object detection sits between two tasks you already know. Classification answers "what is in this image?" with one label; segmentation answers "which class is every pixel?". Detection answers "what objects are here, and where?" — drawing a labeled, scored bounding box around each one. The hard part is that the number of objects is unknown and varies per image, and each must be both classified and precisely localized at any scale or position.

Two families

Two-stage detectors (the R-CNN lineage) first propose candidate regions likely to contain objects, then classify and refine each:

  • R-CNN ran a CNN on ~2000 region proposals independently — accurate but very slow.

  • Fast R-CNN shared one convolutional pass and pooled features per region (ROI pooling).

  • Faster R-CNN replaced external proposals with a learned Region Proposal Network using anchors, making the whole pipeline trainable and much faster.

One-stage detectors skip the proposal step and predict boxes and classes directly over a dense grid of anchors in a single pass:

  • YOLO frames detection as one regression over a grid — extremely fast.

  • SSD predicts at multiple feature-map scales.

  • RetinaNet added focal loss to overcome the background imbalance that had held one-stage accuracy back.

Scoring

Detections are matched to ground truth by Intersection over Union (IoU); sweeping the confidence threshold yields a precision–recall curve whose area is Average Precision, averaged into mAP. Newer transformer detectors (DETR) recast detection as direct set prediction, removing anchors and NMS entirely.

How It Compares

Two-Stage vs One-Stage vs Set-Prediction Detectors

DimensionTwo-Stage (Faster R-CNN)One-Stage (YOLO/RetinaNet)Set Prediction (DETR)
PipelinePropose regions, then classify/refinePredict boxes directly over dense anchorsTransformer predicts a fixed set of boxes
SpeedSlowerFast / real-timeModerate
AccuracyHistorically highestMatched two-stage with focal lossCompetitive, simpler pipeline
Needs anchors & NMS?Yes (anchors + NMS)Yes (anchors + NMS)No — neither
Class imbalance handlingProposal stage filters backgroundFocal loss / hard-negative miningBipartite matching loss
TakeawayPick one-stage (YOLO/RetinaNet) for real-time budgets, two-stage (Faster R-CNN) when accuracy is paramount, and DETR-style models when you want to drop the anchor/NMS machinery for an end-to-end pipeline.

When to Use It

Reach for this when

  • You need to count, locate, or track multiple objects in a scene (driving, surveillance, retail shelf analysis, robotics).
  • A bounding box is enough spatial precision — you do not need exact pixel shape (else use segmentation).
  • You have a latency budget to respect: choose a one-stage detector for real-time, a two-stage one for maximum accuracy.
  • Box-level annotations are available or affordable, which they usually are relative to per-pixel masks.

Avoid it when

  • You only need a single image-level label — plain classification is far cheaper.
  • You need the exact outline of objects (medical contours, background removal) — use instance segmentation instead.
  • Objects are rotated, articulated, or extremely crowded/overlapping, where axis-aligned boxes and NMS struggle (consider rotated boxes, keypoints, or set prediction).
  • You cannot tune the post-processing (NMS/score thresholds) and need a fully end-to-end model — prefer a DETR-style detector.

Rules of thumb

  • Start from a pretrained detector (Faster R-CNN or a YOLO variant) and fine-tune; rarely train from scratch.
  • Report mAP@[.5:.95], not accuracy, and tune the NMS IoU and confidence thresholds on a validation set.
  • If a dense one-stage detector underfits objects, suspect class imbalance and use focal loss.

Implementation

Reference code implementation

Python
model_fitting.py
1import torch
2from torchvision.models.detection import fasterrcnn_resnet50_fpn
3from torchvision.ops import nms
4
5# Pretrained two-stage detector (COCO, 91 classes).
6model = fasterrcnn_resnet50_fpn(weights="DEFAULT").eval()
7
8img = torch.rand(3, 480, 640)                 # [channels, H, W]
9with torch.no_grad():
10    out = model([img])[0]                     # dict: boxes, labels, scores
11
12boxes, scores, labels = out["boxes"], out["scores"], out["labels"]
13
14# Keep confident boxes, then suppress overlapping duplicates (NMS).
15keep_conf = scores > 0.5
16boxes, scores, labels = boxes[keep_conf], scores[keep_conf], labels[keep_conf]
17keep = nms(boxes, scores, iou_threshold=0.5)  # greedy non-maximum suppression
18print("Detections after NMS:", len(keep))
19
20def iou(a, b):
21    x1, y1 = torch.max(a[:2], b[:2]); x2, y2 = torch.min(a[2:], b[2:])
22    inter = (x2 - x1).clamp(min=0) * (y2 - y1).clamp(min=0)
23    area = lambda z: (z[2] - z[0]) * (z[3] - z[1])
24    union = area(a) + area(b) - inter
25    return (inter / union).item() if union > 0 else 0.0

Strengths & Advantages

  • Localizes and labels a variable number of objects at once — the workhorse task for autonomous driving, surveillance, retail, and robotics.
  • A clear speed/accuracy spectrum (one-stage real-time vs two-stage high-accuracy) lets you match the detector to the deployment budget.
  • Mature, well-pretrained backbones and heads transfer well, and box annotation is far cheaper than per-pixel segmentation masks.

Limitations & Drawbacks

  • Anchors, NMS thresholds, and IoU/score cutoffs add hyperparameters and hand-tuned post-processing (which DETR-style set prediction tries to remove).
  • Dense detectors suffer severe foreground–background imbalance and struggle with very small, crowded, or heavily overlapping objects.
  • Boxes are coarse: they cannot capture exact shape (use segmentation) and axis-aligned boxes fit rotated or articulated objects poorly.

Real-World Case Studies

Faster R-CNN makes region proposals learnable — and near real-time

Object detection
Scenario

By 2015, the R-CNN family was accurate but bottlenecked by an external, hand-crafted region-proposal step (selective search) that ran on the CPU and dominated runtime, making detection far too slow for practical or real-time use.

Approach

Ren et al. introduced the Region Proposal Network (RPN): a small convolutional network that slides over the shared feature map and, using a set of reference anchor boxes at multiple scales and aspect ratios, predicts objectness scores and box refinements. The RPN shares convolutional features with the detection head, so proposals become a learned, nearly free part of one end-to-end network.

Outcome

Faster R-CNN reached about 73.2% mAP on PASCAL VOC 2007 while running at roughly 5 frames per second on a GPU — orders of magnitude faster than selective-search pipelines — and won multiple tracks of the ILSVRC and COCO 2015 detection challenges. Anchors and the RPN became foundational ideas reused across later detectors.

Source: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks — Ren, S., He, K., Girshick, R. and Sun, J.

Common Misconceptions

MisconceptionObject detection is just image classification run on cropped regions.
CorrectionDetection must also localize (regress precise box coordinates) and handle a variable, unknown number of objects at many scales and positions. Naively classifying every crop is the slow original R-CNN idea; modern detectors share computation and predict boxes and classes jointly.
MisconceptionMore predicted boxes mean better detection.
CorrectionDetectors emit thousands of overlapping candidate boxes per object; raw output is massively redundant. Non-maximum suppression (or set-based training, as in DETR) is required to collapse duplicates to one box per object — otherwise precision collapses.
MisconceptionOne-stage detectors are always less accurate than two-stage ones.
CorrectionOne-stage detectors (YOLO, SSD) were initially less accurate mainly because of extreme foreground–background class imbalance. Focal loss (RetinaNet) fixed this, letting a one-stage detector match two-stage accuracy while staying faster.

References & Further Reading

  1. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkspaper

    By Ren, S., He, K., Girshick, R. and Sun, J.

    View Resource →
  2. You Only Look Once: Unified, Real-Time Object Detection (YOLO)paper

    By Redmon, J., Divvala, S., Girshick, R. and Farhadi, A.

    View Resource →
  3. Focal Loss for Dense Object Detection (RetinaNet)paper

    By Lin, T.-Y., Goyal, P., Girshick, R., He, K. and Dollár, P.

    View Resource →

Related Topics

Computer Vision Foundations

The broad field of enabling computers to see, segment, track, and interpret visual data from the physical world.

Image Segmentation

Dense, per-pixel prediction that assigns every pixel in an image to a class — the most spatially detailed level of visual understanding, scored by how well predicted masks overlap the ground truth.

Convolutional Neural Networks

Neural networks for grid-like data that reuse small filters across space to learn local, translation-aware features.

Vision Transformers (ViT)

Apply the standard Transformer encoder directly to images by splitting the picture into a sequence of patches — trading the convolution's built-in locality for global, content-based attention that, with enough data, matches or beats CNNs.
← Previous Topic
Computer Vision Foundations
Next Topic →
Image Segmentation