Object detection
So far, we have explored two important computer vision tasks:
- Image Classification: What object is present in the image?
- Object Localization: Where is the object located?
But what happens when an image contains multiple objects, possibly belonging to different classes?
This brings us to Object Detection.
Can We Detect Multiple Objects Using Regression?
In classification and localization, a CNN can predict an object's class and its bounding box.
For example:
Here, represents whether an object is present, represents the box position, while and represent its width and height.
Now imagine an image containing a dog, two cats, and a duck.
We could ask the network to predict:
DOG → (x₁, y₁, w₁, h₁)
CAT → (x₂, y₂, w₂, h₂)
CAT → (x₃, y₃, w₃, h₃)
DUCK → (x₄, y₄, w₄, h₄)
This seems straightforward. But there is a problem.
How does the network know how many objects to predict?
If there are four objects, it needs four bounding boxes. If there are twenty objects, it needs twenty.
A conventional regression head produces a fixed number of outputs, while the number of objects in an image can vary.
What If We Predict a Fixed Maximum Number of Objects?
Suppose we design a network that can predict at most 10 objects.
Each prediction contains:
- An object class
- Four bounding-box coordinates
- An object-presence score
If an image contains only three objects, the remaining seven predictions must represent no object.
During training, we can assign special background or empty labels to these unused prediction slots.
But this approach introduces several challenges.
Problem 1: Variable Number of Objects
An image may contain one object, five objects, or hundreds of objects.
A fixed output size places a limit on how many objects the network can detect.
Problem 2: Multiple Objects of the Same Class
Imagine an image containing twenty cats.
Predicting the label "CAT" is not enough.
The network must identify each individual cat and predict a separate bounding box.
This becomes especially difficult when objects overlap or appear close together.
Problem 3: Assigning Predictions to Objects
Suppose the network predicts ten bounding boxes, but the ground truth contains only four objects.
Which prediction should correspond to which object?
Unlike ordinary classification, object detection requires matching predictions with ground-truth objects.
This makes training more complicated.
Key intuition: Object detection is not simply classification with four additional regression values. It must also handle an unknown number of objects and their locations.
What If We Search the Entire Image?
Instead of predicting all objects simultaneously, consider another approach.
What if we examine different parts of the image and ask:
Does this region contain an object?
This leads to the idea of a sliding window detector.
Sliding Window Approach
Imagine placing a rectangular window over an image.
We move the window across different positions and use a classifier to determine whether it contains an object.
Why different scales?
Because objects can appear at different sizes.
A nearby dog might occupy most of the image, while a distant dog might occupy only a small region.
Therefore, the detector must examine windows of different:
- Positions
- Sizes
- Aspect relationships
Why Is Sliding-Window Detection Expensive?
For an image of size , we may evaluate multiple window sizes and aspect ratios at every pixel position.
The approximate number of candidate windows is:
where is the number of window shapes tested per position.
If each window requires a separate CNN classification:
where is the cost of one CNN classification.
Example: A image with 5 window shapes per position produces approximately 1.25 million candidate windows, potentially requiring 1.25 million CNN classifications.
The limitation: Most windows contain background or overlapping regions, resulting in substantial redundant computation.
This motivates region proposals, which identify a smaller set of promising regions instead of examining every possible window.
Can We Search Only Promising Regions?
Instead of checking every possible window, consider a smarter approach.
What if we could first identify a small subset of image regions that are likely to contain objects?
For example, in an image containing a dog and a cat, we might generate candidate regions around:
- The dog
- The cat
- A flower
- A group of nearby objects
We would then classify only these selected regions.
This is the motivation behind Region Proposals.
5. What Are Region Proposals?
A region proposal is a candidate area of an image that might contain an object.
The proposal algorithm does not initially need to know whether the region contains a dog, cat, car, or person.
It only tries to answer:
Does this region look like it might contain an object?
This is called a class-agnostic approach.
A traditional proposal-based detection pipeline looks like this:
Input Image
↓
Generate Region Proposals
↓
Extract Candidate Regions
↓
CNN Feature Extraction
↓
Classify Each Region
↓
Refine Bounding Boxes
↓
Final Detections
For example, instead of evaluating tens of thousands of sliding windows, a proposal method might generate around 2,000 promising regions.
The classifier can focus on these candidates rather than exhaustively searching the entire image.
This is the central idea behind early region-based object detectors such as R-CNN.
6. How Do We Generate Region Proposals?
An important observation is that objects often form visually coherent regions.
For example:
- A cat's fur may have similar colors and textures.
- A car may have distinct edges and surfaces.
- A person may form a connected region with recognizable boundaries.
This suggests that we can use visual properties to identify candidate object regions.
Does Color Help?
Yes. Color can be a useful clue.
Neighboring pixels with similar colors may belong to the same object or surface.
However, color alone is not sufficient.
Two different objects may have similar colors, while one object may contain many colors.
Therefore, traditional region proposal methods often combine several visual cues:
- Color similarity: Regions with similar color distributions
- Texture similarity: Regions with similar patterns
- Region size: Preference for plausible combinations of regions
- Spatial compatibility: Whether neighboring regions fit together
These properties help identify groups of pixels that might belong to objects.
7. Selective Search: A Smarter Way to Find Regions
One influential traditional region proposal algorithm is Selective Search.
Instead of evaluating every possible rectangular window, Selective Search begins by dividing an image into many small regions.
It then progressively merges similar neighboring regions.
Step 1: Initial Segmentation
The image is divided into small, visually coherent regions.
These regions are often called superpixels or initial segments.
Step 2: Compare Neighboring Regions
The algorithm examines nearby regions and measures their similarity using properties such as color, texture, size, and spatial compatibility.
Step 3: Merge Similar Regions
Similar neighboring regions are merged to form larger regions.
This process is repeated, creating candidate regions at multiple scales.
Step 4: Convert Regions Into Bounding Boxes
Each candidate region is enclosed by a rectangular bounding box.
These bounding boxes become the region proposals.
Original Image
↓
Small Image Regions
↓
Merge Similar Regions
↓
Larger Candidate Regions
↓
Bounding Box Proposals
Why Merge Regions at Multiple Scales?
Objects can be small or large.
A small region might represent a cat's face, while a combination of several regions might represent the entire cat.
By merging regions at different scales, Selective Search generates proposals for objects of different sizes.
Importantly, these are only candidate regions. Some may contain complete objects, while others may contain object parts or background.
A later classifier determines what each region actually contains.
8. Why Region Proposals Were Important
Region proposals provided a practical alternative to exhaustive sliding-window search.
Rather than asking a CNN to inspect every possible region, proposal-based detectors could first identify promising locations.
This made it possible to focus expensive classification computations on a smaller number of candidates.
However, early region-based methods still had limitations.
- Proposal generation could be slow.
- Multiple proposals could overlap heavily.
- R-CNN processed candidate regions separately, resulting in substantial repeated CNN computation.
- Proposal algorithms could miss objects entirely.
These limitations motivated more efficient architectures, including Fast R-CNN and Faster R-CNN.
Key Takeaways
- Object detection must predict both what objects are present and where each object is located.
- The number of objects varies between images, making naive fixed-output regression challenging.
- Sliding windows search many positions and scales but can be computationally expensive.
- Region proposals reduce the search space by identifying candidate object locations.
- Selective Search generates proposals by progressively merging visually similar regions.
- Region proposals laid the foundation for early region-based CNN detectors.
The central intuition: Instead of searching everywhere, first identify where objects are likely to be, and then focus the classifier on those regions.
This leads naturally to our next topic: R-CNN — Region-Based Convolutional Neural Networks.