Object Localization: Finding What and Where

Image classification answers one question:

What is present in the image?

Object localization answers two questions:

What is present, and where is it located?

For example, an image-classification model may say that an image contains a dog. An object-localization model goes one step further: it predicts that the image contains a dog and draws a bounding box around it.

Object localization normally assumes that there is one main object in the image. Detecting several objects is a more general problem called object detection.

An Intuitive First Attempt: Divide the Image

Suppose we already have an image-classification model that can recognize a cat. How might we use it to find the cat's location?

One simple idea is to divide the image into two parts. We can classify each part separately:

Left half  → Dog
Right half → No Dog

We can then conclude that the cat is probably located in the left half. This converts localization into repeated classification.

Two regions provide only a rough location. We could divide the image into four regions:

┌─────────────────┬─────────────────┐
│    Top-left     │    Top-right    │
├─────────────────┼─────────────────┤
│   Bottom-left   │  Bottom-right   │
└─────────────────┴─────────────────┘

The classifier is applied independently to every region. If the top-right region receives the highest Dog probability, we treat that region as the predicted location.

We could continue dividing the image into smaller and smaller regions to obtain a more precise answer.

However, this approach quickly becomes inefficient:

• many regions must be classified,
• the model repeats similar calculations,
• large and small objects require different region sizes, and
• the region boundaries may cut through an object.

What If the Object Is in the Middle?

The divide-and-classify method has a fundamental problem: an object does not have to fit neatly inside one predefined region.

Consider an object that lies across the middle of the image. Neither half contains the complete cat. The classifier may fail on both halves because each crop contains only part of the object.

The same problem occurs when an object:

• crosses the boundary between two regions,
• is much smaller than a region,
• occupies most of the image,
• appears at an unusual position, or
• has a shape that does not match the rectangular division.

Using overlapping windows can reduce the problem, but then the number of regions grows dramatically.

This leads to an important conclusion:

Fixed image divisions are too rigid for accurate object localization.

The model should predict the object's boundaries directly.

Representing the Location with a Bounding Box

A bounding box is usually represented using four numbers.

One common representation is:

where:

• $x_c$ is the horizontal coordinate of the box centre,
• $y_c$ is the vertical coordinate of the box centre,
• $w$ is the width of the box, and
• $h$ is the height of the box.

Another common representation uses the two corners:

The coordinates are often normalized relative to the image dimensions, producing values between 0 and 1.

For example:

This says that the box centre is near the centre of the image, its width is 40% of the image width, and its height is 60% of the image height.

What Is the Input to an Object-Localization Model?

The input is still an image tensor.

For example, an RGB image may have the shape $H \times W \times C$:

The difference lies in the desired output. Instead of predicting only a class, the model predicts:

• a class label or class probabilities, and
• four bounding-box coordinates.

One Network with Two Output Branches

A localization model can use a shared CNN backbone to extract features from the image. The extracted features are then sent to two separate branches.

Classification Branch

The classification branch predicts what the object is (use AlexNet, VGGNet, GoogleNet etc.)

For a multiclass problem, its output may look like:

Cat   0.91
Dog   0.05
Bird  0.04

A softmax activation is commonly used when exactly one class is correct.

Localization Branch

The localization branch predicts the bounding-box coordinates:

Bounding-box coordinates can take many continuous values like: 0.21, 0.47, 0.63, 0.58

The location branch does not select one box from a fixed list. It estimates numerical coordinates. These are continuous numerical values, so predicting them is a regression problem.

The complete architecture can be summarized as:

The two branches share the visual features learned by the CNN. The classification branch learns which features identify the object, while the regression branch learns how those features relate to its position and size.

Do We Need Annotations?

Yes. If we want the model to learn exact object locations directly, the training data must normally include bounding-box annotations.

For each training image, an annotation may contain:

Class: Cat, Bounding box: $(x_c, y_c, w, h)$

For example:

image_001.jpg → Cat, (0.52, 0.46, 0.38, 0.61)
image_002.jpg → Dog, (0.41, 0.55, 0.48, 0.50)

The class label teaches the classification branch. The bounding-box coordinates teach the localization branch.

Weakly Supervised Localization

In weakly supervised localization, the training data contains image-level class labels but does not contain bounding boxes.

The model knows that a cat appears somewhere in the image, but it is not told where the cat is.

It may try to infer the location from the image regions that contribute most strongly to the classification decision. Techniques such as class-activation maps can highlight influential spatial regions.

Weak supervision has an important advantage:

• class labels are much cheaper and faster to collect than bounding boxes.

However, it also has limitations:

• the predicted region may cover only the most distinctive part of the object,
• the model may focus on contextual clues rather than the object,
• boundaries are often imprecise, and
• evaluation is generally worse than with box supervision.

For example, a classifier may recognize a dog primarily from its face. The highlighted area may contain only the face rather than the dog's complete body.

Weak supervision tells the model what is present, but not exactly where it is.

Fully Supervised Localization

In fully supervised localization, every training image includes both:

• the object class, and
• the correct bounding box.

The network can directly compare its predicted coordinates with the annotated coordinates. This provides a much clearer training signal for learning object position and extent.

Fully supervised localization usually produces more accurate boxes, but annotation is more expensive because someone must draw or verify the box around every object.

Training the Two Branches

The model produces two kinds of error:

1. Classification error: Was the predicted class correct?
2. Localization error: How different was the predicted box from the true box?

The total loss combines both:

where:

• $L_{\text{class}}$ is the classification loss,
• $L_{\text{box}}$ is the bounding-box regression loss, and
• $\lambda$ controls the importance of localization relative to classification.

Cross-entropy is commonly used for classification. Possible box-regression losses include:

• mean squared error,
• L1 loss,
• Smooth L1 loss, and
• IoU-based losses.

During backpropagation, both branches update the shared feature extractor. The network gradually learns features that are useful for recognizing the object and estimating its location.

Measuring Localization Quality with IoU

A predicted box may be close to the correct box without matching it exactly. Intersection over Union, or IoU, measures their overlap.

Intersection over Union (IoU)

Intersection over Union (IoU) measures how closely a predicted bounding box matches the actual object location (ground-truth box).

• Intersection: The overlapping area between the predicted and actual boxes.
• Union: The total area covered by both boxes, counting the overlap only once.
• IoU = 1: Perfect overlap.
• IoU = 0: No overlap.

Example: If the intersection area is 60 pixels² and the union area is 100 pixels²:

An IoU of 0.6 means the boxes have 60% overlap relative to their combined area.

Key takeaway: Higher IoU indicates more accurate bounding-box localization. IoU provides a more meaningful geometric evaluation than checking each coordinate independently.

What If There Is No Object?

Some datasets guarantee that every image contains one object. In real applications, however, an image may contain no target object.

The output can therefore include an object-presence value:

The full output may be represented as:

If no object is present, the model should predict a low objectness probability. The bounding-box loss can then be ignored for that training example because there is no valid box to learn.

Localization Versus Detection

Object localization normally predicts one primary object and one bounding box. Object detection must handle a variable number of objects.

Localization provides the conceptual bridge from image classification to object detection.

Key Takeaways

• Classification predicts what is present.
• Localization predicts what is present and where it is.
• Dividing an image and classifying every part provides an intuitive starting point but fails when objects cross region boundaries.
• A bounding box can be represented using centre coordinates, width and height.
• A localization network commonly uses a shared CNN followed by two branches.
• The classification branch predicts the object class.
• The regression branch predicts continuous bounding-box coordinates.
• Weakly supervised localization uses only image-level class labels.
• Fully supervised localization requires both class and bounding-box annotations.
• Classification and localization losses are optimized together.
• IoU measures the overlap between the predicted box and the ground-truth box.
• Object localization is a stepping stone from image classification to multi-object detection.

The central idea is simple:

Learn the visual features once, then use them to predict both the identity and the location of the object.