Feature Extraction in Computer Vision

Teaching Machines How to See

When we look at a dog, our brain does parallel processing of colors, shapes, edges, textures, motion etc. Our brain has prior knowledge from years of experience, so it knows that dogs have ears, fur, legs, tails and cars have wheels, headlights, windows. So when you see something with four legs + fur + tail and a specific shape, our brain immediately decides that it is a dog (even if it’s partially hidden).

Our brain also uses reasoning & context. If you see something with headlights on a road → then probably a car. If you see something barking in a park → then probably a dog. So context helps disambiguate visual information.

Machines just see numbers or image pixel intensity values

How an Image Is Represented

• An image is a grid of pixels
• Each pixel stores a numerical value
• The grid has height × width dimensions
• In Gray-scale image, each pixel has one value representing intensity (dark → bright). 0 → black and 255 → white

• In color image, each pixel has three values for Red, Green, Blue and Represented as three stacked grids

What Features To Extract From Image?

Machines need to extract features from these pixel values, so that it can identify images. The original images had a lot of information, like the shape of the object, its color, the edges, texture, may be background noise, etc. Models don’t need everything. Often, shape and edges are enough to distinguish objects. Edges highlight structure, hence in most of the cases, we can extract only the shape and the edges from the image.

Manual Feature Extraction techniques in Machine Learning

Earlier computer vision relied on manual features, designed by humans and based on domain knowledge

Two popular methods:

• HOG (Histogram of Oriented Gradients)
• SIFT (Scale-Invariant Feature Transform)

HOG (Histogram of Oriented Gradients):

Consider this is pixel values for one small part of an image.

HOG Calculates Gradients or small change in the x and y direction for each pixel in the image.

Gradient (or change) in the x-direction $G_x= 223 - 13 = 210$ [subtract the value on the left from the pixel value on the right]

Gradient (or change) in the y-direction $G_y= 64 - 30 = 34$ [subtract the pixel value above from the pixel value below]

The Gradient magnitude would be higher when there is a sharp change in intensity, such as around the edges.

Total Gradient Magnitude $=\sqrt{G_x^2 + G_y^2}$ (using Pythagoras' theorem)

Next, we will calculate the orientation (or direction) for the same pixel.

And using this Gradient magnitude and direction it creates a HOG histogram like below.

Now this can also be viewed as applying two filters in x and y direction. Applying a filter means selecting a region of the image around that specific pixel and take sum of element-wise product of the pixel values and the filter values.

x-filter $= [-1 \; 0 \; 1]$

y-filter $=\begin{bmatrix} -1\\ 0 \\ 1 \end{bmatrix}$

Sobel Filter Application to Detect Edges

Now we will apply two 3x3 filters, named Sobel filters to detect horizontal or vertical edges.

Let's say we have this simple image below with a horizontal edge and we see the 3x3 filter moving across the images.

Sobel Filters

Sobel filters approximate gradients.

Sobel-X →

Measures intensity change along the x-direction. Detects vertical edges

Sobel-Y →

Measures intensity change along the y-direction. Detects horizontal edges

Consider the 3x3 blocks of image where all pixel values are same (or almost same -> which means there is no edge). Place the Sobel-y filter onto that block and calculate element-wise product sum. It will be zero or closer to zero.

Now consider a 3x3 blocks of image where there is a horizontal edge -> means pixel intensities change drastically. Place the Sobel-y filter onto that block and calculate element-wise product sum. It will be a higher value.

Filter output:

• Can be negative or positive
• Can exceed the 0–255 pixel range

It Represents the edge strength. It does not represent pixel value. So we will take absolute values and normalize them to visualize

0 → 0 (black) 760 → 255 (white)

So we can visualize the edge as shown below.

Next, Consider one simple image with vertical edge. To detect a vertical edge we need to track pixel intensity changes in the x-direction.

Applying Sobel-X filter we can detect the vertical edge as shown below.

Limitations of Manual Feature Engineering

Although methods such as HOG and SIFT have been highly successful, they have several important limitations.

Human-Designed Features

Researchers must manually determine which features are important for the task.

Limited Flexibility

Features designed for one task may not work effectively for another.

Difficult to Scale

As the complexity and variety of images increase, handcrafted features often become less effective.

Domain Dependence

Successful feature engineering depends heavily on expert knowledge of the problem domain.

These limitations motivated the development of deep-learning approaches that automatically learn useful features directly from data.

How Deep Learning learns features?

Deep learning learns features in an image by applying learnable filters (kernels) to the image through convolution layers. These filters slide over the image and respond strongly to specific patterns, such as edges, corners, or color changes. At first, the filters are random, but during training they are updated using backpropagation so they become good at detecting useful visual patterns. As images pass through deeper layers, the network combines simple features into more complex ones, gradually learning shapes, object parts, and finally complete objects.

• We will have Feature Extraction layers(with filters) and dense neural network layers(task specific)

• Start With Random Filters with random weight values

• Feed an image (Forward-pass)

• Model will output some prediction (not necessarily correct as we have random filters)

• Calculate Loss/Error = CrossEntropy(predicted=Cat, actual=Dog)

• Backpropagation computes how much each filter contributed to the error.

• It calculates gradients like ( stands for filter weights)

• Apply Gradient Descent to learn the filter weight values

What Do Filters Eventually Learn?

Surprisingly, they learn structured patterns:

Early Layers

• edges (vertical, horizontal)
• corners
• simple textures

Middle Layers

• curves
• patterns
• shapes (e.g. wheel, eye, ear)

Deeper Layers

• semantic features (dog face, car wheel, eye shape)

. .

The model discovers these concepts automatically through optimization.

A deep neural network begins with random filters and gradually transforms raw pixel values into increasingly meaningful representations.

The progression typically looks like:

Pixels → Edges → Corners → Shapes → Object Parts → Complete Objects

Key Takeaways

• Computers represent images as numerical pixel values.
• Traditional computer vision relies on handcrafted features such as HOG and SIFT.
• Edge detection is one of the most important forms of feature extraction.
• Sobel filters use image gradients to detect horizontal and vertical edges.
• Deep learning learns filters automatically instead of relying on manually designed features.
• Backpropagation and gradient descent optimize filter weights during training.
• Learned features become increasingly abstract across network layers.
• Modern Convolutional Neural Networks learn a hierarchy of representations, ranging from simple edges to complete objects.

Feature extraction is the foundation of computer vision.

Traditional methods rely on manually designed features, while deep learning learns useful representations directly from data. This allows modern computer-vision systems to achieve near-human-level performance on many tasks.