Common Computer Vision Problems

We understand much of the world by looking. We recognize people, read signs, notice obstacles, and see when something has changed. Computer vision is the field of AI that helps computers use images and videos to make sense of what they see.

Why does this matter today?

Cameras are everywhere, and they produce far more visual information than people can examine one image at a time. Computer vision can help doctors review medical scans, help drivers notice hazards, spot defects in factories, make photos searchable, and describe surroundings to people with visual impairments.

Each of these uses begins with a human need. In this course, we will explore the questions computer vision can answer, the ideas behind its methods, and where its answers can be useful or unreliable.

One Scene, Many Questions

Consider a picture of a girl playing with toys. We can ask several different questions about the same image:

• What is this image about?
• Where is the girl?
• What objects are present, and where is each one?
• Which pixels belong to the girl, the toys, and the background?

The question determines the kind of answer a computer vision system must produce.

Image Classification

Image classification assigns one or more labels to an entire image. For example, the picture above might receive the label child playing. The picture below is a picture of a Dog. Training labels are image-level categories. The right label can be ambiguous when several objects appear or the background dominates.

Object Localization

Localization gives the position of one target object, usually as a bounding box. The output may include a class label and four box coordinates, such as the top-left and bottom-right corners. It answers “Where is the dog?” when the target is specified, or “What is the main object and where is it?” in a single-object setting.

Object Detection

Detection finds multiple objects in one image. For each object, the model predicts a class, a bounding box, and usually a confidence score. In the girl-and-toys scene, it might return boxes for the girl, the teddy bear, and the toy car. The number of detections can vary from image to image.

Image Segmentation

Segmentation identifies regions at the pixel level. Three common forms answer slightly different questions:

• Semantic segmentation assigns a class to each pixel. Two touching people can both be labeled person without being separated.
• Instance segmentation separates individual objects of the same class, such as two different people.
• Panoptic segmentation identifies individual countable objects and also labels other regions, such as the floor or the sky.

Countable objects such as people and cars are often called things. Broad, less clearly bounded regions such as sky and road are often called stuff.

Other Important Vision Problems

Keypoints, Landmarks, and Pose Estimation

A keypoint model predicts important coordinates, such as human body joints, facial landmarks, or corners of an object. Pose estimation uses these points to describe position and orientation. A person can have a bounding box and still need a separate skeleton of head, shoulder, elbow, and wrist points. Annotation requires landmark coordinates, often with visibility information. Evaluation may use keypoint similarity or distance thresholds.

Tracking in Video

Tracking links the same object across consecutive frames. A detector can find a person in every frame, but a tracker must decide which new detection belongs to the same person seen earlier. The output is a sequence of positions with a persistent identity. Tracking must handle motion, temporary occlusion, and objects entering or leaving the scene. Training and evaluation may use frame-level boxes or masks together with identity labels.

In the tracking example, ID 1 follows the woman in red as she moves behind a pillar and reappears. The person entering in blue receives a separate ID 2.

Optical Flow and Depth Estimation

Optical flow estimates the apparent motion of image points or pixels between frames, usually as a two-component displacement field. Depth estimation predicts how far scene points are from the camera, often as a dense depth map. Both produce spatially aligned numerical outputs rather than categories. Flow helps analyze motion; depth supports 3D scene understanding. From a single image, absolute depth is often uncertain unless the method or data provide scale information.

Text Recognition and Document Understanding

Optical character recognition, or OCR, turns text in an image into machine-readable characters. Some systems first locate text regions and then recognize their contents. Document understanding also uses the layout and relationships among fields, tables, and paragraphs. For example, reading a paystub involves recognizing its text and connecting each amount to the correct field.

Image Restoration and Generation

Restoration changes an image while preserving its intended content. Examples include denoising, deblurring, and super-resolution. Generation creates a new image from a prompt or another input; image editing modifies selected content. These tasks produce images rather than labels or coordinates. Assessing quality requires attention to visual fidelity and the requirements of the application; one score rarely captures every aspect.

In the illustration, restoration repairs a damaged lighthouse photo, while generation creates a new greenhouse image from a text prompt.

Image Captioning and Visual Question Answering

Captioning produces a sentence about an image. Visual question answering produces an answer based on both an image and a question. These tasks connect visual perception with language and may require object recognition, understanding relationships, reading text, and reasoning. A plausible-sounding description can still be wrong about what the image actually shows.

The captioning example places a photograph next to a sentence describing it.

The visual question answering example uses a photograph to answer a specific question about a yellow umbrella.

Medical Image Analysis

Computer vision can help analyze X-rays, CT scans, MRI scans, retinal photographs, and microscope images. A model might highlight a suspicious region, outline a tumor, or compare scans taken at different times. These tasks can help clinicians examine detailed images and measure changes over time.

Clockwise from the top left: brain CT detection, retinal screening, tissue segmentation, and mammogram lesion detection. The highlighted regions are illustrative.

Choosing the Right Task

Key Takeaways

Computer vision is a family of problems, not a single task.

• Classification names the image;
• Localization places one object;
• Detection finds multiple objects;
• Segmentation traces regions;
• Tracking preserves identities over time.
• Keypoints, depth, motion, text recognition, and image generation answer other questions about the same visual world.

Start by deciding what answer you need from the image or video.