Instagram System Design

Instagram is a globally distributed, media-heavy social platform optimized for content creation, engagement, and extremely fast feed consumption. The system must balance write-heavy workloads with ultra-low-latency reads at massive scale.

Functional Requirements

The system must support the following functional requirements:

• Users can create posts (images / videos).
• Users can like posts.
• Users can comment on posts.
• Users can follow / unfollow others.
• Users can view:
  • The home timeline (personalized feed)
  • A user timeline (profile posts)

Non-Functional Requirements

The key system constraints are:

• Eventual consistency is acceptable for writes.
• Feed generation must be extremely fast.
• The system must be highly available.
• Storage must be durable and persistent.
• The system must support hot vs. cold data.
• It must handle multiple classes of users.
• It must operate at global scale.

User Behavior Diversity

Different classes of users influence the architecture:

• Famous Users → Users who have millions of followers
• Active Users → Frequent consumers
• Live Users → Require real-time updates
• Passive Users → Rarely open the app
• Inactive Users → No optimization needed

The system's behavior must adapt to each type of user.

Scale Estimation

Assumptions:

• 2 billion monthly active users
• 1 billion daily active users
• 500 million posts per day

For simplicity, assume:

1 day ≈ $10⁵$ seconds (rounded for mental math)

POST TPS (transactions per second):

$\frac{500 \;million}{10^5}$ posts per sec = $\frac{500 \times 10^6}{10^5}$ posts per sec = posts per sec

Engagement TPS:

Assume that each user likes 10 posts and comments on 3 posts (on average).

$\frac{1\; Billion \times 10} {10^5}$ likes per sec = $\frac{10^9 \times 10} {10^5}$ likes per sec = 100k likes per sec

$\frac{1\; Billion \times 3} {10^5}$ comments per sec = $\frac{10^9 \times 3} {10^5}$ comments per sec = 30k comments per sec

Feed Read QPS

Reads vastly outnumber writes. Assume 20 feed requests per user per day.

$10^9$ users×20=$2×10^{10}$ feed requests/day

$\frac{2×10^{10}}{10^5}$ feed requests/sec = 200,000 feed reads/sec

If we also consider a peak QPS of 3× to 5× the average QPS => $200K×5=1M$ feed reads/sec, which is huge.

Core Entities

The primary data models are:

• User
• Post
• Like
• Comment
• Media / Asset

Each entity has distinct scaling and storage needs.

APIs

Post Creation

POST /posts
{
  caption,
  mediaUrl,
  ...
}

Media upload should use a presigned URL, so that the client uploads directly to blob storage:

Client → Blob Storage (direct upload)

This avoids backend bottlenecks.

Engagement APIs

POST /likes/{postId}
POST /comments/{postId}
POST /follow/{userId}
POST /unfollow/{userId}

Timeline APIs

GET /timelines          → Home feed
GET /timelines/{userId} → User profile feed

High-Level Design

Instagram’s architecture separates responsibilities into independent services to support massive scale, high availability, and low-latency reads.

Consider the different responsibilities:

• User Creation → identity-heavy
• Follow/Unfollow Users → graph network
• Post Creation → write-heavy
• Timeline / Feed → read-heavy and latency-critical

So let's create separate microservices for these different responsibilities. Databases and other storage integrations are handled within each service boundary, allowing each service to optimize its own storage and delivery strategies.

API Gateway

Role: The entry point for all client interactions.

Responsibilities

• Routes requests to internal services
• Handles authentication / authorization
• Applies rate limiting & throttling
• Centralizes logging & monitoring

The gateway prevents clients from accessing backend services directly and simplifies request management.

User Service

Owns: The user identity and profile domain.

Responsibilities

• User creation & updates
• Profile retrieval
• Account metadata management
• User-related validations

A relational database or a strongly consistent key-value store is the right choice for the User Database.

This service is accessed frequently by nearly all system components, making it a foundational dependency.

Follow Service

Owns: Social graph relationships.

Responsibilities

• Follow / unfollow operations
• Fetching followers & followees
• Maintaining user relationship edges

A wide-column database (Cassandra) or a graph-optimized store is the best choice for the Follow Database, because the connections form a graph-type network.

The follow graph is a core driver of feed generation and requires high scalability and efficient querying.

Post Creation Service

Owns: The post write workflow.

Responsibilities

• Validating post payloads
• Handling post creation requests
• Storing post images/videos in blob storage, from where they can be pulled into the CDN (content delivery network)
• Persisting post metadata (caption, description, creator_id, etc.) in a metadata database

Post creation is a write-heavy operation and must scale independently of feed reads.

A distributed NoSQL database with high write throughput is the best choice for post metadata because of the continuous high-volume writes.

Timeline / Feed Service

Owns: Feed generation and retrieval.

Responsibilities

• Generating the user's home timeline
• Fetching relevant posts
• Ranking & ordering feed items
• Serving low-latency responses

Feed retrieval is the most latency-sensitive and read-heavy path in the system.

A cache-first design (Redis / Memcached) with a backing store is the right approach for the Timeline / Feed Service.

Deep Dive - Post Creation:

Media Processing & Multi-Resolution Support

In media-heavy systems like Instagram, storing a single uploaded image/video is insufficient. Different devices, screen sizes, and network conditions require multiple optimized variants of the same media. This introduces additional components into the post creation workflow. The system never serves the raw original directly to most users.

The optimized pipeline:

Client Upload → Post Ingestion Service → Blob Storage (Original) → Media Processing Pipeline → Blob Storage (Variants) → CDN

Media Processing Service (Critical Component)

Responsibilities

• Generating multiple resolutions (thumbnail, medium, high)
• Resizing & compressing images
• Transcoding videos into adaptive formats
• Optimizing for bandwidth & device constraints

Why it is needed:

• Different devices require different sizes.
• It reduces payload size & latency.
• It improves the user experience.
• It saves CDN bandwidth costs.

Example variants:

• Thumbnail (low resolution)
• Feed version (compressed)
• High-resolution viewer version

Async Processing Trigger

When media is uploaded:

Post Ingestion Service → Emit MediaUploaded Event

The Media Processing Service consumes these events and:

• Fetches the original media
• Generates variants
• Stores the derived assets

This avoids adding to post creation latency.

Blob Storage

Blob storage now holds:

• Original media
• Processed variants
• Device-optimized formats

Consider these variants:

/media/{postId}/original
/media/{postId}/thumbnail
/media/{postId}/medium
/media/{postId}/high

CDN Integration

The CDN sits in front of blob storage. If media is not found in the CDN, it is pulled from blob storage.

Client → CDN → Blob Storage

Benefits:

• Edge caching
• Low-latency global delivery
• Offloading origin traffic

The CDN primarily serves processed variants.

Metadata Implications

The Post Metadata Store now keeps:

• Media identifiers / keys
• URLs for variants
• Media type & attributes

Example:

{
  postId,
  media: {
    thumbnail_url,
    medium_url,
    high_url
  }
}

Timeline Generation

Timeline generation is one of the most critical read paths in Instagram.
A simple design often works at small scale but quickly collapses under real-world traffic.

Below are the intuitive naïve strategies for both home and user timelines.

User Timeline

A simpler naïve strategy:

1. Query the Post Store by creator_id
2. Sort by timestamp
3. Paginate the results

Why Does This Work Better?

• Single-partition access pattern
• Predictable query cost
• No cross-user aggregation
• Naturally cacheable

The user timeline is fundamentally easier than the home timeline.

Home Timeline - Naïve Approaches

A straightforward approach:

1. Fetch the users followed by User A from the Follow DB
2. For each followed user:
   • Query the Post Store for recent posts
3. Aggregate the posts
4. Sort by creation time
5. Return the top N results

Although this approach is simple, it fails at scale because of explosive fan-out reads.

If User A follows 1,000 users:

• 1 Follow query
• 1,000 Post queries

Latency and database pressure explode, and the slowest dependency determines the response time. This direct aggregation approach becomes extremely expensive.

Even a few slow queries → feed delays.

How should we shard the post table?

If User A follows users B, C, and D, we need the posts from users B, C, and D.

• Should we shard posts by post ID or by user ID?
• We don't know the post IDs; we need the posts of all these users. So let's shard based on user ID.
• But then we have to aggregate the data.

Will caching posts solve the problem?

No. Even if posts are cached, the system still performs fan-out reads.

Alternative Naïve Strategy: Full Scan + Filter

Another bad but intuitive idea:

1. Scan recent posts globally
2. Filter posts from followed users
3. Sort & limit
4. Cache

Why is this worse?

This approach is inefficient because it requires scanning a massive global dataset for every feed request, leading to extreme read amplification and wasted computation. Caching does not fix the problem, because the system must still perform the expensive global scan before knowing what to cache, and personalized feeds have low cache reuse.

Timeline Generation Deep Dive: Solution

Populate Feed Cache on Write (Fan-Out on Write)

Instead of generating timelines at read time, the system shifts the work to the write path.

When a user creates a post:

• Identify their followers from the Follow Service
• For each follower:
  • Insert the post_id into their feed cache

Result:

• Feed reads become simple lookups.
• There is no expensive aggregation during read requests.
• Timeline retrieval has predictable low latency.

This converts read amplification into controlled write amplification, which is far more scalable for feed-heavy systems.

Problem: High-Follower Users & Write Amplification

With fan-out on write, post creation requires updating the feed cache of every follower.

For highly popular users, this means:

• Millions of followers
• Millions of feed updates per post
• Increased latency on the write path
• A risk of system overload

Synchronous updates become impractical.

Solution: Asynchronous Fan-Out

Shift feed updates to an asynchronous pipeline.

The Post Ingestion Service:

• Persists the post metadata
• Emits a PostCreated event to the Post Topic

Background workers / consumers:

• Process the follower fan-out
• Update feed caches independently
• Retry on failures

Benefits:

• Write latency remains low.
• The workload is spread across workers.
• Traffic spikes are prevented from blocking requests.
• System resilience improves.

Asynchronous processing decouples user-facing latency from heavy fan-out computation.

Hybrid Approach

Famous Users (High-Inflow / Influencers)

• Read-driven fan-out.
• Posts are not pushed to followers' caches on write. Instead, the user's followers pull these posts into their timelines during feed generation at read time. This prevents a massive write bottleneck.

Active Users (High Frequency)

• Write-driven fan-out (active within the last 30 days).
• Posts are eagerly pushed to populate the feed caches of their active followers immediately upon creation.

Live Users (Real-Time Engagement)

• Write-driven push with push notifications.
• Posts instantly populate the feed caches of online followers. Simultaneously, a notification server delivers a live payload directly over established WebSockets or Server-Sent Events (SSE).

Passive Users (Low Frequency)

• Selective/deferred fan-out (inactive for 30+ days).
• The system skips active feed cache generation on write. Caches are populated lazily, only if and when the user logs back in.

Inactive Users (Dormant)

• No fan-out.
• These users are excluded completely from all write-path cache generation pipelines to optimize storage resources.

How Do We Inform Live Users?

Live users need real-time awareness of new posts without waiting for a feed refresh.

Flow:

1. A post is created → the Post Ingestion Service emits an event.

2. Post workers process the fan-out / feed updates.

3. The timeline cache is updated.

4. A notification event is pushed to the WebSocket Manager.

5. The WebSocket Manager has a mapping of WebSocket Handlers -> connections (stored in a cache).

6. The WebSocket Handler delivers the update to active connections.

Result:

✔ Connected users instantly see new posts

✔ No polling / refresh is required

✔ A low-latency user experience

Hot and Cold Posts:

Separating hot and cold data is essential in Instagram because of extreme read skew.

Most content access is concentrated on:

• Recent posts
• Popular posts
• Frequently viewed media

Older content is rarely accessed.

New Post → Hot Tier (cache / fast DB)

→ Gradual Demotion

→ Cold Tier (cheap storage)

Promotion/demotion is driven by:

• Access frequency

• Recency

• Engagement signals

An Archival Service fetches old/cold posts from the post database and moves them to an archival database.