URL Shortener System Design

A URL shortener is a service that takes a long, complex URL and converts it into a short, easy-to-share link. For example:

• https://www.example.com/products/category/item?id=12345&ref=abcd becomes → https://bit.ly/xyz123

Common URL shortening services include:

• Bitly — bit.ly/abc123
• TinyURL — tinyurl.com/xyz789
• Twitter’s t.co (used for all outbound links)
• Firebase Dynamic Links (used in mobile apps)

These services are useful for social media, messaging, QR codes, and marketing campaigns. Despite appearing simple, building a scalable URL shortener requires handling billions of redirects, ensuring high availability, minimizing latency, and managing storage efficiently.

Functional Requirements

A URL shortener should convert long URLs into short, compact identifiers that can be easily shared. The main functional requirements are:

• Create a unique short URL for a given long URL

  • The service accepts a long URL and returns a unique shortened identifier.
  • Optional features may include:
    • Custom alias (e.g., /my-link instead of a random ID)
    • Expiration time, so that shortened links expire automatically

• Redirect from the short URL to the long URL

  • When a user accesses the short URL, the system should redirect them to the original URL with minimal latency.

Out of scope (for this design):

• Analytics, such as:
  • Click counts
  • Traffic source tracking
  • Geographic or device analytics

Non-Functional Requirements

Beyond basic functionality, the system must satisfy several quality attributes:

• Consistency model

  • Eventual consistency is acceptable for writes (as per the CAP theorem).

• High availability

  • The service should handle failures gracefully and continue serving redirects.

• Low latency

  • Redirects should be very fast because they are user-facing.
  • A typical target is ~200ms or less.

• Scalability

  • The system should support large traffic volumes:
  • 100 million Daily Active Users (DAU)
  • 10 billion redirect requests per day

Entities

The essential conceptual data entities are:

• User Represents an authenticated user who may create or manage short links. (This is optional in a public service.)
• LongUrl Stores the original, full-length URL submitted by a user.
• ShortUrl Stores the shortened identifier and metadata such as:
  • Creation time
  • Expiration time
  • Associated user (optional)
  • Custom alias flag

API Design

The external interface can be exposed through simple RESTful APIs.

Create Short URL

POST /urls     -> 201 Created
Content-Type: application/json

{
  "longUrl": "https://www.example.com/very/long/link",
  "alias": "custom-name",         // optional
  "expirationTime": "2025-01-01"  // optional
}

• The response returns metadata, including the generated short URL.

• The server generates a unique slug if no alias is provided.

Redirect Short URL

    GET {"{"}shortUrl {"}"} -> redirecting to long url

• A 301 status indicates a permanent redirect.

• A 302 status indicates a temporary redirect.

• The browser automatically follows the redirect to the target URL.

High-Level Design

At a very high level, the system consists of:

• A primary service layer that creates short URLs and redirects them to long URLs

• A database layer that stores URL mappings

• A caching layer that speeds up frequent redirects

The architecture diagram:

Scale Calculations

To estimate the required capacity:

Daily Active Users (DAU): 100 million

Assume that, on average, each user creates one URL per day.

Also assume that 10B redirects happen per day.

Lifetime storage

100M users * 365 days * 10 years ≈ 365 billion URLs

Request Rates

URL creation throughput: 100M / (24h * 3600s) ≈ 1000 TPS (transactions per second)

Redirect throughput: 10B redirects/day ≈ 100k QPS (queries per second)

Storage Consumption

Assuming an average metadata size of ~500 bytes per entry:

500 bytes * 365B entries ≈ ~200 TB over 10 years

Database Selection

Workload characteristics

• No complex schema or relational joins
• A read-heavy pattern (dominated by redirects)
• Eventual consistency is acceptable

SQL databases (PostgreSQL, MySQL)?

• ✔ Strong consistency
• ❌ May struggle at large scale without sharding

NoSQL databases (DynamoDB, Cassandra)?

• ✔ Horizontal scalability
• ✔ Flexible schema
• ✔ High throughput

Conclusion:

NoSQL is typically preferred for large-scale URL redirect systems.

Service Decomposition: URL Creation vs. Redirect Service

In large-scale URL shortening systems, it is common to split the architecture into two primary services:

• URL Creation Service
• Redirect Service

This decomposition is not accidental — it addresses performance, scaling, consistency, and cost concerns.

1. Different Workload Characteristics

URL Creation

• Low QPS (e.g., hundreds or thousands per second)
• Write-heavy (inserting records and metadata)
• Latency is important but not ultra-critical
• Requires validation, authentication, rate limiting, etc.

Redirect

• Extremely high QPS (could be millions or more per second)
• Read-heavy (lookup and redirect)
• Must have extremely low latency (ideally sub-10ms)
• Often unauthenticated public traffic

By separating these, each service can scale according to its own access pattern instead of over-provisioning a single monolith.

2. Independent Scaling & Cost Optimization

Redirect traffic is massive. If both workloads lived in a monolith:

• Scaling to support the redirect QPS would force the creation logic to scale too.
• This drives up compute and network costs unnecessarily.

With decomposition:

• The Redirect Service can use cheap, horizontally scalable caching layers (CDN, Redis, in-memory caching).
• The URL Creation Service can run smaller, since it sees minimal traffic.

The result is better cost efficiency and operational flexibility.

3. Separation of Critical Paths & Latency

Redirects define the latency SLA of the product:

• Users expect pages to load instantly.
• Adding synchronous metadata writes or validation would slow redirects down.
• Caching can push latency down to microseconds or edge-level delivery.

Meanwhile, URL creation can afford minor overhead, since it is not on the end users' hot path.

4. Different Consistency Requirements

This aligns with typical distributed system strategies:

• Write path → stronger guarantees
• Read path → optimized and cached

5. Deployment & Infrastructure Flexibility

Splitting the services also gives us deployment advantages:

• The Redirect Service can be deployed across CDN edges, close to users.
• The Creation Service can live behind private APIs with stronger authentication.
• Database writes can be centralized.
• Database reads for redirects can be replicated globally.

This enables:

• Geo-distributed read replicas
• Edge caching
• Reduced cross-region latency

In short: URL Creation writes data reliably, and Redirect reads it fast — these are fundamentally different jobs, so they deserve fundamentally different services.

How to Decide the Short URL Size?

To prevent collisions, we need to support roughly 365 billion short URLs over the lifetime of the system.

Let us explore possible character set sizes and URL lengths:

Base-10 (digits only)

10^6 ≈ 1 million combinations (considering 6 char short url)

10^7 ≈ 10 million combinations (considering 7 char short url)

➡️ This is far too small compared to hundreds of billions.

Base-62 (0–9, A–Z, a–z)

62^6 = 56 billion (considering 6 char short url)

62^7 = 3.52 trillion (considering 7 char short url)

Hence, Base-62 encoding with 7-character short URLs is a popular choice.

How to Create a Short URL

There are multiple techniques for generating a short identifier (slug) from a long URL. Each technique has different trade-offs around uniqueness, collisions, and operational overhead.

• Random 7-character Base62 key

  • The system generates a random key of fixed length (e.g., 7 characters).
  • Characters are selected from a Base62 set: 0-9, A-Z, a-z.
  • This provides a large key space (62^7 ≈ 3.52 trillion), making collisions unlikely.
  • It requires collision detection logic (e.g., retrying if the generated key already exists).

• Hashing approach

  • Compute a hash of the long URL (e.g., SHA-256, MD5).
  • Take a subset of the hash bits and Base62-encode them.
  • Advantages:
    • The same long URL always produces the same short URL (idempotent).
  • Disadvantages:
    • Hash collisions are still possible and must be handled.
    • Full hash values are too long, so truncation is required.

• Counter-based approach

  • Maintain a global incrementing counter.
  • For each new URL, increment the counter and Base62-encode the integer value.
  • This guarantees uniqueness without collision checks.
  • It is commonly used in distributed systems when combined with sharded counters.

What Type of Database Should Store the Counter?

If you use a counter-based URL generation strategy, you need a datastore that can safely store and increment the counter. Options include:

SQL Databases

• They provide AUTO_INCREMENT (MySQL) or SEQUENCE (PostgreSQL).
• Advantages:
  • Simple to set up and reason about.
  • Strong consistency and ACID guarantees.
• Disadvantages:
  • Harder to scale horizontally for very high write throughput.
  • Write contention can become a bottleneck at large scale.

NoSQL Databases (e.g., Cassandra, DynamoDB)

• They support distributed storage and high write throughput.
• Advantages:
  • Horizontally scalable architecture.
  • A good fit for distributed workloads.
• Disadvantages:
  • No built-in auto-increment.
  • Increment logic must be implemented manually.
  • Eventual consistency may lead to counter conflicts if not carefully managed.

Redis

• It provides the INCR and INCRBY atomic operations.
• Advantages:
  • Extremely fast (in-memory).
  • Atomic increments ensure uniqueness without race conditions.
  • It can be sharded for scalability.
• Disadvantages:
  • Because storage is in-memory, durability must be configured (e.g., AOF snapshots).
  • Failover must be handled to avoid a single point of failure.

Redis is often used for short-URL counters because it provides atomic increments and low-latency, high-throughput number generation, with a simple model that maps cleanly to Base62 encoding.

Redis as a Single Point of Failure

If we rely on Redis as the only component responsible for generating and storing counters, it becomes a single point of failure.
If that Redis instance crashes, becomes unavailable, or loses data, the entire URL generation workflow is affected.
In the worst case, the system would stop generating new short URLs until Redis is restored.

Mitigation Strategy: Sharded Counter Space

One common approach is to avoid a single global counter and instead shard the counter space across multiple Redis servers.

Each Redis instance manages a dedicated integer range:

Server 1 → 0 – 1,000,000

Server 2 → 1,000,000 – 2,000,000

Server 3 → 2,000,000 – 3,000,000 and so on.

With this approach:

• If Server 1 fails, only the range 0 → 1M is temporarily unavailable.
• The other servers continue issuing IDs from their ranges.
• There is no global lock or contention on a single instance.
• New shards can be added by allocating new ranges.

Redis Counter Sharding Architecture

• ID Shard #1 is down → it cannot issue IDs in the range 0–1M.
• Shards #2 and #3 continue serving requests.
• The application remains partially operational.

Operational Considerations

Sharding improves resilience, but it introduces new responsibilities:

• Tracking which ranges are allocated and which are free
• The risk of range exhaustion (a shard may run out of numbers)
• Assigning new counter ranges to servers
• A graceful handover if a shard fails mid-range
• Client or service logic that decides which shard to ask for the next ID