Distributed URL Shortener

Production-style distributed system with stateless FastAPI services, Redis-backed caching and rate limiting, PostgreSQL persistence, and Nginx load balancing.

Why This Project

This project demonstrates how production-grade backend systems handle scalability, caching, rate limiting, and fault tolerance beyond basic CRUD applications.

Key goals:

• Design a scalable API with clear request/response contracts
• Introduce caching and rate limiting as core system-level concerns
• Support multiple runtime environments (local, Docker, cloud)
• Ensure reliability through automated testing and CI validation
• Provide end-to-end interaction through a lightweight frontend

Key Achievements

• Reduced database load for hot URLs using Redis caching, cutting local redirect latency from ~10–32 ms to 8–11 ms and improving read-heavy scalability
• Built horizontally scalable stateless FastAPI services behind an Nginx load balancer
• Implemented distributed rate limiting across replicas using Redis
• Designed and deployed a distributed system with stateless services and shared infrastructure (PostgreSQL + Redis)
• Deployed the React frontend and FastAPI backend to Render

Live Demo

• Frontend: https://url-shortener-frontend-av1x.onrender.com
• Backend API: https://url-shortener-gfp0.onrender.com/docs

Key Metrics

• Cache hit latency: ~8–11 ms
• Cache miss latency: ~10–32 ms
• Supports horizontal scaling through stateless application replicas
• Redis reduces repeated database queries for hot URLs
• CI: automated backend tests on every push and pull request (GitHub Actions)

Frontend

A lightweight React frontend enables:

• Creating short URLs
• Viewing generated links
• Retrieving click statistics

The frontend communicates with the deployed FastAPI backend via REST APIs, enabling end-to-end interaction with the distributed system.

When the backend returns structured error details, such as rate-limit responses, the frontend surfaces those messages in the UI.

Deployment

• Frontend and backend hosted on Render
• Neon PostgreSQL provides the persistent source of truth in the hosted deployment
• Upstash Redis provides shared caching and distributed rate limiting in the hosted deployment, with graceful fallback when unavailable
• Reverse proxies should preserve X-Forwarded-For so Redis-backed rate limiting can identify individual clients correctly
• Environment-based configuration enables seamless switching between local, Docker, and cloud deployments
• The hosted backend is configured through DATABASE_URL and REDIS_URL

System Design Summary

• Stateless FastAPI services behind an Nginx load balancer
• Stateless design eliminates the need for sticky sessions, enabling seamless horizontal scaling
• PostgreSQL serves as the single source of truth for durability and consistency
• Enforced strong consistency for URL creation using PostgreSQL uniqueness constraints under concurrent requests
• Redis handles shared caching and distributed rate limiting
• Designed for fault tolerance with Redis treated as optional; system falls back to PostgreSQL on cache failures
• Rate limiting uses the first X-Forwarded-For address when requests pass through a reverse proxy or load balancer

Scalability Characteristics

• Stateless services enable horizontal scaling without sticky sessions
• Designed to handle high read throughput workloads by offloading repeated requests to Redis cache
• Shared Redis keeps caching and rate limiting behavior consistent across replicas
• PostgreSQL ensures strong consistency and durability for URL creation under concurrent requests

Design Tradeoffs

• Tradeoff: prioritized read performance via caching while accepting eventual consistency for cached redirect data

Features

• Create short URLs via API (POST /shorten) or through the frontend UI
• Redirect short URLs with GET /{short_code}
• Track click counts with GET /stats/{short_code}
• Cache redirect lookups using Redis
• Enforce request rate limits using Redis-backed distributed rate limiting
• Run the full stack locally using Docker Compose
• Validate backend behavior with pytest, frontend behavior with Vitest, and backend CI checks with GitHub Actions
• Support horizontal scaling via stateless application instances behind a load balancer
• Benchmark cache performance (miss vs. hit latency)

Cache Performance Benchmark

To validate the effectiveness of Redis caching, redirect latency was measured for cache misses (first request) and cache hits (subsequent requests).

Run locally:

python3 scripts/benchmark_cache.py

Optional environment overrides:

BENCHMARK_BASE_URL=http://127.0.0.1:8000 \
BENCHMARK_ORIGINAL_URL=https://www.google.com \
BENCHMARK_HIT_RUNS=20 \
python3 scripts/benchmark_cache.py

Example results:

• Cache miss latency: ~10–32 ms
• Average cache hit latency: ~8–11 ms
• Reduced repeated database queries for hot URLs

While latency improvement is modest in local testing, Redis caching reduces repeated database queries and improves scalability for read-heavy workloads.

Quick Test

Frontend: https://url-shortener-frontend-av1x.onrender.com

Backend API:

curl https://url-shortener-gfp0.onrender.com/health
curl -X POST "https://url-shortener-gfp0.onrender.com/shorten" \
  -H "Content-Type: application/json" \
  -d '{"original_url":"https://www.google.com"}'

Backend Highlights

• Designed RESTful APIs using FastAPI for URL creation, redirection, and analytics
• Implemented PostgreSQL-backed persistence as the durable source of truth for URL mappings and analytics
• Integrated Redis for shared caching and distributed rate limiting with graceful fallback when unavailable
• Containerized and orchestrated multiple application instances using Docker Compose to simulate a distributed environment
• Built automated test coverage with pytest for the backend and Vitest for the frontend
• Configured GitHub Actions CI to run backend tests on every push and pull request
• Introduced Nginx as a load balancer to distribute traffic across multiple FastAPI instances
• Demonstrated a split hosted architecture with Render for app hosting, Neon for PostgreSQL, and Upstash for Redis
• Validated Redis caching effectiveness using benchmark measurements (cache miss vs. hit latency)

Tech Stack

• Python
• FastAPI
• PostgreSQL
• Redis
• SQLAlchemy
• Alembic
• React
• Docker / Docker Compose
• Nginx
• pytest
• Vitest
• React Testing Library
• GitHub Actions

Project Structure

app/                  # FastAPI backend application
  __init__.py
  main.py             # application entrypoint
  core/               # application configuration and environment setup
    __init__.py
    config.py
  services/           # external services and infrastructure logic
    __init__.py
    cache.py
    rate_limiter.py
  crud.py             # database operations
  database.py         # database connection and setup
  models.py           # SQLAlchemy models
  schemas.py          # Pydantic schemas
  utils.py            # helper utilities

alembic/              # Alembic migration environment
  versions/           # migration revision files
alembic.ini           # Alembic configuration

frontend/             # React frontend (Vite, API integration)
  src/
    App.jsx           # main frontend application
    App.test.jsx      # frontend UI tests
    test/             # frontend test setup

nginx/                # Nginx configuration for load balancing
tests/                # automated tests

scripts/
  seed.py             # sample data loader
  benchmark_cache.py  # measures cache miss vs. hit latency

docker-compose.yml    # service orchestration
Dockerfile            # app container definition
requirements.txt      # backend dependencies

Environment Variables

• ENV — runtime environment (development / production)
• DATABASE_URL — PostgreSQL connection string Neon-hosted deployments should keep sslmode=require; the deployed app typically uses the pooled Neon URL.
• REDIS_URL — Redis connection string (optional) Upstash-hosted deployments typically use a rediss://... URL.
• BASE_URL — base URL for generated short links
• CORS_ALLOW_ORIGINS — comma-separated frontend origins allowed to call the API
• AUTO_CREATE_SCHEMA — enables automatic table creation on startup; defaults to enabled in development/test and disabled in production-style environments
• PORT — application port

Health Check

The service exposes a health check endpoint:

GET /health

curl http://127.0.0.1:8000/health

How to Run Locally

Run Backend Stack with Docker

docker compose up --build

Open:

• Backend API docs: http://127.0.0.1:8000/docs
• Backend health: http://127.0.0.1:8000/health

The Docker Compose backend stack runs python -m alembic upgrade head before starting the FastAPI replicas, so a fresh database is migrated automatically.

Run Backend Locally (Services in Docker)

• Start required services:

docker compose up -d db redis

• Activate virtual environment:

source venv/bin/activate

• Install dependencies:

python3 -m pip install -r requirements.txt

• Configure local environment:

export BASE_URL=http://127.0.0.1:8000

For local runs, make sure DATABASE_URL, REDIS_URL, BASE_URL, and AUTO_CREATE_SCHEMA are set through your .env file or exported in the shell.

Example local backend environment:

export DATABASE_URL=postgresql://postgres:postgres@localhost:5432/urlshortener
export REDIS_URL=redis://localhost:6379/0
export BASE_URL=http://127.0.0.1:8000
export AUTO_CREATE_SCHEMA=false

Example hosted provider environment:

export DATABASE_URL=postgresql://<user>:<password>@<neon-host>/<database>?sslmode=require
export REDIS_URL=rediss://default:<password>@<upstash-host>:6379
export BASE_URL=https://your-backend-domain.onrender.com
export AUTO_CREATE_SCHEMA=false

• Apply migrations:

python3 -m alembic upgrade head

• Run backend:

python3 -m uvicorn app.main:app --reload

BASE_URL controls the short links returned by the API, so set it to the backend address you want clients to use.

Run Frontend Locally

cd frontend
npm install
export VITE_API_BASE_URL=http://127.0.0.1:8000
npm run dev

Open: Frontend: http://localhost:5173

VITE_API_BASE_URL tells the frontend which backend API to call in local development or deployment environments.

Run Tests

• Backend tests:

python3 -m pytest -v

• Frontend tests:

cd frontend
npm run test:run

The frontend uses Vitest and React Testing Library to cover core UI flows such as shorten success, stats fetch success, and backend error display.

GitHub Actions currently runs backend tests on every push and pull request. Frontend tests are available locally with npm run test:run.

Database Migrations

This project uses Alembic for schema migrations.

For the hosted Neon setup:

• Use the pooled Neon URL for the deployed application runtime

• Use the direct Neon URL for Alembic, psql, and other admin tasks when needed

• Apply the latest migrations:

python3 -m alembic upgrade head

• Create a new migration after changing models:

python3 -m alembic revision --autogenerate -m "describe change"

• Check the current revision:

python3 -m alembic current

• For normal development and deployment, prefer migrations with:

AUTO_CREATE_SCHEMA=false

AUTO_CREATE_SCHEMA=true is still available for quick demo or prototyping workflows, but AUTO_CREATE_SCHEMA=false should be the default once the schema is managed by Alembic.

Seed Sample Data

python3 -m alembic upgrade head
python3 -m scripts.seed

Run the migration step first when seeding a fresh database.

Quick Demo Flow

• Start the backend stack:

docker compose up --build

• In another terminal, start the frontend:

cd frontend
export VITE_API_BASE_URL=http://127.0.0.1:8000
npm run dev

• Open the frontend UI: http://localhost:5173
• Enter a URL (e.g., https://www.google.com) and generate a short link
• Open the returned short URL in your browser to verify redirection
• Check click statistics using the UI or via: GET /stats/{short_code}

Optional (API-level testing)

• Open API docs: http://127.0.0.1:8000/docs
• Create a short URL using POST /shorten

{
  "original_url": "https://www.google.com"
}

Distributed Architecture

This project demonstrates a horizontally scalable, distributed system with a React frontend and a FastAPI backend running locally using Docker Compose.

The architecture is designed for scalability, fault tolerance, and consistent behavior across multiple application instances.

The local distributed simulation uses Docker Compose, while the hosted deployment keeps the same application design with Render for app hosting, Neon for PostgreSQL, and Upstash for Redis.

Components

• Nginx load balancer
• 3 FastAPI application replicas
• PostgreSQL as the durable, shared source of truth
• Redis for shared caching and distributed rate limiting

System Request Flow

1. User interacts with the React frontend or sends a request directly to Nginx
2. Nginx routes the request to one FastAPI replica
3. The selected instance processes the request and interacts with shared PostgreSQL and Redis
4. Redirect requests use Redis as a cache-first lookup layer
5. Rate limiting is enforced globally across replicas through Redis

Architecture

React Frontend / Browser
      ↓
Nginx Load Balancer
      ↓
+-----------------------------+
|  FastAPI Application Layer  |
|  - app1                     |
|  - app2                     |
|  - app3                     |
+-----------------------------+
      ↓
+--------------------------------+
|  Shared Infrastructure         |
|  - Redis                       |
|    (cache + rate limit)        |
|  - PostgreSQL                  |
|    (source of truth)           |
+--------------------------------+

Key Design Decisions

• Introduced Nginx as a load balancer to distribute traffic across multiple FastAPI instances
• Ensured a stateless application design so any instance can handle any request
• Used Redis as a shared cache and coordination layer for rate limiting across replicas
• Used PostgreSQL as the single source of truth for URL mappings and analytics
• Enforced uniqueness of short codes at the database level to ensure correctness under concurrent distributed writes

Distributed Validation

• Confirmed load balancing by observing requests handled across multiple instances
• Verified shared Redis cache behavior across replicas
• Validated global rate limiting enforcement across instances
• Ensured consistent state via shared PostgreSQL storage

Request Flow

Create Short URL (POST /shorten)

• Request enters the FastAPI service
• Redis-backed rate limiter validates request frequency
• A short code is generated and checked for uniqueness
• Mapping is stored in PostgreSQL

Redirect (GET /{short_code})

• FastAPI checks Redis cache for the short code
• On cache hit → return redirect immediately (low latency)
• On cache miss:
  • Query PostgreSQL
  • Store result in Redis for future requests
  • Increment click count
  • Return redirect response

Stats (GET /stats/{short_code})

• Retrieve URL metadata and click count from PostgreSQL

Design Notes

• Click counts are updated even on cache hits to maintain consistency between cache and persistent storage
• Short codes are generated randomly and validated with a database uniqueness constraint to avoid collisions
• Redis is treated as an optional dependency, with graceful fallback to database queries to maintain system availability

Future Improvements

• Introduce background workers for asynchronous click tracking
• Enhance rate limiting with sliding window or token bucket algorithms
• Implement custom aliases and expiration policies
• Build analytics aggregation pipeline for high-volume traffic
• Deploy multi-instance setup to the cloud using container orchestration (e.g., Kubernetes)