Distributed URL Shortener
Production-inspired distributed URL shortening platform designed around scalable microservices, event-driven communication, caching, rate limiting, and cloud-native deployment.
Overview
Distributed URL Shortener is a production-inspired distributed backend platform built to explore how large-scale URL shortening systems can be designed beyond basic CRUD functionality. The system uses Spring Boot microservices, Spring Cloud, Apache Kafka, Redis, PostgreSQL, Docker, and Kubernetes while focusing on low-latency redirects, asynchronous analytics, distributed caching, rate limiting, resilience, and observability.
Problem Statement
A basic URL shortener can generate a short code and store the original URL in a database. At production scale, however, the system must handle high read traffic, invalid short-code requests, concurrent users, analytics processing, authentication, abuse prevention, and service failures without increasing redirect latency. This project explores those challenges by separating responsibilities across independently deployable services and introducing Redis caching, Bloom Filters, Kafka-based event processing, distributed rate limiting, retry mechanisms, and cloud-native deployment.
Features
- Distributed microservices architecture
- URL creation and short-code management
- Low-latency URL redirection
- JWT access and refresh token authentication
- Refresh token rotation
- Centralized API Gateway
- Service discovery using Eureka
- Redis-based URL caching
- Bloom Filter validation for invalid short codes
- Kafka-based click event processing
- Asynchronous analytics pipeline
- CQRS-inspired analytics architecture
- Distributed Token Bucket rate limiting
- Redis and Lua-based atomic rate limiting
- Kafka retry handling
- Dead Letter Queue processing
- Browser, device, and operating system analytics
- Performance benchmarking using k6
- Prometheus metrics and Grafana visualization
- Dockerized microservices
- Kubernetes deployment using Minikube
Core Services
The system is composed of independently deployable services, each responsible for a single business capability.
API Gateway
Provides centralized request routing, JWT validation, authentication, distributed rate limiting, CORS handling, and other cross-cutting concerns.
Discovery Server
Provides service registration and discovery using Spring Cloud Eureka.
Auth Service
Handles user registration, login, JWT access tokens, refresh token management, and authentication workflows.
URL Service
Creates and manages shortened URLs, ownership information, metadata, and URL-created events.
Redirect Service
Resolves short codes with Bloom Filter validation and Redis caching while publishing click events asynchronously.
Analytics Service
Consumes Kafka click events and maintains aggregated browser, device, operating system, and click analytics.
Apache Kafka
Acts as the asynchronous event backbone for URL and click-related events between services.
Redis
Provides low-latency URL caching and shared state for distributed rate limiting.
PostgreSQL
Stores persistent authentication, URL metadata, and aggregated analytics data.
Technology Stack
Technologies, frameworks, infrastructure, and tooling used throughout the development of this project.
Backend
Java 21 • Spring Boot • Spring Cloud • Spring Security • Spring Data JPA • Spring Cloud Gateway • Eureka • Maven
Database
PostgreSQL • Redis • Bloom Filter
Messaging
Apache Kafka • Spring Kafka • Event-Driven Architecture • Asynchronous Messaging • Retry Processing • Dead Letter Queue • CQRS-Inspired Analytics
Infrastructure
Docker • Docker Compose • Kubernetes • Minikube • kubectl
Developer Tools
k6 • Prometheus • Grafana • Spring Boot Actuator • JWT • Lua Scripts
Engineering Decisions
Architectural choices made to improve scalability, maintainability, reliability, and long-term evolution of the system.
Microservice Architecture
Authentication, URL management, redirects, analytics, service discovery, and gateway responsibilities are separated so individual parts of the platform can evolve and scale independently.
Redis-Backed Redirect Cache
Frequently accessed short URLs are cached in Redis to reduce PostgreSQL traffic and improve latency on the read-heavy redirect path.
Bloom Filter Before Database Lookup
A Bloom Filter is used to reject short codes that definitely do not exist before querying Redis or PostgreSQL, reducing unnecessary database work for invalid requests.
Event-Driven Analytics
Redirect requests publish click events to Kafka instead of processing analytics synchronously, keeping analytics work outside the latency-sensitive redirect path.
CQRS-Inspired Analytics
Analytics processing and querying are separated from the main redirect workflow, allowing analytics workloads to scale independently without affecting URL resolution.
Distributed Rate Limiting
A Redis-backed Token Bucket algorithm with Lua scripts and atomic operations provides consistent rate-limit enforcement across multiple API Gateway instances.
Kafka Retry and Dead Letter Queue
Failed analytics events are retried before being moved to a Dead Letter Queue, preventing transient consumer failures from causing silent event loss.
Stateless JWT Authentication
JWT access tokens and refresh token rotation allow authentication to remain suitable for horizontally scaled services without relying on traditional server-side sessions.
Pre-Aggregated Analytics
Analytics such as total clicks and browser, device, and operating system distributions are aggregated during event processing instead of repeatedly scanning raw click events.
Cloud-Native Deployment
Services are containerized with Docker and Kubernetes manifests are used to run the distributed platform on Minikube.
Challenges
The most significant engineering challenges encountered while building the system.
Designing a low-latency redirect path without repeatedly querying PostgreSQL.
Keeping analytics processing completely outside the user-facing redirect path.
Coordinating synchronous REST communication with asynchronous Kafka events.
Implementing distributed rate limiting consistently across gateway instances.
Handling Kafka consumer failures without losing click events.
Preventing unnecessary database traffic caused by invalid short codes.
Managing JWT authentication and refresh tokens under concurrent requests.
Containerizing multiple Spring Boot services and deploying them on Kubernetes.
Benchmarking the platform under concurrent traffic and identifying concurrency-related failures.
What I Learned
Key technical concepts and engineering practices gained during the implementation.
How production URL shorteners differ from simple CRUD implementations.
How Redis can reduce latency and database load in read-heavy systems.
How Bloom Filters can protect databases from unnecessary negative lookups.
How Kafka decouples analytics processing from latency-sensitive request paths.
How retry and Dead Letter Queue strategies improve event-processing resilience.
How distributed rate limiting can be implemented with Redis and Lua scripts.
How microservices can be separated around clear responsibilities.
How load testing can expose concurrency problems that functional testing does not reveal.
How Docker and Kubernetes are used to package and orchestrate Spring Boot microservices.
How observability and benchmarking help evaluate distributed backend systems.
Future Improvements
Potential enhancements that could improve scalability, reliability, and maintainability.
Expand Kubernetes deployment beyond local Minikube environments.
Add stronger automated integration and end-to-end testing.
Introduce additional tracing across distributed request and event flows.
Benchmark redirect performance under larger concurrent workloads.
Explore horizontal scaling of redirect and analytics consumers.
Improve automated CI/CD for container builds and Kubernetes deployments.
Add deeper operational dashboards and alerting.