Distributed Streaming Platform
Kafka-powered event ingestion and real-time data processing platform.
Overview
Distributed Streaming Platform is a production-grade distributed microservices project built to simulate a real-world video streaming backend. The system is organized around separate services for authentication, user management, content metadata, content ingestion, streaming, video processing, and analytics. It is designed to demonstrate how modern backend systems are structured using microservices, Kafka-based event communication, service discovery, API gateway routing, Docker containerization, and Kubernetes deployment.
Problem Statement
Streaming platforms need to handle content uploads, asynchronous processing, streaming delivery, and analytics at scale. A monolithic architecture makes it harder to isolate responsibilities and scale individual parts of the system. This project explores how a streaming backend can be decomposed into independently deployable services so each part of the workflow can scale, evolve, and fail more independently.
Features
- Microservice-based streaming backend
- API Gateway routing
- Service discovery with Eureka
- Kafka-based event communication
- Asynchronous content ingestion
- Video processing pipeline
- Streaming service
- Analytics service
- Dockerized local setup
- Kubernetes deployment
Core Services
The system is composed of independently deployable services, each responsible for a single business capability.
API Gateway
Routes all incoming requests and acts as the single entry point.
Discovery Server
Provides service registry and discovery through Eureka.
Auth Service
Handles authentication and JWT-based access control.
User Service
Manages user-related operations.
Content Service
Stores and manages content metadata.
Content Ingestion Service
Handles the upload pipeline and produces events for downstream processing.
Streaming Service
Handles streaming-related logic.
Video Processing Service
Processes uploaded video content asynchronously.
Analytics Service
Processes events and generates analytics.
Technology Stack
Technologies, frameworks, infrastructure, and tooling used throughout the development of this project.
Backend
Java • Spring Boot • Spring Cloud • Spring Security • JWT
Database
MySQL • Redis • MinIO
Messaging
Apache Kafka • Kafka KRaft
Infrastructure
Docker • Docker Compose • Kubernetes • Minikube • NGINX Ingress
Developer Tools
Maven • Prometheus • Git • Postman
Engineering Decisions
Architectural choices made to improve scalability, maintainability, reliability, and long-term evolution of the system.
Microservice Architecture
The platform is split into independent services so content ingestion, streaming, analytics, and authentication can evolve separately and scale independently.
API Gateway Pattern
All external requests go through a central API Gateway, which simplifies client interaction and hides the internal service topology.
Service Discovery
Each service registers with Eureka so requests can be routed dynamically without hardcoding service locations.
Event-Driven Communication
Kafka is used for asynchronous workflows such as content ingestion, video processing, and analytics so services stay loosely coupled.
Database per Service Style
The repository organizes storage around separate data responsibilities, using MySQL for persistence, Redis for caching, and MinIO for file storage.
Kubernetes Deployment
The system is packaged for Kubernetes so services, Kafka, databases, and ingress can be orchestrated in a production-like environment.
Challenges
The most significant engineering challenges encountered while building the system.
Splitting a streaming backend into clear service boundaries.
Coordinating synchronous REST calls with asynchronous Kafka events.
Managing the upload, processing, and streaming pipeline across services.
Handling several infrastructure components during local development.
Keeping the system observable and deployable at scale.
What I Learned
Key technical concepts and engineering practices gained during the implementation.
How distributed streaming systems are structured with microservices.
How Kafka supports asynchronous backend workflows.
How service discovery and API gateways simplify distributed communication.
How Kubernetes changes the way backend systems are deployed.
How different storage systems can be combined for different backend needs.
Future Improvements
Potential enhancements that could improve scalability, reliability, and maintainability.
Add CI/CD with GitHub Actions.
Enable autoscaling with Kubernetes HPA.
Expand monitoring and tracing.
Add stronger test coverage.
Polish the video upload and processing pipeline.