Distributed Workflow Engine
Workflow orchestration engine with event-driven execution.
Overview
Distributed Workflow Engine is an Airflow-inspired workflow orchestration platform built using a fully event-driven microservice architecture. The project explores how modern workflow schedulers execute Directed Acyclic Graphs (DAGs) by coordinating independent services through Apache Kafka instead of direct service-to-service communication. The platform separates workflow orchestration, scheduling, and task execution into dedicated services, making the system scalable, fault tolerant, and easier to evolve independently.
Problem Statement
Workflow orchestration platforms must execute complex business processes while respecting task dependencies, handling failures, and scaling execution across distributed workers. Implementing these responsibilities inside a single application quickly becomes difficult as workflows grow in size and complexity. This project demonstrates how workflow orchestration can be modeled using independent microservices that communicate asynchronously through Kafka events while executing DAG-based workflows reliably.
Features
- DAG-based workflow execution
- Workflow creation and management
- Task dependency management
- Event-driven orchestration using Kafka
- Independent scheduler and worker services
- Idempotent task execution
- Workflow completion detection
- Retry and failure handling
- REST APIs for workflow management
- Docker-based local deployment
Core Services
The system is composed of independently deployable services, each responsible for a single business capability.
Workflow Service
Creates workflows, tasks, and dependencies before publishing workflow start events.
Scheduler Service
Acts as the orchestration engine by evaluating DAG dependencies, selecting executable tasks, and publishing task-ready events.
Worker Service
Consumes task-ready events, executes tasks, updates execution status, and publishes completion events.
Common Libraries
Provides shared DTOs and Kafka event contracts across all services.
Technology Stack
Technologies, frameworks, infrastructure, and tooling used throughout the development of this project.
Backend
Java 21 • Spring Boot
Database
MySQL
Messaging
Apache Kafka (KRaft)
Infrastructure
Docker • Docker Compose
Developer Tools
Maven • Git
Engineering Decisions
Architectural choices made to improve scalability, maintainability, reliability, and long-term evolution of the system.
Workflow Orchestration using DAGs
Workflows are represented as Directed Acyclic Graphs where each task executes only after all its dependencies have successfully completed. This ensures deterministic execution order and supports complex workflow pipelines.
Event-Driven Communication
Instead of direct service-to-service communication, every major state transition is propagated through Kafka events. This significantly reduces coupling between services while improving scalability.
Dedicated Scheduler Service
Scheduling logic is isolated into a dedicated service responsible for evaluating task dependencies, selecting executable tasks, and triggering workers through Kafka events.
Idempotent Worker Execution
Workers are designed to safely process duplicate Kafka deliveries without executing the same task multiple times, improving reliability in distributed environments.
Stateless Scheduler
The scheduler itself remains stateless while workflow execution state is persisted in the database, allowing scheduler instances to scale horizontally.
Challenges
The most significant engineering challenges encountered while building the system.
Designing DAG execution algorithms.
Coordinating asynchronous workflows across multiple services.
Maintaining idempotent task execution.
Handling retries and task failures.
Detecting workflow completion without centralized execution logic.
What I Learned
Key technical concepts and engineering practices gained during the implementation.
How workflow orchestration engines operate internally.
Difference between orchestration and task execution.
Designing event-driven distributed systems using Kafka.
Implementing DAG-based scheduling algorithms.
Building fault-tolerant distributed services.
Future Improvements
Potential enhancements that could improve scalability, reliability, and maintainability.
Dead Letter Queue (DLQ) support.
Task timeout management.
Distributed tracing using OpenTelemetry.
Support for real production task execution.
Workflow visualization dashboard.