A lightweight event-driven container orchestration system built with Node.js, TypeScript, Docker, BullMQ, Kafka, and PostgreSQL.
Supports transactional configs, container scaling, queue-based processing, and event-driven orchestration.
apps/api-service– Accepts config, writes to DB +outbox_serviceatomicallyapps/watcher-service– Pollsoutbox_servicetable and pushes tasks to BullMQapps/processor– Processes queue, scales containers, pushes Kafka eventsapps/worker– Kafka consumer, updatesoutbox_servicestatus
packages/common– YAML parser & Zod validationpackages/db– Prisma & DB utilitiespackages/kafka-service– Kafka client & reusable logicpackages/queue– BullMQ queue creation and export
- User sends POST
/config→ api-service inserts intoservicesandoutbox_servicetables transactionally. - watcher-service polls
outbox_serviceforPENDINGentries → pushes to BullMQ queue. - processor consumes queue → reconciles containers based on
desired_state→ emits Kafka events when done. - worker listens to Kafka → updates
outbox_servicestatus toDONE.
Fully event-driven and transactional orchestration system.
- Redis for BullMQ:
docker run -p 6379:6379 --name redis redis- Kafka + Zookeeper (Docker):
docker run -d --name zookeeper -p 2181:2181 zookeeper
docker run -d --name kafka -p 9092:9092 --link zookeeper wurstmeister/kafkaCreate .env in root:
DATABASE_URL=postgresql://user:password@localhost:5432/db
REDIS_URL=redis://localhost:6379
KAFKA_BROKER=localhost:9092
npm installWith Turborepo, running from the root automatically starts all apps:
npm run devEach service builds TypeScript and runs automatically.
- Transactional outbox pattern for reliability.
- BullMQ queues with retries for async task processing.
- Kafka events for acknowledgment and decoupling.
- Container orchestration with Docker (scaling, dynamic ports, volume bindings).
- Monorepo utilities (
common,db,queue,kafka-service).
Node.js, TypeScript, Prisma, PostgreSQL, Docker, BullMQ, Redis, Kafka, Zod, YAML.