Vichara AI is a production-grade distributed task orchestration platform for processing background jobs, AI workloads, and asynchronous pipelines at scale — with intelligent retry logic, cron scheduling, real-time monitoring, and distributed worker management.
Quick Start • Architecture • API Reference • Dashboard • Deployment
- Single Binary — one Go binary (
vichara) runs all services or individual roles - Redis Streams — durable message queue with consumer groups and auto-claim
- Distributed Workers — concurrent, horizontally scalable worker pools
- Smart Retries — exponential backoff with per-task configurable limits
- Cron Scheduler — full cron expression support (with seconds precision)
- Priority Queues — critical / high / normal / low priority routing
- Task Deduplication — Redis-backed idempotency by dedupe key
- Distributed Locking — per-task Redis lock prevents double-execution
- Dead Letter Queue — exhausted tasks captured for manual review
- Worker Heartbeats — detect stale workers, auto-prune records
- Prometheus Metrics — queue depth, latency histograms, worker activity
- Real-time Dashboard — Next.js + WebSocket live updates
- Helm Chart — production Kubernetes deployment with HPA
- Task Cancellation — cancel pending/running tasks via API
# Clone repo
git clone https://github.com/vikukumar/vichara-ai.git
cd vichara
# Start everything: Postgres, Redis, Vichara, Dashboard, Prometheus, Grafana
docker compose -f deployments/docker-compose.yml up -d
# Check health
curl http://localhost:8080/api/v1/health
# → {"status":"ok","version":"v1.0.0","uptime":"...","go":"go1.22"}
# Open dashboard
open http://localhost:3000# Prerequisites: Go 1.22+, PostgreSQL, Redis
# Build
make build
# Configure (edit database/redis credentials)
cp configs/config.example.yaml configs/config.yaml
# Run all services
./dist/vichara --mode=all
# Or run services individually:
./dist/vichara --mode=server # API + metrics only
./dist/vichara --mode=worker # Workers only
./dist/vichara --mode=scheduler # Cron scheduler onlyhelm install vichara ./helm/vichara \
--set postgresql.existingSecret=my-db-secret \
--set redis.existingSecret=my-redis-secret \
--namespace vichara --create-namespace┌─────────────────────────────────────────────────────────────────┐
│ Vichara AI Platform │
│ │
│ ┌──────────┐ REST/WS ┌──────────────────────────────────┐ │
│ │ Next.js │ ──────────▶│ API Server │ │
│ │ Dashboard │ │ POST /tasks GET /tasks │ │
│ └──────────┘ │ POST /cron GET /workers │ │
│ └────────────┬─────────────────────┘ │
│ │ │
│ ┌───────────▼───────────┐ │
│ │ Redis Streams │ │
│ │ vichara:queue:* │ │
│ └───────────┬───────────┘ │
│ │ │
│ ┌─────────────────────────┼─────────────┐ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌──────────┐ │
│ │ Worker [1] │ │ Worker [2] │ │ Worker N │ │
│ └─────────────┘ └─────────────┘ └──────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ PostgreSQL │ │
│ │ tasks task_results cron_jobs workers │ │
│ └──────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────┐ ┌──────────────────────────────┐ │
│ │ Scheduler │ │ Prometheus Metrics │ │
│ │ (cron jobs) │ │ :9090/metrics │ │
│ └──────────────┘ └──────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
| Component | Description |
|---|---|
| API Server | REST API + WebSocket hub, served on :8080 |
| Worker Pool | Configurable concurrency, polling Redis Streams with consumer groups |
| Scheduler | Cron expression runner (robfig/cron with seconds), persists jobs in Postgres |
| Redis Streams | Durable queue transport; supports XAUTOCLAIM for stale message recovery |
| PostgreSQL | Source of truth for task state, worker registry, cron definitions, results |
| Metrics Server | Prometheus scrape endpoint on :9090 |
POST /api/v1/tasks
Content-Type: application/json
{
"type": "email.send",
"payload": { "to": "user@example.com", "subject": "Hello", "body": "..." },
"queue": "default",
"priority": 5,
"max_retries": 3,
"timeout": "2m",
"dedupe_key": "email-user-123",
"metadata": { "user_id": "123" }
}| Field | Type | Description |
|---|---|---|
type |
string | Required. Registered handler name |
payload |
object | Handler-specific JSON input |
queue |
string | Target queue (default: vichara:queue:default) |
priority |
int | 1=low, 5=normal, 10=high, 20=critical |
max_retries |
int | Max retry attempts (default: 3) |
timeout |
duration | Max execution time (e.g., "5m") |
dedupe_key |
string | Deduplicate in-flight tasks by this key |
scheduled_at |
ISO8601 | Delay execution until this time |
GET /api/v1/tasks?status=pending&queue=default&limit=25&offset=0
GET /api/v1/tasks/{id}
DELETE /api/v1/tasks/{id} # CancelGET /api/v1/queues # Per-queue statistics
GET /api/v1/queues/dlq # Dead letter queue entriesGET /api/v1/workers # List all workers with heartbeat statusPOST /api/v1/cron
{
"name": "daily-report",
"expression": "0 0 8 * * *", # Every day at 08:00:00
"task_type": "data.process",
"payload": { "dataset": "daily-summary" },
"queue": "default"
}
GET /api/v1/cronGET /api/v1/health # Liveness probe
GET /api/v1/ready # Readiness probe
GET /metrics # Prometheus metrics (port 9090)| Task Type | Description |
|---|---|
email.send |
Simulate email delivery |
data.process |
Multi-step data pipeline |
ai.inference |
AI model inference simulation |
http.request |
Outbound HTTP webhook |
noop |
No-op (testing/scheduling) |
Register custom handlers with the tasks.Registry:
registry.Register("my.task", func(ctx context.Context, payload json.RawMessage) (json.RawMessage, error) {
// Your logic here
return json.Marshal(map[string]string{"status": "ok"})
})All settings can be overridden via environment variables (VICHARA_ prefix):
VICHARA_MODE=worker
VICHARA_DATABASE_HOST=my-postgres
VICHARA_DATABASE_PASSWORD=secret
VICHARA_REDIS_HOST=my-redis
VICHARA_WORKER_CONCURRENCY=20
VICHARA_LOG_LEVEL=debugSee configs/config.example.yaml for all options.
The Next.js dashboard provides:
- Overview — platform health, queue stats, recent tasks
- Tasks — filterable/paginated task explorer with full detail view
- Queues — live queue depths with visual breakdowns
- Workers — heartbeat monitoring and current task visibility
- Scheduler — cron job management with next/last run times
- Metrics — chart-based queue visualization
Start the dev dashboard:
cd web && npm install && npm run dev
# → http://localhost:3000docker build -t vichara:latest .
docker run -p 8080:8080 -p 9090:9090 \
-e VICHARA_DATABASE_HOST=postgres \
-e VICHARA_REDIS_HOST=redis \
vichara:latesthelm install vichara ./helm/vichara \
--set replicaCount=3 \
--set autoscaling.enabled=true \
--set autoscaling.maxReplicas=20 \
--set postgresql.host=my-postgres \
--set redis.host=my-redis# Install Go dependencies
go mod download
# Run all tests
make test
# Run with live reload (requires air)
air
# Format and lint
make fmt lint
# Build for all platforms
make build-allPrometheus metrics exposed at :9090/metrics:
| Metric | Description |
|---|---|
vichara_queue_size{queue} |
Current pending tasks |
vichara_task_duration_seconds{queue,type,status} |
Execution latency histogram |
vichara_tasks_succeeded_total{queue,type} |
Success counter |
vichara_tasks_failed_total{queue,type} |
Failure counter |
vichara_tasks_retried_total{queue,type} |
Retry counter |
vichara_tasks_dead_letter_total{queue,type} |
DLQ counter |
vichara_workers_active |
Active worker goroutines |
vichara_scheduled_jobs_fired_total{job_name} |
Cron fires |
See CONTRIBUTING.md. All contributions welcome!
MIT — see LICENSE.
Built with ❤️ by the Vichara AI team · v1.0.0