Skip to content

REDIS_MANDATORY.md

Latest commit

 

History

History
292 lines (223 loc) · 8.59 KB

REDIS_MANDATORY.md

File metadata and controls

292 lines (223 loc) · 8.59 KB

Redis Mandatory Configuration

Overview

Redis is now mandatory for production deployments. The application will refuse to start if Redis connectivity cannot be established, ensuring quotas, rate limiting, and background job queues work correctly in distributed environments.

Why Redis is Required

Redis provides essential infrastructure for:

  1. Distributed Rate Limiting: Prevent quota bypass across multiple workers
  2. Counter Synchronization: Track API usage across load-balanced instances
  3. Background Job Queue: Async validation processing with RQ
  4. Multi-Worker Safety: Shared state for horizontal scaling

Configuration

Production Mode (Default)

# Required - Application will fail to start without this
REDIS_URL=redis://localhost:6379

# Or with authentication
REDIS_URL=redis://:password@localhost:6379

# Or Redis Cloud
REDIS_URL=redis://username:password@host:port

Development Mode (Demo)

For local development without Redis:

DEMO_MODE=true

⚠️ Warning: DEMO_MODE uses ephemeral in-memory backends. Data is lost on restart. NOT safe for production.

Startup Behavior

Production (REDIS_URL required)

✓ Counter backend initialized (Redis)
✓ Queue service initialized (Redis)
→ Application starts

Production (No REDIS_URL)

❌ Redis validation failed: REDIS_URL environment variable is required
→ Application REFUSES to start

Demo Mode (Fallback allowed)

⚠ Using in-memory counters (DEMO_MODE)
⚠ Using in-memory queue (DEMO_MODE)
→ Application starts with warnings

Health Check Endpoints

/health

Main health check with Redis status:

{
  "status": "healthy",
  "version": "2.0.0",
  "environment": "production",
  "redis": {
    "status": "connected",
    "backend": "redis"
  },
  "queue": {
    "status": "connected",
    "backend": "redis",
    "queue_depth": 0
  }
}

Returns HTTP 503 if Redis is down in production.

/admin/health/redis

Detailed Redis monitoring:

{
  "counter_backend": {
    "status": "connected",
    "backend": "redis"
  },
  "queue_service": {
    "status": "connected",
    "backend": "redis",
    "queue_depth": 0
  },
  "demo_mode": false
}

Installation

Redis Server

Local Development:

# macOS
brew install redis
brew services start redis

# Ubuntu/Debian
sudo apt-get install redis-server
sudo systemctl start redis

# Docker
docker run -d -p 6379:6379 redis:7-alpine

Python Dependencies:

pip install redis[hiredis] rq

Cloud Redis

  • Render: Add Redis add-on to your service
  • Railway: Add Redis database
  • Upstash: Serverless Redis
  • Redis Cloud: Managed Redis hosting

Testing

Run the Redis connectivity tests:

# With DEMO_MODE (uses in-memory fallback)
DEMO_MODE=true pytest tests/test_redis_mandatory.py -v

# Production mode tests (requires Redis)
REDIS_URL=redis://localhost:6379 pytest tests/test_redis_mandatory.py -v

Migration Guide

From Optional to Mandatory Redis

  1. Set DEMO_MODE for existing dev environments:

    # .env
DEMO_MODE=true

  2. Install Redis for production:

    • Add Redis add-on to hosting platform
    • Update REDIS_URL environment variable
    • Remove DEMO_MODE (or set to false)

  3. Verify health checks:

    curl http://localhost:8000/health
curl http://localhost:8000/admin/health/redis

  4. Monitor startup logs:

    ✓ Redis counter backend connected: localhost:6379
✓ Counter backend initialized
✓ Redis queue service connected
✓ Queue service initialized

Troubleshooting

Error: "REDIS_URL environment variable is required"

Cause: Running in production mode without Redis configured.

Solutions:

  • Set REDIS_URL environment variable
  • Or enable DEMO_MODE=true for local development only

Error: "Failed to connect to Redis at redis://..."

Cause: Redis server is not running or unreachable.

Solutions:

  • Start Redis server: redis-server or brew services start redis
  • Check Redis is listening: redis-cli ping (should return PONG)
  • Verify REDIS_URL is correct
  • Check firewall/network connectivity

Health check returns HTTP 503

Cause: Redis connectivity lost during runtime.

Solutions:

  • Check Redis server status
  • Review Redis logs
  • Restart application to re-establish connection

In-memory fallback warning in logs

Cause: DEMO_MODE=true is enabled.

Impact:

  • Rate limits and quotas NOT synchronized across workers
  • Background jobs NOT durable across restarts
  • Data loss on application restart

Solution: Deploy Redis and disable DEMO_MODE for production

Environment Variable Reference

Variable Required Default Description
REDIS_URL Yes (prod) - Redis connection string
DEMO_MODE No false Allow in-memory fallback for local dev
ENVIRONMENT No development Environment name (logs/monitoring)

Files Modified

  • app/services/state_backend.py - Enforces Redis for counter backend
  • app/services/queue.py - Enforces Redis for job queue
  • app/main.py - Validates Redis on startup
  • .env.example - Documents mandatory Redis configuration
  • tests/test_redis_mandatory.py - Comprehensive test suite

Architecture

┌─────────────────────────────────────────────────────────┐
│                  FastAPI Application                     │
│                                                          │
│  ┌──────────────────────────────────────────────────┐  │
│  │            Startup Validation                     │  │
│  │  1. Check DEMO_MODE                               │  │
│  │  2. Validate REDIS_URL                            │  │
│  │  3. Test Redis connectivity (ping)                │  │
│  │  4. Initialize counter backend                    │  │
│  │  5. Initialize queue service                      │  │
│  │  → Fail fast if Redis unavailable in production  │  │
│  └──────────────────────────────────────────────────┘  │
│                                                          │
│  ┌──────────────┐           ┌──────────────┐           │
│  │ Rate Limiter │──Redis──→ │   Counters   │           │
│  └──────────────┘           └──────────────┘           │
│                                                          │
│  ┌──────────────┐           ┌──────────────┐           │
│  │ Queue Worker │──Redis──→ │ Background   │           │
│  │              │           │ Jobs (RQ)    │           │
│  └──────────────┘           └──────────────┘           │
└─────────────────────────────────────────────────────────┘
                         ▲
                         │
                         │ Health Checks
                         │ /health
                         │ /admin/health/redis
                         ▼
                   ┌──────────┐
                   │  Redis   │
                   │  Server  │
                   └──────────┘

Security Notes

  • Redis should not be exposed to the internet
  • Use authentication: REDIS_URL=redis://:password@host:port
  • Enable TLS for cloud deployments
  • Restrict network access with firewall rules
  • Monitor Redis logs for unauthorized access attempts

Performance

  • Redis adds ~1ms latency per operation
  • Use connection pooling (handled by redis-py automatically)
  • Monitor Redis memory usage
  • Consider Redis cluster for high-traffic deployments

Support

For issues related to Redis connectivity:

  1. Check application logs for detailed error messages
  2. Verify Redis server is running: redis-cli ping
  3. Test connectivity: redis-cli -u $REDIS_URL ping
  4. Review health endpoint: GET /health
  5. File an issue with logs and Redis configuration