diff --git a/.env.example b/.env.example index f222a0b..28892d9 100644 --- a/.env.example +++ b/.env.example @@ -3,15 +3,19 @@ ASPNETCORE_URLS=http://+:3010 LIS_EXEC_HOST=false # true = exec commands run on host via nsenter (requires pid:host + privileged) # Database +POSTGRES_PASSWORD=changeme DATABASE_URL=Host=postgres;Database=lis;Username=lis;Password=changeme # AI Provider (Anthropic) ANTHROPIC_ENABLED=true +# Accepts standard API keys (sk-ant-api03-...) or long-lived OAuth tokens from `claude setup-token` (sk-ant-oat01-...). +# OAuth tokens are auto-detected by prefix; set ANTHROPIC_AUTH_MODE=bearer to force bearer auth if needed. ANTHROPIC_API_KEY=sk-ant-... -ANTHROPIC_MODEL=claude-sonnet-4-20250514 -ANTHROPIC_MAX_TOKENS=4096 -ANTHROPIC_CONTEXT_BUDGET=12000 -ANTHROPIC_THINKING_EFFORT= +ANTHROPIC_AUTH_MODE= +ANTHROPIC_MODEL=claude-sonnet-4-6 +ANTHROPIC_MAX_TOKENS=16000 +ANTHROPIC_CONTEXT_BUDGET=300000 +ANTHROPIC_THINKING_EFFORT= # off | low | medium | high | ANTHROPIC_CACHE_ENABLED=true ANTHROPIC_CACHE_TTL=5m @@ -25,7 +29,9 @@ MEMORIES_EMBEDDING_BASE_URL= # Channel (WhatsApp / GOWA) GOWA_ENABLED=true GOWA_BASE_URL=http://gowa:3000 -GOWA_DEVICE_ID=default +# After pairing your WhatsApp device (via the GOWA UI QR code), replace this with the real device UUID +# shown in GOWA's /app/devices endpoint. GOWA v8.3.3+ rejects requests with an unknown X-Device-Id. +GOWA_DEVICE_ID= GOWA_BASIC_AUTH= GOWA_WEBHOOK_SECRET=changeme @@ -37,6 +43,12 @@ LIS_REACT_ON_MESSAGE_QUEUED=false LIS_REACT_ON_MESSAGE_QUEUED_EMOJI=πŸ• LIS_TOOL_NOTIFICATIONS=true LIS_MAX_TOOL_ITERATIONS=10 +LIS_GROUP_CONTEXT_MESSAGES= # empty = default; last N consecutive user msgs kept in group context +LIS_NEW_SESSION_ON_AGENT_SWITCH=false + +# Web Search (Brave) β€” optional, enables the web_search tool +LIS_WEB_SEARCH_ENABLED=false +LIS_WEB_SEARCH_API_KEY= # Context Compaction (0 or empty = percentage of ANTHROPIC_CONTEXT_BUDGET) LIS_KEEP_RECENT_TOKENS=4000 diff --git a/README.md b/README.md index f326d7b..0f85203 100644 --- a/README.md +++ b/README.md @@ -163,7 +163,9 @@ Lis.Tests β€” xUnit test suite ### Prerequisites - Docker & Docker Compose -- Anthropic API key ([console.anthropic.com](https://console.anthropic.com)) +- Anthropic credentials β€” either: + - An API key from [console.anthropic.com](https://console.anthropic.com) (`sk-ant-api03-...`), or + - A long-lived OAuth token from `claude setup-token` (`sk-ant-oat01-...`) β€” requires Claude Code installed locally ### Setup @@ -171,9 +173,10 @@ Lis.Tests β€” xUnit test suite git clone && cd lis cp .env.example .env # Edit .env β€” set at minimum: -# ANTHROPIC_API_KEY=sk-ant-... +# ANTHROPIC_API_KEY=sk-ant-... # or sk-ant-oat01-... from `claude setup-token` # LIS_OWNER_JID=@s.whatsapp.net # GOWA_WEBHOOK_SECRET= +# POSTGRES_PASSWORD= docker compose up -d # Scan QR code at http://localhost:3000 @@ -277,6 +280,7 @@ OPENAI_API_KEY=sk-... # enables Whisper transcription | Doc | Topic | |-----|-------| +| [DEPLOYMENT.md](docs/DEPLOYMENT.md) | Production deployment (Caddy + managed Postgres, two-stack compose) | | [AGENTS.md](docs/AGENTS.md) | Multi-agent system, per-chat config, agent switching | | [CONTEXT_COMPACTION.md](docs/CONTEXT_COMPACTION.md) | Rolling compaction, sessions, token tracking, prompt caching | | [SECURITY_MODEL.md](docs/SECURITY_MODEL.md) | 5-layer defense, tool auth, workspace sandbox | diff --git a/docker-compose.yml b/docker-compose.yml index d113e9e..c4516d0 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -20,7 +20,7 @@ services: postgres: container_name: backend-postgres - image: postgres:17-alpine + image: pgvector/pgvector:pg17 restart: always environment: POSTGRES_DB: lis @@ -31,7 +31,7 @@ services: gowa: container_name: gowa - image: aldinokemal/go-whatsapp-web-multidevice:latest + image: aldinokemal2104/go-whatsapp-web-multidevice:v8.3.3 restart: unless-stopped ports: - "3000:3000" @@ -40,6 +40,24 @@ services: WHATSAPP_WEBHOOK: "http://backend:3010/webhook/whatsapp" WHATSAPP_WEBHOOK_SECRET: "${GOWA_WEBHOOK_SECRET}" WHATSAPP_WEBHOOK_EVENTS: "message" + volumes: + - gowa-data:/app/storages + + # Optional web UI for browsing/editing the Postgres DB during development. + # Enable with: docker compose --profile db-ui up -d + # Then open http://localhost:8081 + pgweb: + container_name: pgweb + image: sosedoff/pgweb:latest + restart: unless-stopped + profiles: ["db-ui"] + ports: + - "8081:8081" + environment: + PGWEB_DATABASE_URL: "postgres://lis:${POSTGRES_PASSWORD}@postgres:5432/lis?sslmode=disable" + depends_on: + - postgres volumes: pgdata: + gowa-data: diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md new file mode 100644 index 0000000..501e430 --- /dev/null +++ b/docs/DEPLOYMENT.md @@ -0,0 +1,340 @@ +# Deployment Guide + +Production replication guide for Lis on a single Linux host, reachable over a public domain. + +> For local development, see the root `README.md` Quick Start β€” it uses the repo's bundled `docker-compose.yml` with a local Postgres container. This guide is for **running Lis as an always-on service** with TLS and managed data. + +## Topology + +Two Docker Compose stacks sharing a `proxy` bridge network: + +| Stack | Services | Role | +|---|---|---| +| `services/` | `caddy`, `gowa` | TLS reverse proxy + WhatsApp gateway | +| `lis/` | `lis` | .NET 10 agent (Semantic Kernel + Anthropic) | + +Data flow: + +``` +WhatsApp <──> GOWA ──webhook──> Lis (Anthropic, tools) + β–² + β”‚ https + Caddy ── your.domain.com +``` + +**Persistent data lives on managed Postgres (Neon recommended):** both the `lis` application database and the `gowa` session database. Nothing is stored in a local Postgres container. + +## Final directory layout + +``` + +β”œβ”€β”€ services/ +β”‚ β”œβ”€β”€ docker-compose.yml +β”‚ β”œβ”€β”€ Caddyfile +β”‚ β”œβ”€β”€ .env +β”‚ β”œβ”€β”€ config/ # Caddy config volume (created by Caddy) +β”‚ β”œβ”€β”€ data/ # Caddy data volume β€” TLS certs live here +β”‚ └── gowa-data/ # GOWA WhatsApp session (IMPORTANT β€” back this up) +└── lis/ + β”œβ”€β”€ docker-compose.yml + β”œβ”€β”€ .env + └── lis/ # git clone of this repo + └── Lis.Api/Dockerfile +``` + +--- + +## Prerequisites + +On the target host: + +- Linux with Docker Engine + Compose v2 (`docker compose version` must work). +- Public IP, with a DNS A/AAAA record pointing your chosen domain at it. +- Ports **80/tcp**, **443/tcp**, **443/udp** open to the internet (Caddy ACME + HTTP/3). + +Accounts / keys: + +- **[Neon](https://neon.tech)** (or any managed Postgres with `pgvector`) β€” create a project, then create two databases with separate roles: + - database `lis`, role `lis` (used by the Lis app β€” requires `vector` extension) + - database `gowa`, role `gowa` (used by GOWA) + - For each, copy the pooled connection string. +- **Anthropic** credentials β€” an API key (`sk-ant-api03-...`) or a long-lived OAuth token from `claude setup-token` (`sk-ant-oat01-...`). +- *(optional)* **OpenAI** API key β€” only if you want vector search over memories. +- *(optional)* **Brave Search** API key β€” only if you want the `web_search` tool. + +--- + +## Step 1 β€” Create `services/` + +```bash +mkdir -p services/{config,data,gowa-data} +cd services +``` + +### `services/docker-compose.yml` + +```yaml +services: + caddy: + image: caddy:2-alpine + restart: always + ports: + - "80:80" + - "443:443" + - "443:443/udp" + networks: + - proxy + env_file: + - .env + volumes: + - ./config:/config/caddy + - ./data:/data/caddy + - ./Caddyfile:/etc/caddy/Caddyfile:ro + + gowa: + image: aldinokemal2104/go-whatsapp-web-multidevice:v8.3.3 + restart: always + networks: + - proxy + env_file: + - .env + volumes: + - ./gowa-data:/app/storages + +networks: + proxy: + name: proxy + driver: bridge +``` + +Notes: +- This file **creates** the `proxy` network (not marked `external` here). The Lis stack attaches to it as `external`. +- `gowa` publishes no host ports; it's reached via the Docker network as `gowa:3000` (and through Caddy from the public side). +- **Pin the GOWA image tag** β€” upgrades can break the webhook payload shape. + +### `services/Caddyfile` + +```caddyfile +{ + email YOUR_ACME_EMAIL@example.com +} + +{$GOWA_DOMAIN} { + reverse_proxy gowa:{$APP_PORT} +} +``` + +Replace `YOUR_ACME_EMAIL@example.com` with the address Let's Encrypt should register against. `{$GOWA_DOMAIN}` and `{$APP_PORT}` come from `services/.env`. + +### `services/.env` + +Generate your own secrets for the marked fields (`openssl rand -hex 16`). `WHATSAPP_WEBHOOK_SECRET` and `APP_BASIC_AUTH` **must match** the matching keys in `lis/.env`. + +```env +# Caddy +GOWA_DOMAIN=your.domain.com + +# GOWA β€” App +APP_PORT=3000 +APP_HOST=0.0.0.0 +APP_DEBUG=false +APP_OS=Chrome +# Basic auth (user1:pass1,user2:pass2) β€” protects the GOWA web UI and REST API. +APP_BASIC_AUTH=lis:CHANGE_ME_STRONG_PASSWORD +# Base path for subpath deployment (e.g. /whatsapp) β€” leave empty for root. +APP_BASE_PATH= +APP_TRUSTED_PROXIES= + +# GOWA β€” Database (managed Postgres) +# Paste the pooled connection string for the `gowa` database. +DB_URI=postgres://gowa:PASSWORD@HOST/gowa?sslmode=require&channel_binding=require + +# GOWA β€” WhatsApp +WHATSAPP_AUTO_REPLY=false +WHATSAPP_AUTO_MARK_READ=false +WHATSAPP_AUTO_DOWNLOAD_MEDIA=true +WHATSAPP_ACCOUNT_VALIDATION=true +WHATSAPP_PRESENCE_ON_CONNECT=available +WHATSAPP_AUTO_REJECT_CALL=true + +# GOWA β€” Webhook (points at the Lis container on the proxy network) +WHATSAPP_WEBHOOK=http://lis:3010/webhook/whatsapp +WHATSAPP_WEBHOOK_SECRET=CHANGE_ME_WEBHOOK_SECRET +WHATSAPP_WEBHOOK_EVENTS=message +WHATSAPP_WEBHOOK_INSECURE_SKIP_VERIFY=true + +# GOWA β€” Chatwoot (disabled) +CHATWOOT_ENABLED=false +``` + +--- + +## Step 2 β€” Clone Lis and create `lis/` + +```bash +mkdir -p lis +cd lis +git clone lis # clones into ./lis/ +``` + +### `lis/docker-compose.yml` + +```yaml +services: + lis: + mem_limit: 512m + build: + context: ./lis/ + dockerfile: ./Lis.Api/Dockerfile + container_name: lis + restart: unless-stopped + env_file: .env + # pid: host + privileged are ONLY required when LIS_EXEC_HOST=true. + # Drop both if you don't need the agent to shell out to the host. + pid: host + privileged: true + networks: + - proxy + ports: + - "3010" + +networks: + proxy: + external: true +``` + +Notes: +- `container_name: lis` matters β€” GOWA's webhook URL (`http://lis:3010/...`) resolves by container name inside the `proxy` network. +- `pid: host` + `privileged: true` are required **only** because `LIS_EXEC_HOST=true` lets the agent shell out to the host via `nsenter`. If you don't want that capability, set `LIS_EXEC_HOST=false` and drop both fields β€” safer container. +- `networks.proxy.external: true` β€” this stack fails to start if the services stack hasn't created the `proxy` network yet. Always bring `services` up first. +- `ports: - "3010"` (no left side) publishes 3010 to a random host port for optional debugging β€” not needed for the webhook flow. + +### `lis/.env` + +Start from the canonical [`.env.example`](../.env.example) at the repo root. Values that differ on a production host: + +```env +# App β€” memory-tuned GC for the 512m container limit +ASPNETCORE_URLS=http://+:3010 +DOTNET_gcServer=0 +DOTNET_GCHeapHardLimit=0x10000000 +DOTNET_GCDynamicAdaptationMode=1 + +# Database β€” managed Postgres connection string (pgvector required) +DATABASE_URL=Host=HOST;Database=lis;Username=lis;Password=PASSWORD;SSL Mode=Require;Channel Binding=Require + +# Channel β€” GOWA (public URL via Caddy) +GOWA_ENABLED=true +GOWA_BASE_URL=https://your.domain.com +GOWA_DEVICE_ID=lis +# MUST equal APP_BASIC_AUTH in services/.env +GOWA_BASIC_AUTH=lis:CHANGE_ME_STRONG_PASSWORD +# MUST equal WHATSAPP_WEBHOOK_SECRET in services/.env +GOWA_WEBHOOK_SECRET=CHANGE_ME_WEBHOOK_SECRET + +# Shell execution on the host (requires pid:host + privileged in compose) +LIS_EXEC_HOST=true +``` + +Every other key β€” model, context budget, compaction tuning, memory embeddings, etc. β€” lives in [`.env.example`](../.env.example). Copy the example and override only what's different for your deployment. + +--- + +## Step 3 β€” Boot order + +The **services stack must come up first**, because it owns the `proxy` network that the Lis stack declares as external. + +```bash +cd services +docker compose up -d + +cd ../lis +docker compose up -d --build # first run builds the .NET image (slow β€” pulls Playwright/Chromium) +``` + +Verify three containers are running: + +```bash +docker ps +# Expected: services-caddy-1, services-gowa-1, lis β€” all Up +``` + +Tail logs during first boot: + +```bash +docker logs services-caddy-1 -f # ACME issues a cert for GOWA_DOMAIN +docker logs services-gowa-1 -f # DB connect, HTTP server up +docker logs lis -f # EF migrations, Anthropic client init +``` + +--- + +## Step 4 β€” Database migrations (Lis) + +Lis uses EF Core migrations; the Lis container applies them on startup. On the first boot, confirm they succeeded by tailing `docker logs lis -f` until you see `Application started`. + +For manual control (requires the .NET 10 SDK and `dotnet-ef` on the host): + +```bash +dotnet tool install --global dotnet-ef --version 10.* # once, if missing + +cd lis/lis +dotnet ef database update \ + --project Lis.Persistence/Lis.Persistence.csproj \ + --startup-project Lis.Api/Lis.Api.csproj +``` + +GOWA handles its own schema creation on first connect β€” no manual migration needed. + +--- + +## Step 5 β€” Pair WhatsApp (GOWA) + +1. Browse to `https:///`. +2. Basic auth prompt β€” log in with the credentials you set in `APP_BASIC_AUTH`. +3. From the GOWA UI choose **Login** / **Scan QR**. +4. On your phone: WhatsApp β†’ Settings β†’ Linked devices β†’ Link a device β†’ scan. +5. The session is persisted to `services/gowa-data/`. **Back that directory up** β€” losing it means re-pairing. + +--- + +## Step 6 β€” Smoke test + +Send yourself a WhatsApp message (from any chat you own, to your `LIS_OWNER_JID`). In another terminal: + +```bash +docker logs lis -f +``` + +You should see the webhook arrive, the conversation service invoke Anthropic, and a reply get posted back through GOWA. + +--- + +## Secret-matrix cheat sheet + +Values shared between the two `.env` files that **must be identical**: + +| `services/.env` | `lis/.env` | Purpose | +|---|---|---| +| `APP_BASIC_AUTH` | `GOWA_BASIC_AUTH` | Lis authenticates to GOWA's REST API | +| `WHATSAPP_WEBHOOK_SECRET` | `GOWA_WEBHOOK_SECRET` | HMAC on inbound webhook | +| `GOWA_DOMAIN` | host portion of `GOWA_BASE_URL` | Public URL Lis calls back on | + +Everything else is independent. + +--- + +## Backup / migration + +- **Managed Postgres** holds both databases β€” PITR is managed server-side, nothing to back up locally. +- **`services/gowa-data/`** β€” WhatsApp session. Tar before migrating hosts to skip re-pairing. +- **`services/data/`** β€” Caddy ACME state. Copy across hosts to avoid Let's Encrypt rate limits (optional; Caddy will re-issue on a fresh host). +- **`.env` files** β€” the only copies of your secrets. Back them up out-of-band (password manager, encrypted vault). + +## Gotchas + +- `GOWA_BASE_URL` in `lis/.env` points at the **public HTTPS** URL (through Caddy), not `http://gowa:3000`. Deliberate β€” keeps the basic-auth + TLS flow consistent β€” but means Lisβ†’GOWA traffic exits the host and comes back in through Caddy. +- The `proxy` network is non-external in **services** and external in **lis**. If you `docker compose down` the services stack, the Lis container refuses to start until it's back. +- Lis's container is memory-constrained (`mem_limit: 512m`); the GC env vars exist to respect that limit. Raising `mem_limit` without also relaxing `DOTNET_GCHeapHardLimit` is a footgun β€” raise both or neither. +- `LIS_EXEC_HOST=true` gives the agent shell access to the host. That's why the compose file has `pid: host` + `privileged: true`. If you don't want that, flip the env var to `false` and drop both flags. +- Only the `default` agent is seeded automatically. Create additional agents at runtime with `/agent new [display]`, or insert rows directly in the `agent` table if you prefer.