Improve logging: structured app-wide logging + optional external log shipping

[rado0x54]

Background

Current logs are dominated by health-check 200 request lines, while important subsystems (push notifications, WebAuthn signing flow, WebSocket lifecycle, MCP) produce little to no useful output. Debugging issues like the recurring iOS PWA push-notification disablement is hard because we can't see whether pushes were sent, accepted, or rejected (e.g. 410 Gone from APNs/FCM bridge).

Concrete recent example: notifications silently disable on the installed iPhone PWA. Logs show /api/push/subscribe being called on every app open (suggesting subscription eviction) and [WebAuthn Agent] Sign rejected … Signing request expired warnings, but nothing about whether web-push.sendNotification succeeded or what the push provider returned.

a) Structured app-wide logging

• Audit log call sites and add structured logs (with reqId correlation where applicable) for:
  • Push: subscribe / unsubscribe / send result (status, endpoint host, ttl) / dead-subscription cleanup on 404/410
  • WebAuthn signing flow: request created, delivered to client, resolved/expired/rejected (with timing)
  • WebSocket: connect / auth / disconnect / reconnect with reason
  • MCP: tool invocations and errors
  • Auth: login success/failure, session creation
• Demote noisy health-check / polling request logs (separate logger or level: debug, or filter from access log).
• Standardize on a single logger config (pino) with consistent fields (module, event, userId where safe).

b) Optional external log shipping

Make log destination configurable via env so a self-hosted aggregator can be plugged in. Candidates worth evaluating:

• Loki + Grafana (Grafana Labs) — lightweight, label-based, pairs naturally with the existing Grafana ecosystem. Probably the best fit.
• SigNoz — full observability (logs + traces + metrics), ClickHouse-backed.
• OpenObserve — single-binary, S3-backed, low resource use.
• Graylog — heavier but mature.

Likely approach: ship pino JSON logs to stdout (already the case) and let the operator wire up Promtail/Vector/Fluent Bit to forward to the chosen backend. Document the recommended setup in docs/.

Acceptance

• Triggering a push notification produces a clear log line including the provider response.
• Failed/expired subscriptions are logged and cleaned up from the DB.
• Health-check spam is no longer the dominant log output.
• README/docs describe how to forward logs to an external aggregator.

[rado0x54]

Adding (c): file-based logging with rotation

Pairing with (a) and (b), there's a third gap worth folding in: out-of-the-box support for writing logs to a file with rotation, parallel to what Caddy does (output file ... { roll_size, roll_keep }). Today ShellWatch's Fastify just emits pino JSON to stdout; for self-hosted operators not yet running a log aggregator, the only options are:

• Rely on Docker's json-file driver with max-size / max-file (works, but logs live inside /var/lib/docker/containers/<id>/... and are only accessible via docker logs).
• Pipe stdout through tee in the entrypoint (breaks rotation).
• Configure host-level logrotate against the docker storage path (gnarly, container-id paths shift).

None of these are first-class.

Proposal

Add an optional server.logFile config block:

server:
  logFile:
    path: /app/logs/shellwatch.log
    rollSize: 50MiB
    rollKeep: 10
    rollKeepFor: 720h

When set, route Fastify's pino output through pino-roll (or pino.transport({ targets: [...] }) for tee-style stdout + file). When unset, behavior unchanged (stdout only). Document the bind-mount pattern (./logs:/app/logs) in README alongside the existing reverse-proxy / config sections.

Why this matters

• Operators running a single VPS without Loki/etc. still get queryable, capped, persistent logs on the host filesystem — same model they're already using for Caddy access logs.
• Logs survive container recreation (which docker logs doesn't, depending on lifecycle).
• It's a stepping stone to (b): once logs are on disk in a known path, Promtail/Vector/Fluent Bit have something stable to tail.

Interim guidance

Until this lands, the recommended workaround for self-hosted deployments is the Docker json-file driver with rotation:

services:
  shellwatch:
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "10"
        compress: "true"

Worth mentioning in the README under "Deploying" so people don't end up with unbounded log growth before this is implemented.