Response envelope schema validation + schema drift canary

[stackbilt-admin]

Problem

Our current resilience story (circuit breaker + provider failover + validateToolCalls) handles loud provider failures well, but leaves two gaps for the classic "API silently deprecated a field at 2am" scenario:

1. No response envelope schema validation. Each provider's parser reads fields like stop_reason, usage.input_tokens, choices[0].message.content via direct property access. If a provider renames/drops a field, we get an uncaught TypeError or a silently-undefined value propagating into LLMResponse. Failover rescues us, but only after a failure; the first requests hitting the broken shape either throw or return corrupt data.
2. No schema drift detection. We learn about breaking changes from production errors, not from canaries. There's no golden-sample comparison, no nightly probe, no changelog watcher.

The existing validateToolCalls (src/providers/base.ts:335) is the right pattern — drop malformed entries with a warning instead of crashing — but it only covers the tool_calls array. We should extend the same defensive-parse posture to the whole response envelope.

Proposal

Part 1 — Response envelope schema validation (per provider)

Add a zod (or valibot, lighter runtime) schema per provider describing the raw upstream response shape. Parse through it before field access in generateResponse. On validation failure:

• Log a structured warning with provider, model, and the offending path
• Emit a schema_drift hook event (see src/utils/hooks.ts) so observability can alert
• Throw a new SchemaDriftError that the circuit breaker/failover path treats like a transient provider error (so we fail over to a healthy provider instead of crashing the caller)

Files likely touched:

• src/providers/anthropic.ts, openai.ts, groq.ts, cerebras.ts, cloudflare.ts — add per-provider schema, parse before access
• src/errors.ts — add SchemaDriftError
• src/factory.ts — treat SchemaDriftError as fallback-eligible in getFallbackDecision
• src/utils/hooks.ts — new schema_drift event type

Part 2 — Schema drift canary

A separate opt-in module (src/utils/schema-canary.ts) that:

• Sends a minimal known-good request to each configured provider
• Captures the raw response
• Compares against a committed golden fixture (src/__tests__/fixtures/response-shapes/<provider>.json)
• Reports added/removed/renamed top-level and usage fields

Consumer-facing surface: a runSchemaCanary(providers) function that returns a diff report. Leave scheduling to the consumer (cron Worker, GitHub Action, whatever) — the library shouldn't own cadence.

Stretch: a scripts/update-golden-shapes.ts that refreshes fixtures after human review of a diff (never automatically).

Tests

• Unit: each provider's schema rejects known-bad shapes (missing usage, wrong type on content, etc.) and accepts current golden shapes
• Integration: factory fails over on SchemaDriftError like it does on ProviderError
• Canary smoke: golden shape comparison logic — fixture-driven, no network

Out of scope

• Changelog/RSS polling — separate concern, different cadence
• Auto-healing / auto-patching parsers — humans review drift diffs, period
• Schema validation of request payloads — we already validate in validateRequest

Priority

Medium. Current failover + circuit breaker means this is a defense-in-depth upgrade, not a hair-on-fire gap. But the cost of the 2am JSON parse error is high enough that being proactive wins.

[stackbilt-admin]

Slice 1 shipped — PR #40 merged (1a57f1f)

What landed

• SchemaDriftError class — non-retryable, code SCHEMA_DRIFT, statusCode 422 (Unprocessable Entity, not 502)
• validateSchema utility — zero-dep path-based validator with:
  • Primitive type matching (string | number | boolean | array | object | string-or-null)
  • Optional fields via optional: true
  • Discriminated-union array element validation via items.discriminator + items.variants
  • Forward-compat for unknown discriminator values (additive upstream changes don't break consumers)
  • Fail-fast semantics with structured path / expected / actual surfaced on drift
• Anthropic provider fully wired — validates envelope + nested content blocks (text, tool_use) before parsing
• Factory routing — SchemaDriftError is fallback-eligible, routes to next healthy provider
• onSchemaDrift observability hook — fires from factory with { provider, model, requestId, path, expected, actual }
• Package exports — SchemaDriftError, validateSchema, SchemaField, SchemaFieldType, SchemaDriftEvent all exported from package root

Tests: 201 → 212 (11 new). Covers validator primitives, Anthropic drift detection, factory fallback, hook firing, circuit breaker opening after repeated drift, end-of-chain propagation, security (no value leakage in error surface), nested content-block validation, forward-compat, optional fields.

CI green on Node 18/20/22. #42 closed in-PR.

---

Remaining work (follow-up PRs)

Part 1 — Response envelope schema validation

• openai — wire up using the items pattern for choices[0].message.tool_calls[]
• groq — same structure as openai
• cerebras — same structure as openai
• cloudflare — same structure as openai

Each provider is a ~50-line PR following the Anthropic template. Parallelizable.

Part 2 — Schema drift canary (src/utils/schema-canary.ts)

• Opt-in runSchemaCanary(providers) that sends known-good minimal requests, captures responses, and diffs against committed golden fixtures
• Fixtures directory: src/__tests__/fixtures/response-shapes/<provider>.json
• Return type: structured diff report with added/removed/renamed fields per provider
• Scheduling stays with the consumer (cron Worker, GitHub Action) — library doesn't own cadence
• Stretch: scripts/update-golden-shapes.ts for human-reviewed fixture refresh

Part 3 — Streaming path validation (#41)

• Separate concern, tracked in its own issue
• Each provider's streamResponse needs its own delta-shape schema + drift handling
• Stop swallowing JSON parse errors silently in SSE chunk handlers

Review findings deferred from PR #40

• L-1 — optional validateSchemaAll variant that collects all drifts instead of fail-fast (for debuggability in logging paths)
• L-2 — hook ordering doc: onRequestError fires before onSchemaDrift for the same incident
• L-3 — consolidate provider schemas into src/providers/schemas.ts or per-provider colocated exports (after slice 2 wire-ups land)

---

Closing criteria for #39

This parent issue closes when:

• All 5 providers have envelope + nested shape validation
• Streaming paths validated (Validate streaming response path against schema drift #41)
• Schema canary module shipped
• Documentation / README section on how to subscribe to onSchemaDrift

Keeping this issue open until then.