Proposal: Extensibility hooks for tracing, metrics, retries, rate limiting, pagination & auth providers

[dougborg]

Summary

Open discussion/proposal for a cohesive extensibility layer to support upcoming runtime features: tracing / instrumentation, metrics, retries (#102), rate limiting (#103), pagination helpers (#104), configurable timeouts (#105), richer auth & pluggable security schemes (#106), and enhanced HTTPException context (#100). Several of these have individual issues already; this proposal focuses on the unifying mechanism rather than each feature’s business logic.

Motivation

Right now generated clients perform a straightforward request → status check → parse path. As we add resilience and observability features we risk:

• Bloated per-endpoint templates (harder to maintain & test)
• Divergent behavior between libraries (httpx / requests / aiohttp)
• Limited ability for users to inject custom cross-cutting concerns (custom headers, distributed tracing spans, OpenTelemetry metrics, circuit breakers, caching, additional auth flows, mock/test shims)

A small, well-defined lifecycle contract can unlock these without turning the generator into a framework.

Goals

1. Provide a minimal, composable mechanism to run user- or framework-supplied code at key points of an operation.
2. Keep generated method bodies lean (readability + diff friendliness / regression stability).
3. Support both synchronous (requests) and asynchronous (httpx/aiohttp) workflows with parallel semantics.
4. Avoid hard dependency on heavy tracing libs (e.g. OpenTelemetry) while making integration trivial.
5. Preserve current zero-config simplicity and performance when features not enabled (nearly zero overhead path).

Non-Goals (initially)

• Full middleware micro-framework
• Dynamic runtime loading of plugins via entry points
• In-process caching / circuit breaker implementations (can be added later using same hook scaffold)
• Rewriting transport layer (we leverage existing http clients)

Proposed Design Options

Option A: Lifecycle Callback Set

Generate an OperationHooks (or ClientHooks) dataclass with optional callables:

• before_request(context) (inspect/modify request – method, url, headers, body)
• after_response(context) (inspect response before parsing)
• on_error(context) (enrich / transform exception)
• build_retry_state(context) (optional state container reused across attempts)

Context would carry:

• operation_id, tag, path_template
• request: method, url, headers, query params, path params, raw body (pre-serialization & post-serialization?)
• attempt: current attempt number
• client_config: reference to APIConfig
• (after response) response: status, headers, body bytes (lazy streamed?)

Pros: Minimal, explicit. Cons: Harder to chain multiple independent concerns unless user wraps them manually.

Option B: Ordered Policy / Middleware Pipeline

Modelled loosely after Azure SDK or HTTPX custom dispatchers: a list of policies each exposing async/sync send(context, next).

• Generator wraps core transport into final policy.
• Policies can short-circuit, retry, mutate request & response.
• Retries, rate limiting, timeouts become first-class reusable policies.

Pros: Composable, scalable for future features. Cons: Slightly more abstraction; more boilerplate for simple customization.

Option C: Event Bus + Subscribers

Emit simple events (request.start, request.finish, request.error, retry.scheduled, etc.) with a lightweight dispatcher.

Pros: Very low friction to listen; multiple subscribers easy. Cons: Mutating request/response safely needs conventions; ordering ambiguities.

Recommended Direction

Adopt Option B (Policy Pipeline) internally, while exposing a convenience layer that auto-builds policies from simple callbacks (covering Option A ergonomics). This hybrid keeps advanced extensibility without forcing complexity on casual consumers.

Minimal Initial Scope

1. Introduce a BasePolicy interface (sync & async variants) and a Pipeline orchestrator generated once per library.
2. Implement built-in policies (behind flags / config):
   • TimeoutPolicy (maps per-operation or global default)
   • RetryPolicy (basic exponential backoff; uses status code / network error classification)
   • RateLimitPolicy (token bucket or simple sleep when >N in interval) – optionally deferred to phase 2
   • AuthPolicy (wraps existing bearer token logic; omits Authorization header if token is None) – incremental step toward Enhancement: Flexible Auth Handling (omit empty Bearer, support API keys & securitySchemes) #106
   • TracingPolicy (no-op stub; user can subclass to start spans)
3. Generated service methods become ~:

   return await self._pipeline.send(request)

   Where request is a small object (method, url, headers, query, json/data, expected_statuses, parse_json=bool, model parser callback).
4. Response parsing & HTTPException enrichment (Enhance generated HTTPException: rich context, parsed error models, configurable handling #100) moved into a ResponseParsingPolicy to keep templates tiny.

Performance Considerations

• When only the parsing policy is present (default), overhead is one or two function calls over current code path.
• Users opting in to retries / tracing accept added indirection intentionally.
• Could add micro-benchmarks later to ensure 95th percentile overhead < ~5% for simple operations.

Testing Strategy

• Unit tests per policy (retry branch coverage, timeout edge cases, token absence => header omitted, 204 skip logic retained).
• Golden output tests ensure generated methods shrink rather than grow in complexity.
• Property-based tests (Schemathesis) later leverage retry / timeout policies for resilience validation (Enhancement: Comprehensive Validation Strategy for Generated Clients (mock server, schema validation, property-based, golden requests) #101).

Migration / Backwards Compatibility

• Keep current public surface (e.g., method signatures). Policies internal unless user opts in by supplying a list through APIConfig (new field policies: list[Policy] | None).
• Default behavior identical; Authorization header omission when token is None is benign improvement, documented in CHANGELOG.

Open Questions / Feedback Sought

1. Does the community prefer a callback-only model (simpler) vs full pipeline? Any strong reasons against the pipeline?
2. Should retries and timeouts be included in the first cut, or stage timeouts first then layer retries (Enhancement: Built-in Retry Strategy for Generated Clients #102) for simpler initial review?
3. Should pagination helpers (Enhancement: Pagination Helper Abstractions (header & payload based) #104) be implemented as a policy (yielding items) or as generated paginator wrapper functions referencing the pipeline?
4. Is an event emission mechanism still desired in addition (for passive metrics) or should that be layered later via a TracingPolicy emitting events/spans?
5. Any edge cases in existing generators that would be harder to port into a pipeline abstraction we should call out early?

Alternatives Considered

• In-template inline logic (status quo): Simpler now, exponential complexity later.
• Externally maintained wrapper/adapter: Puts burden on users to monkeypatch generated code.
• Hard dependency on OpenTelemetry SDK: Increases weight & version friction.

Next Steps (if accepted)

1. Agree on interface shape (finalize naming & minimal request/response context objects).
2. Implement core interfaces + no-op pipeline in a small PR.
3. Migrate response parsing & error enrichment to policy (HTTPException enhancements Enhance generated HTTPException: rich context, parsed error models, configurable handling #100) in subsequent PR.
4. Add TimeoutPolicy (Enhancement: Configurable Timeouts (global + per-call, sync & async) #105) + omit Authorization if token None.
5. Add RetryPolicy (Enhancement: Built-in Retry Strategy for Generated Clients #102) reusing timeout boundaries.
6. Evaluate RateLimitPolicy (Enhancement: Automatic Rate Limit Handling (429 & standard rate limit headers) #103) & pagination helpers (Enhancement: Pagination Helper Abstractions (header & payload based) #104) in later PRs.

---

Please share thoughts, concerns, preferences (Option A/B/C, hybrid, naming) and any must-have hooks we’ve overlooked before we lock in the baseline.