Skip to content

[Conformance] Shared Cross-Language Test Suite Implementation #10

@tercel

Description

@tercel

Problem Description

While conformance/fixtures/ exists with JSON files (e.g., acl_evaluation.json, normalize_id.json), there isn't a standardized, automated runner to verify these across all SDKs (Python, TypeScript, Rust, Go, etc.) to ensure 100% behavioral parity.

Why solve it

Consistency is a core promise of apcore. A module defined in one language must behave identically in another. Manual verification is error-prone as the spec grows.

  • Interoperability: Ensuring consistent execution across different platforms.
  • Regression Prevention: Catching bugs early when updating SDKs or the core spec.
  • Trust: Providing a reliable 'Conformance Suite' for third-party SDK developers.

How to solve it

  1. Runner Framework: Create a test runner (likely in Rust or Python) that consumes conformance/fixtures/*.json.
  2. Standardized Interface: Define a 'Conformance Driver' (e.g., a simple CLI or JSON-RPC wrapper) for each SDK to implement.
  3. Automated CI: Integrate the runner into the CI pipelines of all official SDK repositories.
  4. Coverage: Expand the fixtures to cover more complex scenarios like the 11-step execution pipeline, streaming, and error propagation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions