test: pass-1 recovery + pass-2 LLM/API discipline#16
Merged
Conversation
Establishes a coverage floor and the targeted tests to back it: - pyproject.toml: [tool.coverage.run] + [tool.coverage.report] with fail_under = 77; pytest addopts="-ra" - .github/workflows/tests.yml: ubuntu × 3.11 runs --cov; XML artifact uploaded; other matrix slots stay test-only (faster CI fanout) - tests/README.md: how to run + interpret the coverage gate - tests/unit/test_benchmark.py: benchmark.py was at 10% line coverage; covers _percentile/_load_queries pure helpers and the CLI happy + error paths without spending API tokens - tests/unit/test_contracts.py: pins the public surface (attune_rag.__all__) so silent breakage in consumer layers (attune-gui/help/author) is caught at PR time - tests/unit/test_providers_base.py: pins the LLMProvider protocol + the two dataclasses concrete providers consume Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Layer 2 of test-strategy pass 2 (on top of recovered pass-1 commit): - pyproject.toml: register the `live` pytest marker; update addopts to deselect it by default (`-ra -m 'not live'`). Future tests that hit the real Anthropic API will be marked @pytest.mark.live and opted into via `pytest -m live` locally or the `live-eval` PR label in CI. - tests.yml: add a CI guard step that fails fast if ANTHROPIC_API_KEY is set in the default test environment. Regression alarm — GitHub Actions secrets aren't exposed unless explicitly named in env:, so this only fires if someone wires a secret into the wrong job. - tests/README.md: replace the inline LLM-mocking section with a pointer to `testing-conventions.md` in the attune workspace umbrella. Verified locally: 437 passing + 3 xpassed; coverage gate (`fail_under = 77`) holds. Spec: /Users/patrickroebuck/attune/specs/test-strategy-pass-2/ Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The forward-slash literal 'tests/golden' fails on Windows where Path uses backslashes. Switch to a path-component check that works on either OS. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Combines two bisectable commits:
test: add coverage gate + tests for benchmark, contracts, providers— recovers the pass-1 work that was authored into PR feat(test): coverage gate + close pass-1 gaps (test-strategy pass 1) #13 but got dropped during merge (the dashboard-v0.2.0 spec migration commit landed instead).chore(test): add live marker + CI guard for LLM/API discipline (pass 2)— pass-2 Layer 2 work on top.The two are kept as distinct commits so either can be reverted independently.
Commit 1 — pass-1 recovery (87 new tests; 79.94% branch cov; gate=77)
tests/unit/test_benchmark.pytests/unit/test_contracts.py__all__; pinsbuild_augmented_prompt+RagPipeline.runsignatures; pinsCitedSourceconsumer fieldstests/unit/test_providers_base.pytests/README.mdPlus:
pyproject.tomladdspytest-covwithbranch = trueandfail_under = 77;tests.ymlruns coverage on theubuntu × py3.11cell and uploadscoverage.xmlartifact.Commit 2 — pass-2 LLM/API discipline
pyproject.toml— registerlivemarker;addopts = "-ra -m 'not live'"tests.yml— add CI guard step (before tests) assertingANTHROPIC_API_KEYis unset for default suitetests/README.md— swap inline LLM section for a pointer to workspacetesting-conventions.mdDepends on workspace PR Smart-AI-Memory/attune#4 (testing-conventions.md) for the pointer to resolve.
Test plan
pytest --covlocally — 437 passing + 3 xpassed; 79.94% branch coverageruff checkcleanNote on PR #13 history
The earlier PR #13 ("docs(specs): split dashboard-v0.2.0 + receive template-path-rename") merged the spec-migration commit but dropped the test work. This PR re-applies it cleanly. Pass 1's task tracker has been updated to reflect this.
🤖 Generated with Claude Code