chore(test): live opt-in + rag-gate label gate + CI guards (pass 2 L4)#26
Merged
Conversation
Layer 4 of test-strategy pass 2 — the heaviest because attune-author
owns the real-API CI surface.
- pyproject.toml: update addopts to deselect `live` by default
(`-ra -m 'not live'`). The `live` marker was already registered
(pass 1 inheritance).
- tests.yml: add CI guard step that fails fast if ANTHROPIC_API_KEY
is set in the default test environment.
- cross-repo-compat.yml: add the same CI guard step (this workflow
also runs the default suite).
- rag-gate.yml: three changes
- Add `pull_request: types: [labeled]` trigger with a job-level
`if:` filter so the gate runs only when the `live-eval` label is
applied (in addition to the existing weekly schedule + manual).
Per spend caps in the Anthropic console, the worst case is bounded.
- Switch ANTHROPIC_API_KEY source to
`secrets.ATTUNE_CI_EVAL_KEY || secrets.ANTHROPIC_API_KEY` so the
rotation can happen before/after the new key is provisioned without
breaking either side.
- Add a `::notice::` cost-estimate log line at end of each run for
spend telemetry. v1 emits a fixed range; v2 will compute from
actual token counts once smoke_eval.py exposes them.
- tests/README.md: shrink the LLM-mocking section to a pointer at the
workspace `testing-conventions.md`; keep the reference-implementation
call-out so future contributors know this layer's conftest IS the
pattern.
Verified locally: 791 passing + 31 skipped (the 2 local failures are
environment-specific — they require the user's attune-help install
layout and pass cleanly in CI per Tests workflow status on main).
Spec: /Users/patrickroebuck/attune/specs/test-strategy-pass-2/
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Layer 4 of test-strategy pass 2 — heaviest layer because attune-author owns the real-API CI surface.
pyproject.toml—addoptsnow-ra -m 'not live'(live marker was already registered in pass 1)tests.yml+cross-repo-compat.yml— CI guard step assertingANTHROPIC_API_KEYis unset for default suiterag-gate.yml— three meaningful changes:pull_request: types: [labeled]trigger withif: github.event.label.name == 'live-eval'— opt-in PR gate so contributors can validate eval-affecting changes before merge without paying for every pushANTHROPIC_API_KEYsource tosecrets.ATTUNE_CI_EVAL_KEY || secrets.ANTHROPIC_API_KEYso the rotation works in either order::notice::Estimated cost: $0.10–0.30log line for spend telemetry (v1 = fixed range; v2 will use actual token counts)tests/README.md— shrink the LLM-mocking section to a pointer; keep the reference-implementation call-out so future contributors know attune-author's conftest IS the family patternPrerequisite
Depends on Smart-AI-Memory/attune#4 (workspace
testing-conventions.md).Procedural follow-ups (outside this PR — your action)
These are tracked in
specs/test-strategy-pass-2/tasks.mdProcedural section:attune-ci-evalin the consoleATTUNE_CI_EVAL_KEYrepository secret (Settings → Secrets → Actions)rag-gate.ymlmanually withreason: pass-2 verificationto confirm the new key path worksThe fallback to
secrets.ANTHROPIC_API_KEYmeans this PR can land before the new secret is provisioned without breaking the workflow.Test plan
pytest --covlocally — 791 passing, 31 skipped (2 env-specific local failures pass cleanly in CI per Tests workflow status on main)ruff checkclean (no source touched)rag-gate.ymldispatch succeeds with the new key pathlive-evallabel to a test PR, confirm gate fires🤖 Generated with Claude Code