Skip to content

feat(test): coverage gate + close pass-1 gaps (test-strategy pass 1)#13

Merged
silversurfer562 merged 1 commit into
mainfrom
test/coverage-and-gaps-pass-1
May 13, 2026
Merged

feat(test): coverage gate + close pass-1 gaps (test-strategy pass 1)#13
silversurfer562 merged 1 commit into
mainfrom
test/coverage-and-gaps-pass-1

Conversation

@silversurfer562
Copy link
Copy Markdown
Member

Summary

Pass 1 of the test-strategy spec — Layer 3 (attune-rag, the contract source of truth for the attune product family).

Note: the single commit on this branch is titled `docs(specs): split dashboard-v0.2.0 + receive template-path-rename` due to how it was authored, but its diff is purely the test-strategy work below. PR title and description reflect the actual content.

  • 87 new tests across 3 new files; total 437 passing + 3 xpassed
  • Coverage: 75.88% → 79.94% (branch coverage; +4.06pp)
  • Gate enforced at `fail_under = 77` (measure-then-lock at baseline-2pp)
  • CI workflow gains coverage step on the ubuntu × py3.11 cell only (1 of 12 matrix cells); other cells run plain pytest for speed

What changed

Coverage gate

  • `pytest-cov` configured in `pyproject.toml` with `branch = true`, `fail_under = 77`, and the standard `exclude_lines`.
  • `tests.yml` runs coverage only on `ubuntu-latest × python-3.11` and uploads `coverage.xml` as an artifact.
  • Stretch target for pass 2: 90% — rag is the API contract source of truth and warrants the highest gate of the four layers.

New test files

File Purpose Tests
`tests/unit/test_benchmark.py` Closes the biggest gap (benchmark.py was at 10% line coverage). CLI exit codes, _load_queries paths, _percentile math, _run_benchmark aggregation. 19
`tests/unit/test_contracts.py` Locks the public API: every `all` export importable + correct kind + has docstring. Pins `build_augmented_prompt` and `RagPipeline.run` signatures. Pins `CitedSource` consumer fields. 58
`tests/unit/test_providers_base.py` LLMProvider Protocol runtime-check semantics, dataclass invariants, default `generate_with_citations` NotImplementedError. 10

Notes

  • Task 7 (parametrize golden suite) was a no-op — the existing `tests/golden/test_golden.py` is already correctly table-driven via `pytest.param`. The audit's "hardcodes 40 queries" finding was stale.
  • New `tests/README.md` documents how-to-run, the LLM-mocking pattern (attune-author conftest is the reference), the public-API contract policy, and the golden suite policy.

Test plan

  • `pytest --cov` locally — 437 passing + 3 xpassed; 79.94% branch coverage
  • `ruff check` clean on new test files
  • CI matrix passes on ubuntu/macos/windows × py3.10–3.13
  • Coverage artifact uploads correctly on the py3.11 cell

Out of scope (deferred to subsequent passes)

  • `dashboard/show.py` (172 LOC, 0% covered) — biggest remaining gap; pass 2.
  • `dashboard/refresh.py` and `eval/bench_prompts.py` partial coverage — pass 2.
  • LLM/API-call discipline across layers — pass 2.
  • Slow / flaky test handling, CI duplication audit — pass 3.

🤖 Generated with Claude Code

dashboard-v0.2.0.md split into the standard 3-file structure
with statuses set to "complete" — feature shipped in 0.1.6
(target was 0.2.0; landed early). Drift from the original
locked plan (snapshot baked at render time vs MCP refresh,
new --open flag, new dashboard show subcommand) is documented
in design.md and tasks.md.

template-path-rename moves in from ~/attune/specs/ — the rag-side
work (_plan_template_path_rename / _apply_template_path_rename)
is what the spec primarily targets; co-located with the code.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@silversurfer562 silversurfer562 merged commit 54e78ef into main May 13, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant