feat(evals): seed Phase 4a query set + sentinel + thresholds

[EtanHey]

DRAFT seed PR — drops the data layer of the Phase 4a eval framework. 80 queries across 6 categories. Runner/metrics/ci_gate land in subsequent commits.

Files

• evals/phase4a/queries.yaml — 80 queries (15 Hebrew + 12 health + 15 conceptual + 15 frustration + 8 temporal + 15 entity)
• evals/phase4a/sentinel.yaml — 5 pre-commit smoke queries
• evals/phase4a/thresholds.yaml — recall@20 ≥90% per category, ndcg@10 ≥0.85
• evals/phase4a/README.md — structure + next-iteration plan

Blocks Phase 4b

Per PHASE4-DESIGN.md §5: "Lands BEFORE any retrieval-shape change."

Why DRAFT

Runner/metrics/ci_gate TODO. Mark ready-for-review once those land. Query data shipping first allows independent review of Hebrew spellings, entity names, and frustration phrasing.

🤖 Generated with Claude Code by orcClaude-successor s:42 during autonomous-mode 6-hour window

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Note

Seed Phase 4a eval query set, sentinel, and thresholds

• Adds 80 evaluation queries across six categories (hebrew, health, conceptual, frustration, temporal, entity) in queries.yaml, each with a stable ID and optional per-category expectations.
• Adds a sentinel.yaml with 5 fast-running queries (one per category) intended for pre-commit smoke checks.
• Adds thresholds.yaml specifying per-category recall@20 minima, nDCG@10, regression allowance vs baseline, and latency budgets.
• README.md documents the framework design, category weighting, sentinel concept, and the plan for Phase 4b runner code.

📊 Macroscope summarized ba6cc77. 4 files reviewed, 1 issue evaluated, 0 issues filtered, 1 comment posted

🗂️ Filtered Issues

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: d54e06d3-81da-473b-8f7b-79a2a3d22dbe

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/phase-4a-queries-seed

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[macroscopeapp]

🟡 Medium phase4a/README.md:47

The sentinel.yaml section claims it contains "One representative from each category," but with 6 categories and only 5 queries, the temporal category has no sentinel coverage. A reader relying on this description would incorrectly believe temporal regressions are caught by the pre-commit gate. Either add a 6th temporal sentinel query, or update the description to accurately state which categories are covered.

Suggested change

	5 queries that run in <30s total for pre-commit hooks. One representative from each category. All must pass; CI workflow blocks if any sentinel fails.
	+5 queries that run in <30s total for pre-commit hooks. One representative from each category except `temporal`. All must pass; CI workflow blocks if any sentinel fails.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file evals/phase4a/README.md around line 47:

The `sentinel.yaml` section claims it contains "One representative from each category," but with 6 categories and only 5 queries, the `temporal` category has no sentinel coverage. A reader relying on this description would incorrectly believe temporal regressions are caught by the pre-commit gate. Either add a 6th temporal sentinel query, or update the description to accurately state which categories are covered.

[macroscopeapp]

🟡 Medium phase4a/metrics.py:39

When score is 0.0 or negative, _is_relevant incorrectly returns True for chunks that have a chunk_id but no expectations. The docstring states heuristic #2 requires "score>0", but line 42's fallback condition doesn't verify the score value, so zero/negative scores pass through as relevant when they should not.

Consider adding a score check to line 42, or merge this case into the preceding branch by changing line 39's condition to score >= 0 if the intent is to accept any non-negative score when no expectations are set.

-    if score is not None and isinstance(score, (int, float)) and score > 0:
+    if score is not None and isinstance(score, (int, float)) and score >= 0:

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file evals/phase4a/metrics.py around line 39:

When `score` is `0.0` or negative, `_is_relevant` incorrectly returns `True` for chunks that have a `chunk_id` but no expectations. The docstring states heuristic #2 requires "score>0", but line 42's fallback condition doesn't verify the score value, so zero/negative scores pass through as relevant when they should not.

Consider adding a score check to line 42, or merge this case into the preceding branch by changing line 39's condition to `score >= 0` if the intent is to accept any non-negative score when no expectations are set.

Evidence trail:
evals/phase4a/metrics.py lines 16-45 at REVIEWED_COMMIT: docstring line 22 states 'score>0' requirement for heuristic #2; line 39 checks `score > 0` but line 42 provides an unchecked fallback that returns True when chunk_id is present and no expectations are set, allowing score=0.0 or negative scores through.

[macroscopeapp]

🟠 High phase4a/metrics.py:89

aggregate_category calls recall_at_n(chunks, query_meta, n=20) and ndcg_at_n(chunks, query_meta, n=10), but chunks is built from top_5_chunk_ids which contains at most 5 items. The slicing chunks[:n] silently returns all available chunks (≤5), so the function computes recall@5 and ndcg@5 instead of the claimed recall@20 and ndcg@10. Either pass the full result set to metrics functions, or change the labels to reflect the actual computation.

-        # results.json format from runner.py: each result has top_5_chunk_ids + top_5_scores
+        # results.json format from runner.py: each result has top_N_chunk_ids + top_N_scores
         # Reconstruct chunks list for metric computation
         chunks = []
-        for cid, score in zip(r.get("top_5_chunk_ids", []), r.get("top_5_scores", [])):
+        for cid, score in zip(r.get("top_20_chunk_ids", []), r.get("top_20_scores", [])):
             chunks.append({"chunk_id": cid, "score": score})

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file evals/phase4a/metrics.py around lines 89-106:

`aggregate_category` calls `recall_at_n(chunks, query_meta, n=20)` and `ndcg_at_n(chunks, query_meta, n=10)`, but `chunks` is built from `top_5_chunk_ids` which contains at most 5 items. The slicing `chunks[:n]` silently returns all available chunks (≤5), so the function computes recall@5 and ndcg@5 instead of the claimed recall@20 and ndcg@10. Either pass the full result set to metrics functions, or change the labels to reflect the actual computation.

Evidence trail:
evals/phase4a/runner.py line 150: `top_5_chunk_ids` is `chunks[:5]` (at most 5 items). evals/phase4a/metrics.py lines 98-100: `chunks` built from `top_5_chunk_ids` and `top_5_scores`. evals/phase4a/metrics.py line 105: `recall_at_n(chunks, query_meta, n=20)` called with n=20. evals/phase4a/metrics.py line 106: `ndcg_at_n(chunks, query_meta, n=10)` called with n=10. evals/phase4a/metrics.py lines 57, 73: `top_n = chunks[:n]` slicing on ≤5 items. evals/phase4a/metrics.py lines 111-112: output labeled `recall_at_20_mean` and `ndcg_at_10_mean`.

[macroscopeapp]

🟡 Medium phase4a/metrics.py:85

The IDCG calculation at line 85 assumes binary relevance of 1.0 for each position, but the DCG calculation uses weighted relevance values (rel = max(0.0, min(1.0, float(score)))). This mismatch means ndcg_at_n returns a value less than 1.0 even with a perfect ranking when chunk scores are below 1.0, contradicting the standard NDCG definition where IDCG must use the same relevance model as DCG. Consider computing IDCG by summing the actual relevance values of relevant chunks in descending order, not hardcoded 1.0 values.

-    idcg = sum(1.0 / math.log2(i + 2) for i in range(relevant_in_top_n))
+    idcg = 0.0
+    for i in range(relevant_in_top_n):
+        chunk = top_n[i]
+        score = chunk.get("score")
+        rel = 1.0
+        if isinstance(score, (int, float)):
+            rel = max(0.0, min(1.0, float(score)))
+        idcg += rel / math.log2(i + 2)

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file evals/phase4a/metrics.py around line 85:

The IDCG calculation at line 85 assumes binary relevance of `1.0` for each position, but the DCG calculation uses weighted relevance values (`rel = max(0.0, min(1.0, float(score)))`). This mismatch means `ndcg_at_n` returns a value less than `1.0` even with a perfect ranking when chunk scores are below `1.0`, contradicting the standard NDCG definition where IDCG must use the same relevance model as DCG. Consider computing IDCG by summing the actual relevance values of relevant chunks in descending order, not hardcoded `1.0` values.

Evidence trail:
evals/phase4a/metrics.py lines 74-86 at REVIEWED_COMMIT: DCG uses `rel = max(0.0, min(1.0, float(score)))` (line 80), IDCG uses hardcoded `1.0` (line 85). Standard NDCG definition requires IDCG to use the same relevance values sorted in ideal order.