fix(grading): strip markdown code fences from judge JSON responses

[thisisvk45]

Summary

The grading judge call sets response_format={"type": "json_object"}, which is honored natively by OpenAI and Gemini providers. For Anthropic, LiteLLM simulates this via prompt-level instruction rather than provider-side enforcement, and Claude models still wrap structured output in markdown code fences. The result: every grading attempt with a Claude judge fails parsing, retries 10 times producing identical fenced output, then raises ValueError("Invalid JSON after 10 attempts"). Trajectories grade as 0.0 regardless of agent quality.

This patch adds a small, idempotent strip_json_fences() helper applied at both judge-response parse sites. Bare JSON passes through unchanged, so behavior for Gemini and OpenAI users is unaffected.

Why this matters

Anyone running Archipelago with their own Anthropic key (the most common case for external researchers verifying leaderboard scores) hits this on the first task and assumes the agent failed when actually only the judge failed. The agent trajectory is correct; only the parse step breaks. This is plausibly a contributor to reproducibility discrepancies reported in #4 and #8 when reviewers attempt verification with different judge models.

Reproduction (before this fix)

1. Set examples/simple_task/grading_settings.json to any anthropic/claude-* model.
2. Run examples/simple_task/run.sh.
3. Observe grades.json showing final_score: 0.0 with the log line:

[JUDGE] JSON retry 1/10: 1 validation error for GradingResponseSchema
  Invalid JSON: expected value at line 1 column 1
  [type=json_invalid, input_value='```json\n{\n  "rationale"...,
  "is_criteria_true": true\n}\n```', input_type=str]

The JSON itself is valid; only the markdown wrapping breaks the parser.

Diagnosis

Two affected call sites in grading/runner/evals/output_llm/:

1. main.py:326 calls json.loads(raw_content) inside a 10-retry loop. Falls through to model_validate_json at line 339 which also fails.
2. negative_criteria.py:196 calls model_validate_json(neg_raw_content) with no retry loop. First fenced response is fatal.

The prompt at utils/prompts.py:253 says "Respond with a JSON object" but does not constrain markdown formatting. Anthropic models interpret this as "format JSON readably" and fence it. Gemini happens to comply with bare output.

Fix

New helper at grading/runner/evals/output_llm/utils/json_utils.py:

def strip_json_fences(text: str) -> str:
    """Strip Markdown code fences from an LLM JSON response.

    Idempotent: bare JSON passes through unchanged.
    """

Applied at both parse sites. ~5 production lines. No changes to LLM call interface, retry logic, or prompts.

Testing

• Added grading/tests/test_json_utils.py with 8 unit tests covering: bare JSON, fenced JSON (json and plain), trailing whitespace, empty input, whitespace-only input, idempotency, and the negative case where backticks appear inside legitimate JSON values.
• All 8 pass under pytest.
• Verified end-to-end on examples/simple_task with anthropic/claude-opus-4-5 agent and anthropic/claude-haiku-4-5-20251001 judge: trajectory.status: completed, final_score: 1.0.

Scope

Intentionally minimal. Does not change the prompt to ask Claude not to fence (provider-dependent and fragile), does not migrate to Pydantic structured output mode (more invasive, would conflict with negative_criteria.py's existing schema-based path), and does not touch the retry loop. The defensive parse helper is the smallest surface area that fixes the actual observed failure across all providers.

Files changed

• grading/runner/evals/output_llm/utils/json_utils.py (new, 26 lines)
• grading/runner/evals/output_llm/main.py (2 line changes + 1 import)
• grading/runner/evals/output_llm/negative_criteria.py (1 line change + 1 import)
• grading/tests/__init__.py (new, empty)
• grading/tests/test_json_utils.py (new, 8 tests)

[thisisvk45]

Verification update: ran the fix across the Claude judge family on examples/simple_task to confirm it is not Haiku-specific. Same agent (anthropic/claude-opus-4-5), same task, same harness build, only the judge model varies.

Judge model	final_score	runtime	trajectory status	notes
anthropic/claude-opus-4-5	1.0	38s	completed	clean parse
anthropic/claude-sonnet-4-5	1.0	41s	completed	clean parse
anthropic/claude-haiku-4-5-20251001	1.0	32s	completed	clean parse

All three judges return fenced JSON without the fix and parse cleanly with it. The 8 unit tests in this PR cover the helper, and these runs cover the integration path end-to-end across the Claude family.

Without the fix, every run in this table would have produced final_score: 0.0 with ValueError("Invalid JSON after 10 attempts"). The agent trajectory itself was identical across runs (same answer, same files touched), confirming the failure mode is judge-side, not agent-side.