chore: update docs for judge panel feature

[asamal4]

Description

update docs for judge panel feature including llm pool

Type of change

• Refactor
• New feature
• Bug fix
• CVE fix
• Optimization
• Documentation Update
• Configuration Update
• Bump-up service version
• Bump-up dependent library
• Bump-up library or tool used for development (does not change the final image)
• CI configuration change
• Unit tests improvement

Tools used to create PR

Identify any AI code assistants used in this PR (for transparency and review context)

• Assisted-by: Cursor

Related Tickets & Documents

• Related Issue #
• Closes #

Checklist before requesting a review

• I have performed a self-review of my code.
• PR has passed all pre-merge test jobs.
• If it is a core feature, I have added thorough tests.

Testing

• Please provide detailed steps to perform tests related to this code change.
• How were the fix/results from this change verified? Please provide relevant screenshots or results.

Summary by CodeRabbit

• New Features

  • Panel of Judges: run multiple LLM judges concurrently with configurable aggregation (max implemented).
  • Per-judge token tracking for detailed cost analysis.

• Documentation

  • Expanded guides with llm_pool and judge_panel examples and migration/deprecation notes for legacy config.
  • Notes on benefits (reduced bias, robustness) and current limitations (aggregation and threshold support).

[coderabbitai]

Walkthrough

Adds documentation for a new Panel of Judges and an LLM pool: README, Evaluation Guide, and Configuration docs now describe configuring multiple judge LLMs, aggregation strategy (max), per-judge token tracking, examples, and limitations.

Changes

Cohort / File(s)	Summary
Top-level README README.md	Notes Panel of Judges feature and per-judge token usage tracking.
Evaluation guide docs/EVALUATION_GUIDE.md	New "Panel of Judges (Advanced)" section: introduces llm_pool and judge_panel config blocks, YAML examples, explanation of benefits and current limitations (only max aggregation implemented), and extended evaluation data examples.
Configuration docs docs/configuration.md	Adds LLM Pool and Judge Panel sections with defaults, model entries, judges list, enabled_metrics and aggregation_strategy usage examples and sample aggregated output structure.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The PR title clearly and concisely describes the main change: documentation updates for the judge panel feature, which aligns with the substantive content (README, EVALUATION_GUIDE, and configuration.md updates).
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

You can disable the changed files summary in the walkthrough.

Disable the reviews.changed_files_summary setting to disable the changed files summary in the walkthrough.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/configuration.md`:
- Around line 28-37: The documented schema for llm_pool.defaults is incorrect:
temperature and max_completion_tokens are shown directly under llm_pool.defaults
but the code expects them under llm_pool.defaults.parameters; update the example
and table to show that defaults are nested under a parameters object (e.g.,
llm_pool.defaults.parameters.temperature and
llm_pool.defaults.parameters.max_completion_tokens) and adjust the table entry
for `llm_pool.defaults.*` to reflect the nested structure (cache_dir at the same
level if applicable or move it under parameters if the schema requires it) so
the YAML example matches the expected config validation.

In `@docs/EVALUATION_GUIDE.md`:
- Around line 727-731: The defaults block in the llm_pool example uses a flat
shape for parameters; move temperature and max_completion_tokens under
defaults.parameters so the config matches the expected schema (referenced as
llm_pool.defaults and defaults.parameters) and ensure cache_dir remains directly
under defaults while temperature and max_completion_tokens are nested beneath
the parameters object to avoid invalid config errors.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a0881e82-3c75-4f2a-b249-90c9da069a67

📥 Commits

Reviewing files that changed from the base of the PR and between 59749df and ab61a2b.

📒 Files selected for processing (3)

• README.md
• docs/EVALUATION_GUIDE.md
• docs/configuration.md

[coderabbitai]

🧹 Nitpick comments (1)

README.md (1)

13-13: Clarify strategy support status in README feature bullet.

This line reads as if multiple aggregation strategies are fully supported today. Consider adding a short qualifier (e.g., “currently max is implemented”) to match runtime behavior and avoid confusion.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@README.md` at line 13, Update the "Panel of Judges" feature bullet to clarify
which aggregation strategies are actually implemented at runtime by appending a
short qualifier (for example: "currently `max` is implemented") so readers don't
assume multiple strategies are available; locate the "Panel of Judges" bullet in
README.md and modify that line to explicitly mention the supported aggregation
strategy (e.g., `max`) and, if helpful, a brief note that others are planned.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@README.md`:
- Line 13: Update the "Panel of Judges" feature bullet to clarify which
aggregation strategies are actually implemented at runtime by appending a short
qualifier (for example: "currently `max` is implemented") so readers don't
assume multiple strategies are available; locate the "Panel of Judges" bullet in
README.md and modify that line to explicitly mention the supported aggregation
strategy (e.g., `max`) and, if helpful, a brief note that others are planned.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 50ce57f6-5e6e-478a-8b90-aa005b619ede

📥 Commits

Reviewing files that changed from the base of the PR and between ab61a2b and 4cdee66.

📒 Files selected for processing (3)

• README.md
• docs/EVALUATION_GUIDE.md
• docs/configuration.md

🚧 Files skipped from review as they are similar to previous changes (1)

• docs/EVALUATION_GUIDE.md