FEAT: Updating Scorer Metrics Update Workflow and SelfAskRefusalScorer update

[rlundeen2]

Updates refusal scorer and shows how to do end to end metrics updates.

Many scorers depend on some core scorer — objective scorers wrap a SelfAskRefusalScorer inside TrueFalseInverterScorer or
TrueFalseCompositeScorer. When the refusal prompt changes (or a new variant performs better), every dependent scorer's
identity hash changes too, invalidating all their cached metrics. This means re-running evaluations for the entire scorer
suite — a process that takes hours — just because the underlying refusal prompt was tweaked. There was no way to evaluate
refusal variants in isolation, pick the best one, and then selectively re-evaluate only the scorers affected by that
change.

This PR introduces tag-based evaluation and auto-detected dependencies so you can iterate on refusal scorers quickly
without re-running everything.

It also updates the refusal scorers and refusal scorer metrics. GPT-5 refused scores in a different way, making our existing refusal scorers less accurate. So this is the consequence of working through that flow.

Refusal Scorer Human Dataset Updates

First, we ran some refusal tests against gpt-5 and evaluated refusals. Many of these were different from previous models. We added many of these to the human-labeled datasets in both refusal and objective scores. We also trimmed down the refusal dataset to a more manageable size, reducing repetitive samples.

We bumped the dataset version making all previous metrics invalid.

Refusal Scorer update

Replaces the single default refusal scorer with 4 named variants, adds auto-dependency detection so wrapper scorers automatically use the best-performing refusal prompt, and provides tag-based batch evaluation via evaluate_scorers.py

• Replace refusal_default.yaml and refusal_strict.yaml with 4 explicit variants:
• OBJECTIVE_BLOCK_SAFE / OBJECTIVE_ALLOW_SAFE — require an explicit objective
• NO_OBJECTIVE_BLOCK_SAFE / NO_OBJECTIVE_ALLOW_SAFE — work with or without an objective
• Each variant is registered in ScorerInitializer with a refusal tag
• Expanded human-labeled refusal evaluation dataset (refusal.csv + refusal_extra.csv)
• Added objective evaluation datasets: chemical/biological, cyber, harassment, illegal activity, misinformation, phishing

Scorer Registry Initializer Update

Dynamic Best-Refusal Selection; now the registry checks for the best refusal scorer, and uses that for all other metrics that need a refusal scorer

• _register_best_refusal_f1() reads existing metrics from refusal_metrics.jsonl and tags the best variant as
  best_refusal_f1 / default_refusal_scorer
• Dependent scorers (TrueFalseInverterScorer, composites) are built using the best refusal path
• Falls back to OBJECTIVE_ALLOW_SAFE when no metrics exist yet
• ComponentIdentifier._collect_child_eval_hashes() recursively collects eval hashes from child identifiers
• BaseInstanceRegistry.find_dependents_of_tag() discovers wrapper/composite scorers that embed a tagged scorer, using
  eval_hash matching — no explicit depends_on declaration needed

Evaluation of different components

• evaluate_scorers.py now accepts --tags CLI filter (e.g., --tags refusal)
• ScorerInitializer phases: register refusal variants → pick best → build dependents → register remaining → pick best
  objective
• Documented Recommended workflow: --tags refusal first, then full re-run

(credit: @fdubut for help)