YAML document delimiters are sometimes contained in eval results

[valerie-autumn-skye]

Some responses from the chat agent contain a series of three hyphens --- which are also used as YAML document delimiters. In the eval results, these are always indented with the text, so perhaps these are not a problem. Just getting this documented here in case an issue arises in the future.

Partial eval results from literature_mcp_eval:

- model: claude-4-sonnet
  coder: goose
  case_name: PMC4831113_Full_Text
  metric_name: CorrectnessMetric
  score: 0.9
  passed: true
  reason: The actual output fully includes the expected output, providing the exact
    text without contradiction. It goes beyond by offering additional details about
    the new species' characteristics. The only minor issue is the inclusion of a 'Description'
    section, which wasn't explicitly requested, but this aligns with evaluation step
    2 by providing more detail rather than omitting information.
  actual_output: 'I''ll help you get the Conclusions section of PMC4831113. Let me
    retrieve the full text of this paper first.

    ## Conclusions Section of PMC4831113


    Here is the **Conclusions** section from the paper "High-quality draft genome
    sequence and description of Haemophilus massiliensis sp. nov.":


    ---


    ### Conclusions


    On the basis of phenotypic, phylogenetic and genomic analyses, we formally propose
    the creation of **Haemophilus massiliensis** sp. nov. that contains strain FF7T
    (CSUR P859 = DSM 28247) which is the type strain. The strain was isolated from
    a peritoneal fluid specimen from a 44-year-old Senegalese woman admitted to Hôpital
    Principal in Dakar, Senegal.

[github-actions]

Issue Analysis: YAML Document Delimiters in Eval Results

Summary

This issue documents the presence of YAML document delimiter strings (---) appearing within chat agent responses in evaluation results from the literature_mcp_eval system. While potentially benign due to indentation, this warrants documentation for future reference and monitoring.

Technical Context

Current Evaluation Framework

Based on analysis of src/metacoder/evals/runner.py:410-417, the metacoder evaluation system saves results using:

yaml.dump(
    {"results": results_data, "summary": self.generate_summary(results)},
    f,
    default_flow_style=False,
    sort_keys=False,
)

Issue Characteristics

From the provided example:

• Model: claude-4-sonnet
• Coder: goose
• Case: PMC4831113_Full_Text
• Problem: --- appears indented within actual_output text content

Root Cause Analysis

Why This Happens

1. Agent Response Content: Chat agents (especially when processing academic literature) may naturally include content that resembles YAML syntax
2. Indentation Context: The delimiters appear indented within response text, not as structural YAML elements
3. Literary/Academic Content: Processing papers like PMC4831113 may trigger responses that include formatting patterns

Parsing Impact Assessment

Low Risk Factors:

• Delimiters are indented within string content, not at document root level
• YAML parsers typically require delimiters at column 0 for document separation
• The content appears within quoted/escaped string values in the YAML structure

Potential Concerns:

• Could confuse manual inspection of results
• May complicate automated parsing if not properly handled
• Could indicate underlying prompt engineering issues

Recommendations

1. Monitoring Strategy

• Track frequency of occurrence across different models/coders
• Monitor for patterns correlating with specific paper types or content domains
• Log cases where delimiters appear at problematic positions (column 0)

2. Technical Mitigations

Consider implementing content sanitization options:

def sanitize_output_for_yaml(text: str) -> str:
    """Optionally sanitize problematic YAML patterns in agent output."""
    # Replace standalone document delimiters with safe alternatives
    return re.sub(r'^---\s*$', '\\-\\-\\-', text, flags=re.MULTILINE)

3. Evaluation Framework Enhancement

• Add validation step to detect structural YAML conflicts
• Include delimiter frequency in evaluation metadata
• Consider alternative serialization formats (JSON) for problematic cases

4. Documentation Enhancement

• Add this issue to evaluation troubleshooting guide
• Document expected vs. problematic delimiter patterns
• Include guidance for manual result inspection

Related Research Context

The case involves PMC4831113, which appears to be biomedical literature that may contain structured data formats, potentially explaining the presence of delimiter-like patterns in generated responses.

Future Monitoring

This issue serves as a baseline for tracking:

• Evolution of this behavior across model versions
• Effectiveness of any implemented mitigations
• Correlation with specific content types or evaluation scenarios

Classification: Documentation/Monitoring
Priority: Low (monitoring recommended)
Impact: Minimal (current indentation provides natural protection)