[repo-monitor] Medium: robots.txt multi-agent group rules silently ignored

[Liohtml]

Summary

The robots.txt parser cannot handle grouped user-agent entries (multiple User-agent: lines before Disallow:), causing the scraper to silently ignore restrictions that apply to it.

Location

• File: src/spiders/robots.rs
• Line(s): 84–128

Severity

Medium

Details

For a robots.txt grouping multiple agents:

User-agent: MyBot
User-agent: Googlebot
Disallow: /private

The parser sets in_matching_section = true for MyBot, then encounters Googlebot and sets it to false, never processing Disallow: /private. This is a common robots.txt pattern.

Suggested Fix

Implement a two-pass parser: first collect all applicable agent group boundaries, then extract disallow rules. Or collect consecutive User-agent: lines before processing directives:

let mut current_agents: Vec<String> = Vec::new();
for line in content.lines() {
    if line.starts_with("User-agent:") {
        current_agents.push(/* extract agent */);
    } else if line.starts_with("Disallow:") && current_agents.contains(&target_agent) {
        // apply rule
    } else {
        current_agents.clear(); // new group starts
    }
}

---

Automated finding by repo-monitor

[Liohtml]

Analysis

Confirmed. src/spiders/robots.rs:84-128 — when parsing grouped User-agent: blocks, the parser ends the matching section when it encounters any User-agent: line:

// Current (broken) logic:
"User-agent" => {
    in_matching_section = value == agent || value == "*";
    // ^^^ WRONG: this sets in_matching_section = false for the second
    //     User-agent in a grouped block like:
    //     User-agent: MyBot
    //     User-agent: Googlebot
    //     Disallow: /private
}

When parsing User-agent: MyBot\nUser-agent: Googlebot\nDisallow: /private, after seeing User-agent: Googlebot, in_matching_section is set to false (assuming agent is MyBot), so Disallow: /private is never recorded.

The correct behavior per the robots.txt spec: a group of User-agent lines before any rules applies to all listed agents. Start accumulating rules only after all User-agent headers for the current block are processed:

// Track all agents for the current block, apply rules when the block ends
let mut current_block_agents: Vec<String> = Vec::new();
let mut current_block_rules: Vec<(bool, String)> = Vec::new();

for line in content.lines() {
    match (key, value) {
        ("User-agent", v) => {
            if !current_block_rules.is_empty() {
                // new block started, commit the previous one
                if current_block_agents.iter().any(|a| a == agent || a == "*") {
                    for (allow, path) in &current_block_rules {
                        // add to allow/disallow lists
                    }
                }
                current_block_agents.clear();
                current_block_rules.clear();
            }
            current_block_agents.push(v.to_string());
        }
        ("Disallow", path) => current_block_rules.push((false, path.to_string())),
        ("Allow", path) => current_block_rules.push((true, path.to_string())),
        _ => {}
    }
}
// Commit the last block

This is a well-documented parsing gotcha — most naive robots.txt parsers get this wrong.

Add a test case with a grouped User-agent block and verify rules apply correctly to both listed agents.

Effort: 45 min.