[repo-monitor] High: allowed_domains whitelist bypassed when URL has no parseable host

[Liohtml]

Summary

When request.domain() returns None (malformed URL, data: scheme, file://), the allowed_domains whitelist check is silently skipped and the request is allowed through.

Location

• File: src/spiders/engine.rs
• Line(s): 270–278

Severity

High

Details

if !allowed.is_empty() {
    if let Some(domain) = request.domain() {
        if !allowed.contains(&domain) { /* rejected */ }
    }
    // If domain() is None, request is silently ALLOWED!
}

A spider configured with allowed_domains could fetch local files (file://) or internal network resources if follow-URLs include non-HTTP schemes.

Suggested Fix

Reject requests when the domain cannot be parsed:

if !allowed.is_empty() {
    match request.domain() {
        Some(domain) if allowed.contains(&domain) => {} // allowed
        _ => {
            stats.lock().await.offsite_requests_count += 1;
            return; // reject unknown or unparseable domains
        }
    }
}

---

Automated finding by repo-monitor

[Liohtml]

Analysis

Confirmed. This is the highest-priority fix in the repo.

src/spiders/engine.rs:272-278 (approximately):

if let Some(domain) = request.domain() {
    if !spider.allowed_domains().is_empty()
        && !spider.allowed_domains().contains(&domain) {
        // blocked
    }
}
// Implicit else: request.domain() == None → always passes through

A file://, data:, or malformed URL returns None from domain(), and the entire whitelist check is silently skipped. The spider's allowed_domains configuration becomes a lie.

Two things to fix:

1. Reject on None domain when allowed_domains is non-empty:

match request.domain() {
    None if !spider.allowed_domains().is_empty() => {
        tracing::warn!("Rejecting request with unparseable domain: {}", request.url());
        continue; // or return/skip
    }
    None => { /* allowed_domains is empty → allow */ }
    Some(domain) => {
        if !spider.allowed_domains().is_empty()
            && !spider.allowed_domains().contains(&domain) {
            continue;
        }
    }
}

2. Add a test for this: The test suite has tests/spiders_scheduler.rs but no test for engine-level domain filtering. Add one:

// A spider with allowed_domains = ["example.com"]
// Enqueue a request with URL "file:///etc/passwd"
// Assert the engine does not process it

This is one of those bugs that's easy to miss in code review (the if let pattern looks correct at a glance) but has real security implications for anyone using allowed_domains as a crawl scope fence.

Effort: 15 min for the fix, 30 min for the test.