[repo-monitor] High: robots.txt fetched with no timeout — hangs indefinitely, ignores FetcherConfig

[Liohtml]

Summary

fetch_robots uses reqwest::get() which creates a default HTTP client with no timeout and ignores all FetcherConfig settings (proxy, TLS, custom headers), potentially hanging the entire crawl setup indefinitely.

Location

• File: src/spiders/robots.rs
• Line(s): 25

Severity

High

Details

let rules = match reqwest::get(&url).await {

reqwest::get creates a brand-new default client with no timeout. If the target's robots.txt endpoint hangs, the entire crawl setup blocks forever. This client also ignores any proxy, TLS certificate validation settings, and custom headers configured in FetcherConfig.

Suggested Fix

Use a purpose-built client with a short timeout:

let client = reqwest::Client::builder()
    .timeout(Duration::from_secs(10))
    .build()
    .unwrap_or_default();
let rules = match client.get(&url).send().await {

Or reuse the spider's existing Fetcher instance.

---

Automated finding by repo-monitor

[Liohtml]

Analysis

Confirmed. This is a practical P1 — it blocks real crawls.

src/spiders/robots.rs:25:

let resp = reqwest::get(&url).await  // brand-new default client, no timeout, no proxy, no TLS config
    .map_err(...)?;

Problems with this call:

1. No timeout — default reqwest::Client has no timeout. A server that accepts the TCP connection but never sends a response hangs CrawlerEngine::crawl() indefinitely, before the first URL is even scraped.
2. Ignores the spider's proxy config — the spider may be configured to route all traffic through a proxy. robots.txt check bypasses it, potentially revealing the real IP to the target server before the actual crawl begins.
3. Ignores FetcherConfig — TLS settings, custom headers, user-agent are all lost.

Fix — reuse the spider's Fetcher:

The cleanest approach is to pass a reference to the spider's Fetcher (or FetcherConfig) into RobotsTxtManager:

// src/spiders/robots.rs
impl RobotsTxtManager {
    pub async fn is_allowed_via_fetcher(
        &self,
        url: &str,
        agent: &str,
        fetcher: &Fetcher,
    ) -> bool {
        // use fetcher.get(robots_url).await with existing config + timeout
    }
}

If passing Fetcher feels like too much coupling, at minimum add an explicit timeout:

let resp = tokio::time::timeout(
    Duration::from_secs(10),
    reqwest::get(&url)
).await
.unwrap_or_else(|_| {
    tracing::warn!("robots.txt fetch timed out for {}, allowing all", url);
    return Ok(..); // default-allow on timeout
})?;

I'd lean toward the Fetcher approach — it's one parameter change and eliminates the whole class of "robots.txt client doesn't match crawl client" bugs.

Effort: 30-45 min.