[repo-monitor] Medium: Unbounded response body read — OOM risk on large HTTP responses

[Liohtml]

Summary

resp.text().await reads the entire HTTP response body into memory with no size limit, allowing a server to cause OOM by returning a multi-gigabyte response.

Location

• File: src/fetchers/client.rs
• Line(s): 120

Severity

Medium

Details

let body_text = resp.text().await.unwrap_or_default();

There is no Content-Length check or maximum body size configured. A server returning a multi-gigabyte response would exhaust available RAM.

Suggested Fix

Add a size guard before reading:

const MAX_BODY: u64 = 50 * 1024 * 1024; // 50 MB
if let Some(len) = resp.content_length() {
    if len > MAX_BODY {
        return Err(FetcherError::RequestFailed("response too large".into()));
    }
}
let body_text = resp.text().await.unwrap_or_default();

---

Automated finding by repo-monitor

[Liohtml]

Analysis

Confirmed. src/fetchers/client.rs:120:

let text = resp.text().await.unwrap_or_default();

Two problems in one line:

1. No size cap — reads the entire body regardless of Content-Length. A server responding with gigabytes will exhaust heap memory.
2. .unwrap_or_default() silently swallows decoding errors — if the body isn't valid UTF-8 and charset detection fails, the entire response becomes an empty string with no log message.

The practical risk: Most scrapers target known sites, so OOM is unlikely in normal use. But spiders crawling untrusted URLs (the general case this library supports) could hit adversarial servers. The more common real-world failure is the silent empty-string behavior when scraping sites with non-UTF-8 encodings.

Fix — add a size guard and log the decoding error:

// In src/fetchers/client.rs

const MAX_RESPONSE_BYTES: usize = 50 * 1024 * 1024; // 50 MB, configurable

// Check Content-Length before reading (non-binding hint, but catches obvious cases):
if let Some(len) = resp.content_length() {
    if len > MAX_RESPONSE_BYTES as u64 {
        return Err(FetcherError::ResponseTooLarge(len));
    }
}

// Read with a hard cap:
let bytes = resp.bytes().await?;
if bytes.len() > MAX_RESPONSE_BYTES {
    return Err(FetcherError::ResponseTooLarge(bytes.len() as u64));
}
let text = String::from_utf8_lossy(&bytes).into_owned();
// Note: from_utf8_lossy replaces invalid UTF-8 with U+FFFD rather than failing silently

The MAX_RESPONSE_BYTES should be configurable via FetcherConfig — some spider use cases (e.g., large XML sitemaps) legitimately need large responses.

Effort: 30 min including adding ResponseTooLarge to FetcherError and a test.