Skip to content

feat: Add validate_response option#74

Open
vgavro wants to merge 5 commits into
will-ockmore:mainfrom
vgavro:feat/validate_response
Open

feat: Add validate_response option#74
vgavro wants to merge 5 commits into
will-ockmore:mainfrom
vgavro:feat/validate_response

Conversation

@vgavro
Copy link
Copy Markdown

@vgavro vgavro commented May 12, 2026

Adding Retry.validate_response callback to Retry configuration.
Inspired by pydantic_ai.retries approach https://github.com/pydantic/pydantic-ai/blob/main/pydantic_ai_slim/pydantic_ai/retries.py

This should effectively solve two cases:

  1. Timeouts on reading response body, see Retries on response.read and response.aread can't be handled inside RetryTransport #29 - this shouldn't be default, but we're allowing user to do:
async validate_response_read(response: httpx.Response):
    await response.aread()
    
retry = Retry(validate_response=validate_response_read)

While ReadTimeout is already in default Retry.retry_on_exceptions, it will work as expected. httpx will not re-read body on second .read() invocation, body is cached once readed anyway.

  1. For cases with blocks in http body (or with redirect to authorization/captcha page) and access with rotating proxies, when content access is blocked but with 200 status code.
class ContentBlocked(ValueError):
  ...
  
async def validate_response(response: httpx.Response):
    response.raise_on_status()
    await response.aread()
    if "content blocked" in response.text:
        raise ContentBlocked()

retry = Retry(validate_response=validate_response, retry_on_exceptions=[httpx.HTTPError, ContentBlocked])

@codecov
Copy link
Copy Markdown

codecov Bot commented May 12, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (999e2aa) to head (65ec537).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##              main       #74    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files            6         6            
  Lines         1001      1132   +131     
==========================================
+ Hits          1001      1132   +131     
Files with missing lines Coverage Δ
httpx_retries/retry.py 100.00% <100.00%> (ø)
httpx_retries/transport.py 100.00% <100.00%> (ø)
tests/test_transport.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Owner

@will-ockmore will-ockmore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this @vgavro . It's a well thought out addition.

I'd like to take validate_response, but as a response validation hook, not the fix for #29. Your second example is a real problem that we should aim to solve, and validate_response gives users the tool to solve it. The implementation is

Why I don't think this is the right fix in general for #29 (and I'll give more detail there; apologies for my slow response on that thread, I've been moving countries) is twofold:

  1. At the general level: it breaks the layer boundary. Setting the request attribute is a symptom of this, but the transport should not be handling these aspects outside its remit. The first example would break streaming; the whole body would be read into memory, and this would not be clear to the user. This is another symptom of the same issue.
  2. It requires users to understand the library internals to fix something that should be solved by default. They should not need to know the incantation to solve body-phase retries, it should be transparent to them.

So to get this PR ready to merge:

  • Drop the faq.md edit that presents this as the solution to body-phase retries.
  • Document it for content validation; mention that calling .read() inside it will buffer stream=True responses, so it shouldn't be used that way.
  • Consider moving the async/sync-transport-mismatch check to RetryTransport.init so it fails fast

if not retry.is_retryable_status_code(response.status_code):
if self.retry.validate_response is not None:
# normally set by httpx _after_ calling this function, but we want the request in the validator
response.request = request
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the only slightly iffy operation. We should not be changing the behaviour of httpx in general. This is probably fine here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants