Feature: option to retry accessing a site on failure

[HagegeR]

Thanks for this plugin!

When I've been using it on very big projects, I get flaky 504 on external links, so I thought, if it would be possible to configure either a global or a specific retry mechanism (either always retry X times, or only on certain error codes) then it would be a lot easier to have a stable CI using this plugin.

I feel like retrying on 404 might not be needed but I'm not certain, and I also don't want to burden you with the complexity of doing the retry per error codes if that is difficult to integrate...

anyways, I'd love to hear your thoughts on this!

[manuzhang]

@HagegeR Thanks for trying out the project. Have you tried the option https://github.com/manuzhang/mkdocs-htmlproofer-plugin?tab=readme-ov-file#raise_error_excludes for 504?

[HagegeR]

I know about the option, I just think 504 could re-occur, or could not, I'm not sure I want to disregards all 504, recurring one should be an issue

[manuzhang]

Do you have an example where recurring 504 is an issue?

[HagegeR]

I can't provide an example, it's in a closed network

[mflova]

I have a similar issue here. I am getting flaky 503 errors. Sometimes with even fully public URLs such as some repos from github. I guess than when the service is a bit saturated, the ping-based answers can delay a bit. This makes the tool treat the URL as invalid even when the website has been up the whole time

[manuzhang]

@HagegeR @mflova Sorry for long delay. Can you try out 1.4.0 with option retry_max_times? The default is 0, which means no retry to be consistent with the current behavior.

[mflova]

@HagegeR @mflova Sorry for long delay. Can you try out 1.4.0 with option retry_max_times? The default is 0, which means no retry to be consistent with the current behavior.

All fine! I was quite busy as well. I tried within my online pipelines with retry_max_times: 3 but it is still failing. As a reference, my last pipeline failed due to this:

2025-11-26T17:59:14.0512385Z INFO    -  htmlproofer: Retrying URL https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas.DataFrame from development\base_classes\pipeline.md after 2 seconds...
2025-11-26T17:59:16.0579582Z INFO    -  htmlproofer: Retrying URL https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas.DataFrame from development\base_classes\pipeline.md after 4 seconds...
2025-11-26T17:59:20.0632178Z INFO    -  htmlproofer: Retrying URL https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas.DataFrame from development\base_classes\pipeline.md after 8 seconds...
2025-11-26T17:59:28.0702540Z WARNING -  htmlproofer: invalid url - https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas.DataFrame [-1] 

Now I am doubting. Is it because the plugin itself or just the pandas website being down? Chances of the second scenario are low, but I am not sure if this can be double checked

[manuzhang]

Does it fail every time? I'm not able to reproduce locally.

[mflova]

Does it fail every time? I'm not able to reproduce locally.

More in a "flaky" way. Sometimes I just have this issue. If it is not this link, any other. Maybe with a frequency of 10% I would say