It was determined during a PR here, that in its current state mkdocs-htmlproofer will re-validate URLs it has seen before.
In order to maintain an upper bound on memory consumption, perhaps a new option should be added such as cache-size: 500 which will maintain a FIFO queue of the most recent 500 validated URLs. Each possible URL would then be checked for membership in the queue and, if present, the previous validation result repeated. My original idea a month ago was to only cache valid URLs, but I think caching invalid maybe is slightly improved -- although there is possible rationale in trying to re-validate on a set of possible error codes such as 429s, as we saw in the aforementioned PR.
@manuzhang Since you requested I open this issue, is there any additional information or question you have for me? (I apologize that I took a month to open this issue following your request.)
It was determined during a PR here, that in its current state mkdocs-htmlproofer will re-validate URLs it has seen before.
In order to maintain an upper bound on memory consumption, perhaps a new option should be added such as
cache-size: 500which will maintain a FIFO queue of the most recent500validated URLs. Each possible URL would then be checked for membership in the queue and, if present, the previous validation result repeated. My original idea a month ago was to only cache valid URLs, but I think caching invalid maybe is slightly improved -- although there is possible rationale in trying to re-validate on a set of possible error codes such as 429s, as we saw in the aforementioned PR.@manuzhang Since you requested I open this issue, is there any additional information or question you have for me? (I apologize that I took a month to open this issue following your request.)