Crawler bots typically work by finding links and putting that on their processing queue. A great way to minimize that is by removing these complex links from even being displayed to anonymous users. An example is the diffs link on the ?action=history page.
From: @cookmeplox
Crawler bots typically work by finding links and putting that on their processing queue. A great way to minimize that is by removing these complex links from even being displayed to anonymous users. An example is the diffs link on the
?action=historypage.From: @cookmeplox