Re: Stopping webcrawlers.
Hello,
On Sun, Nov 03, 2019 at 10:04:46AM -0500, Gene Heskett wrote:
> I am developing a list of broken webcrawlers who are repeatedly
> downloading my entire web site including the hidden stuff.
[…]
> How do I get their attention to stop the DDOS? Or is this a war you
> cannot win?
Hosting a public web site on a domestic broadband connection with low
data transfer allowance isn't the best way to go in 2019, but you
can have some success with an escalation of:
1. robots.txt
2. UserAgent banning
3. Fail2Ban and/or apache modules for per-IP quota or requests and
bytes.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Reply to: