Re: Stopping webcrawlers.
On Sunday 03 November 2019 10:50:33 Andy Smith wrote:
> Hello,
>
> On Sun, Nov 03, 2019 at 10:04:46AM -0500, Gene Heskett wrote:
> > I am developing a list of broken webcrawlers who are repeatedly
> > downloading my entire web site including the hidden stuff.
>
> […]
>
> > How do I get their attention to stop the DDOS? Or is this a war you
> > cannot win?
>
> Hosting a public web site on a domestic broadband connection with low
> data transfer allowance isn't the best way to go in 2019, but you
> can have some success with an escalation of:
>
> 1. robots.txt
>
> 2. UserAgent banning
>
> 3. Fail2Ban and/or apache modules for per-IP quota or requests and
> bytes.
>
I just installed fail2ban but setting it up looks daunting. Looking for a
tut.
Ideally, I'd like to steer such stuff thru a module that would limit them
to 10% of the available bandwidth. 35 kilobaud I could tolerate, 350kb
is a DDOS to be dealt with when it never ends.
> Cheers,
> Andy
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>
Reply to: