Re: Stopping webcrawlers.
On Sunday 03 November 2019 10:27:34 John Hasler wrote:
> I doubt that you can get their attention and even if you did I doubt
> that they would stop: they may very well be deliberately targeting
> sites with a robots.txt.
>
> One reason they may be doing this is in search of vulnerable sites
> that they can hijack.
>
> You may need to implement access control.
Thru /etc/hosts.deny? The only thing that works is to uncomment the
PARANOID rule. But that blocks legit requests too. I already have all
these bots in my hosts.deny with a CIDR /24 on the end of the address.
I can restart apache2, and get about 15 to 20 minutes of peace, then they
are right back using all my upload bandwitdh reading the whole file.
Thats a DDOS AFAIConcerned.
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>
Reply to: