Re: Stopping webcrawlers.
On Sunday 03 November 2019 10:34:09 john doe wrote:
> On 11/3/2019 4:04 PM, Gene Heskett wrote:
> > Greetings all
> >
> > I am developing a list of broken webcrawlers who are repeatedly
> > downloading my entire web site including the hidden stuff.
> >
> > These crawlers/bots are ignoring my robots.txt files and aren't just
> > indexing the site, but are downloading every single bit of every
> > file there.
> >
> > This is burning up my upload bandwidth and constitutes a DDOS when 4
> > or 5 bots all go into this pull it all mode at the same time.
> >
> > How do I best deal with these poorly written bots? I can target the
> > individual address of course, but have chosen to block the /24, but
> > that seems not to bother them for more than 30 minutes. Its also a
> > too broad brush, blocking legit addresses access. Restarting apache2
> > also work, for half an hour or so, but I may be interrupting a legit
> > request for a realtime kernel whose built tree is around 2.7GB in
> > tgz format
> >
> > How do I get their attention to stop the DDOS? Or is this a war you
> > cannot win?
>
> 'fail2ban' for the bots that does not respect robot.txt.
>
Wasn't installed by this stretch version. Is now, reading man page's.
Frankly this looks dangerous when attempted to be run as beginning users.
There ought to be a startup tutorial based on setting up the logging,
then specifying who you want blocked from reading the logs. Is there a
formal tut of setting this up someplace?
Thanks John Doe.
> --
> John Doe
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>
Reply to: