Re: new, not nice web bots disposal
On Wednesday 26 February 2020 04:21:09 Jonas Smedegaard wrote:
> Quoting Gene Heskett (2020-02-26 09:57:51)
>
> > over the last 90 days or so, we seem to have been plauged with a new
> > breed of bots scanning our web pages, and they are not just indexing
> > our web pages I don't mind that, but they are ignoring our
> > robots.txt and are mirroring anything apache2 can reach, including
> > stuff thats there but not reachable by a normal browser just looking
> > around and clicking on links. Its annoying as hell and when you're
> > out in the pucker-brush on a 10 megabit ADSL, eats up ones available
> > upload bandwidth of about 275kbytes/s.
>
> Download "eating" upload on ADSL might be due to bufferbloat:
> https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Bu
>fferbloat/
>
Maybe, but I can still go read the news IF my browser can sneak a page
request in between upload packets when gkrellm is showing 300k of upload
at the instant.
>
And I'm subscribed Jonas, no need to reply all.
> - Jonas
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>
Reply to: