[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: new, not nice web bots disposal

Quoting Gene Heskett (2020-02-26 09:57:51)
> over the last 90 days or so, we seem to have been plauged with a new 
> breed of bots scanning our web pages, and they are not just indexing 
> our web pages I don't mind that, but they are ignoring our robots.txt 
> and are mirroring anything apache2 can reach, including stuff thats 
> there but not reachable by a normal browser just looking around and 
> clicking on links.  Its annoying as hell and when you're out in the 
> pucker-brush on a 10 megabit ADSL, eats up ones available upload 
> bandwidth of about 275kbytes/s.

Download "eating" upload on ADSL might be due to bufferbloat: 

 - Jonas

 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

Attachment: signature.asc
Description: signature

Reply to: