[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: new, not nice web bots disposal

Gene Heskett wrote: 
> over the last 90 days or so, we seem to have been plauged with a new 
> breed of bots scanning our web pages, and they are not just indexing our 
> web pages I don't mind that, but they are ignoring our robots.txt and 
> are  mirroring anything apache2 can reach, including stuff thats there 
> but not reachable by a normal browser just looking around and clicking 
> on links.  Its annoying as hell and when you're out in the pucker-brush 
> on a 10 megabit ADSL, eats up ones available upload bandwidth of about 
> 275kbytes/s.  According to my cable billing, these A-H's used over 100Gb 
> of my bandwidth in Nov 2019. That describes in printable language as a 
> DDOS in my vocabulary.
> So I asked a few questions and wrote some little 2-3 line scripts after 
> putting a tail on /var/lib/httpd/other_vhosts_access.log, which logs 
> enough info you can generally identify the bots with it.
> I have since have generated 49 iptables rules that have blocked 99% of 
> them.

If you find yourself needing to add lots more rules, you might
want to generate a "set" instead of individual rules:



might be useful.


Reply to: