Re: new, not nice web bots disposal
Gene Heskett wrote:
> over the last 90 days or so, we seem to have been plauged with a new
> breed of bots scanning our web pages, and they are not just indexing our
> web pages I don't mind that, but they are ignoring our robots.txt and
> are mirroring anything apache2 can reach, including stuff thats there
> but not reachable by a normal browser just looking around and clicking
> on links. Its annoying as hell and when you're out in the pucker-brush
> on a 10 megabit ADSL, eats up ones available upload bandwidth of about
> 275kbytes/s. According to my cable billing, these A-H's used over 100Gb
> of my bandwidth in Nov 2019. That describes in printable language as a
> DDOS in my vocabulary.
>
> So I asked a few questions and wrote some little 2-3 line scripts after
> putting a tail on /var/lib/httpd/other_vhosts_access.log, which logs
> enough info you can generally identify the bots with it.
>
> I have since have generated 49 iptables rules that have blocked 99% of
> them.
If you find yourself needing to add lots more rules, you might
want to generate a "set" instead of individual rules:
http://ipset.netfilter.org/
https://www.linuxjournal.com/content/advanced-firewall-configurations-ipset
might be useful.
-dsr-
Reply to: