Re: new, not nice web bots disposal

[john doe <johndoe65534@mail.com>]

On 2/26/2020 9:57 AM, Gene Heskett wrote:
> over the last 90 days or so, we seem to have been plauged with a new
> breed of bots scanning our web pages, and they are not just indexing our
> web pages I don't mind that, but they are ignoring our robots.txt and
> are  mirroring anything apache2 can reach, including stuff thats there
> but not reachable by a normal browser just looking around and clicking
> on links.  Its annoying as hell and when you're out in the pucker-brush
> on a 10 megabit ADSL, eats up ones available upload bandwidth of about
> 275kbytes/s.  According to my cable billing, these A-H's used over 100Gb
> of my bandwidth in Nov 2019. That describes in printable language as a
> DDOS in my vocabulary.
>
> So I asked a few questions and wrote some little 2-3 line scripts after
> putting a tail on /var/lib/httpd/other_vhosts_access.log, which logs
> enough info you can generally identify the bots with it.
>
> I have since have generated 49 iptables rules that have blocked 99% of
> them.
>
> Those scripts I've placed in /etc/iptables and are owned by root.
> To start iptables after a reboot:You might run this one first one
> from /etc/rc.local
>
> root@coyote:iptables$ cat start-iptables
>
> #!/bin/bash
> cd /etc/iptables
> iptables-restore <rules.v4
>
> To add a new rule, covering that whole 256 address block because they
> seem to have a random address, changed about weekly, in that block:
>
> root@coyote:iptables$ cat iptables-add
>
> #!/bin/bash
> iptables -I INPUT -s add.ress.to.block/24 -j DROP
>
> Substituting the address of the offender for add.ress.to.block in the
> last tine above.
>
> to save the rules:
> root@coyote:iptables$ cat iptables-saveem
>
> #!/bin/bash
> iptables-save >rules.v4
>
> To see what you've got so far:
> root@coyote:iptables$ cat iptables-status
>
> #!/bin/bash
> iptables -L -nv --line-numbers
>
> Which will output the rules in effect plus the hits accumulated in this
> uptime so far, in this format:
> lnum   hits  bytes fate
> 24     846   50760 DROP  all  --  *  *   66.249.64.0/24   0.0.0.0/0
>
> Be my guest folks, reclaim the net, we are paying for the bandwidth these
> jerks are burning up.
>

The above is the way the OP has choosen to go about it but configuring
apache properly using fail2ban in addition of the robot.txt file. should
also be considered

See past threads from this OP for an history of this.

--
John Doe