Re: new, not nice web bots disposal
On 2/27/20, email@example.com wrote:
> On Wed, Feb 26, 2020 at 11:25:53PM -0500, Lee wrote:
>> You're advertising your web server in your sig. The "other side"
>> ALREADY KNOWS you have a web server there.
> If that "other side" is reading your emails, that is.
> Not a likely scenario if that "other side" is some malware
> running in some whatever-of-things lightbulb or cat feeder.
This thread is NOT about likely scenarios; we're talking about
| over the last 90 days or so, we seem to have been plauged with a new
| breed of bots scanning our web pages, and they are not just indexing our
| web pages I don't mind that, but they are ignoring our robots.txt and
| are mirroring anything apache2 can reach, including stuff thats there
| but not reachable by a normal browser just looking around and clicking
| on links. Its annoying as hell and when you're out in the pucker-brush
| on a 10 megabit ADSL, eats up ones available upload bandwidth of about
| 275kbytes/s. According to my cable billing, these A-H's used over 100Gb
| of my bandwidth in Nov 2019. That describes in printable language as a
| DDOS in my vocabulary.
| So I asked a few questions and wrote some little 2-3 line scripts after
| putting a tail on /var/lib/httpd/other_vhosts_access.log, which logs
| enough info you can generally identify the bots with it.
| I have since have generated 49 iptables rules that have blocked 99% of
**in this case** is it better to have DROP or REJECT on the iptable rules?
I'm saying it might be better to reject than drop. Watch the logs and
if the A-H's ignore RSTs then go back to drop.