Re: new, not nice web bots disposal
On Wednesday 26 February 2020 14:57:18 deloptes wrote:
> Gene Heskett wrote:
> > over the last 90 days or so, we seem to have been plauged with a new
> > breed of bots scanning our web pages, and they are not just indexing
> > our web pages I don't mind that, but they are ignoring our
> > robots.txt and are mirroring anything apache2 can reach, including
> > stuff thats there but not reachable by a normal browser just looking
> > around and clicking on links. Its annoying as hell and when you're
> > out in the pucker-brush on a 10 megabit ADSL, eats up ones available
> > upload bandwidth of about 275kbytes/s. According to my cable
> > billing, these A-H's used over 100Gb of my bandwidth in Nov 2019.
> > That describes in printable language as a DDOS in my vocabulary.
>
> I have same observations at home. I have setup the modem (ADSL) with
> WLAN infront of the firewall. I block everything except 3 ports:
> 80 - for the web server
> 8080 - for the openvpn
> 22222 - for the SSH
>
> I recently replaced a very old firewall script with shorewall and
> started monitoring the activities in the logs. So it shows too many
> drops, which I think are port scanners.
>
> On the server listening on the open ports in apache, openvpn and ssh I
> see also the attacks described.
> The webserver shows content scanners or someone trying to exploit
> services. SSH and openvpn show signs of brute force attacks. I was
> wondering if its normal, but now I think it is and I am sure few
> months ago it was not. Gene is right - it started perhaps 3 months
> ago, while before it was from time to time.
>
> I can not say how much bandwith gets lost. I do not have much to share
> with the world from this PC :) but it is indeed annoying.
>
> regards
Whereas i've several gigabytes, much of it could be catalogued as blowing
my own horn. So its more than just annoying when there are 100+
machines out of the bots that do play by the rules that want to mirror
the whole thing and have a go get it again rule assuming a 50 gigabit
pipe, and heavens forbit they wouldn't want to serve up stale data!
And its probably 200% coinkydence that it all started when I first
published a fully preemptable realtime kernel for an r-pi4b, built and
running an uptodate buster and debs of linuxcnc built on that pi4b, to
run on that r-pi4b, almost as if they were trying to punish me for doing
it. Not at all plausible, but it does seem like a coincidence.
Murphy's law, I suppose, gotta have someplace to point my finger while
sharpening it. ;-)
Thanks deloptes.
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>
Reply to: