Re: new, not nice web bots disposal
On Wed, Feb 26, 2020 at 03:57:51AM -0500, Gene Heskett wrote:
> over the last 90 days or so, we seem to have been plauged with a new
> breed of bots scanning our web pages, and they are not just indexing our
> web pages I don't mind that, but they are ignoring our robots.txt
I can believe this. But it's better if you SHOW us the actual logfile
entries (not all of them, obviously -- just a reasonable sample).
If the bots have a recognizable user-agent, you can just configure
your web server to block queries from that user-agent. Then you don't
have to deal with iptables, nftables, or whatever they're calling
it this year.