Re: fail2ban for apache2
On 2019-11-09, john doe <johndoe65534@mail.com> wrote:
>
> Note that using IPs directly is an red herring; you need to use other
> means (UserAgent ...) to identify those bots.
Over at semrush they advise the following (with robots.txt in the top
directory of the server):
To stop SEMrushBot from crawling your site, add the following rules to your
"robots.txt" file:
To block SEMrushBot from crawling your site for web graph of links, add:
User-agent: SemrushBot Disallow: /
Please note that there might be a delay up to two weeks before SEMrushBot
discovers the changes you made to robots.txt.
To remove SEMrushBot from crawling your site for different SEO and technical
issues, add:
User-agent: SemrushBot-SA
Disallow: /
They note that it may require up to two weeks for the cure to take effect.
> By the sound of it, you cleerly need to learn the httpd server you are
> using, then if it is not enough, add fail2ban and iptables into the mix.
>
> --
> John Doe
>
>
--
“The cradle rocks above an abyss, and common sense tells us that our existence
is but a brief crack of light between two eternities of darkness.”
"Speak, Memory," Vladimir Nabokov
Reply to: