[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Stopping webcrawlers.



Hi Gene,

On Sun, Nov 03, 2019 at 11:40:23AM -0500, Gene Heskett wrote:
> I just installed fail2ban but setting it up looks daunting. Looking for a 
> tut.

Yes, that could be quite involved. Fail2Ban parses logs, so you'd
first have to decide what constitutes logging of an unwanted
condition (or make sure that such a condition is logged).

Your difficulty there is probably that any given log line by itself
is innocuous, it is the repeated number of requests for large
content that is problematic.

So, one way to go could be to use Fail2Ban with a really high
incidence count like say, 100 requests (access log lines) in a day
per IP.

Still that only counts requests, not bytes.

> Ideally, I'd like to steer such stuff thru a module that would limit them 
> to 10% of the available bandwidth. 35 kilobaud I could tolerate, 350kb 
> is a DDOS to be dealt with when it never ends.

I've never used it but this looks simple and is bundled with Apache:

    https://httpd.apache.org/docs/2.4/mod/mod_ratelimit.html

Idea being you'd use a Location match for your big files and set an
appropriate limit for those directories. Take heed of the warning
that it's applied to each request not to each IP. So, presumably, a
given IP could request the same thing 5 times simultaneously and
each request would get the limit.

Again I've never used it, but this is packaged as
libapache2-mod-qos and looks like it would work on a per-IP basis
for number of requests and bandwidth:

    http://mod-qos.sourceforge.net/

I've only ever used mod_cband but it looks like that is abandonware
now and was never updated for Apache 2.4.x.

Before any of these though I would be blocking by robots.txt and
UserAgent. Maybe that is enough and you don't need to do anything
else.

If you are serving large static files you may also want to put a CDN
in front of your site. Here's some free options:

    https://geekflare.com/free-cdn-list/

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Attachment: signature.asc
Description: Digital signature


Reply to: