On 28 Dec 2014, at 21:52, Mateusz Viste <mateusz@viste.fr> wrote:
I am spidering the gopherspace since a few days, collecting things for a new project I will publish soon.
If you see 178.170.108.36 hammering your gopher server, that would be me.
Do not hesitate complaining if you notice I'm too hard on you, I will try to spare your server then (I already see that sdf.org have blacklisted me).
As far as I remember SDF is running my Gophernicus which has automatic throttling. The default settings are 4096 hits or 4 megabytes of transfer until you are throttled, and throttling in this case means adding 1 second to each request before answering. If you do 8192 hits you’ll get a 2-second punishment etc etc. Session expires in 30 minutes, so to get around the limitations just crawl ~4000 requests and stop for half an hour.
It’s possible that SDF has different settings, but those are the defaults. Gophernicus doesn’t implement complete blacklisting but it’s of course possible to do that on kernel level.
- Kim