Re: [gopher] Spidering the gopherspace
Mateusz,
> I am spidering the gopherspace since a few days, collecting things for a
> new project I will publish soon.
>
> If you see 178.170.108.36 hammering your gopher server, that would be me.
>
> Do not hesitate complaining if you notice I'm too hard on you, I will
> try to spare your server then (I already see that sdf.org have
> blacklisted me).
FWIW, I throttled several minutes between requests to the same IP (or would
find another to visit in the meantime) and I always honour robots.txt if it
can be fetched (and cache it).
However, since V-2 only fetches menus and has a well-known reverse DNS, I
imagine sites are a little friendlier to me.
--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckaiser@floodgap.com
-- All things are possible, except skiing through a revolving door. -----------
_______________________________________________
Gopher-Project mailing list
Gopher-Project@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project
Reply to: