Re: [gopher] Spidering the gopherspace

[James Mills <prologic@shortcircuit.net.au>]

On Mon, Dec 29, 2014 at 9:42 AM, Cameron Kaiser <spectre@floodgap.com> wrote:

FWIW, I throttled several minutes between requests to the same IP (or would
find another to visit in the meantime) and I always honour robots.txt if it
can be fetched (and cache it).

However, since V-2 only fetches menus and has a well-known reverse DNS, I
imagine sites are a little friendlier to me.

Good points for anyone wanting to experiment with

crawling and full-text search engines (i.e: me).

I'll try to keep this in mind :)

@Cameron: How do you presently find new Gopher servers? Manually via email and through discovery of other Gopher servers?

cheers

James

James Mills / prologic

E: prologic@shortcircuit.net.au

W: prologic.shortcircuit.net.au

_______________________________________________
Gopher-Project mailing list
Gopher-Project@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project