[gopher] Re: New Gopher Wayback Machine Bot

[Cameron Kaiser <spectre@floodgap.com>]

> Cameron, floodgap.com seems to have some sort of rate limiting and keeps
> giving me a Connection refused error after a certain number of documents
> have been spidered.

I'm a little concerned about your project since I do host a number of large
subparts which are actually proxied services, and I think even a gentle bot
going methodically through them would not be pleasant for the other side
(especially if you mean to regularly update your snapshot).

Veronica-2 doesn't actually download content other than non-local selectors
in a directory to get around this problem since it doesn't index the
content in any case, just the titles and selector data.

I do support robots.txt, see

	gopher.floodgap.com/0/v2/help/indexer

-- 
---------------------------------- personal: http://www.armory.com/~spectre/ --
 Cameron Kaiser, Floodgap Systems Ltd * So. Calif., USA * ckaiser@floodgap.com
-- "I'd love to go out with you, but I'm joining my split ends individually." -