[gopher] Re: Veronica-2 and robot exclusion

[Cameron Kaiser <spectre@floodgap.com>]

> Should the robots.txt be visible in the index if you request it normally, or
> is it also allowed to be invisible?
> 
> Invisible as in:
> Requesting the index doesnt show any robots.txt -> gopher://foo.bar.com/
> but requesting robots.txt works -> gopher://foo.bar.com/0/robots.txt
> 
> Is this 'invisible' use of robots.txt allowed?

This would be the way I'd do it, and this is easy to do with a link list
(like a Bucktooth gophermap file). There's probably no harm if the robots.txt
file does get presented to the user, though.

In any case, itemtype=0 sel=robots.txt is what the crawler will ask for.
(Note no leading slash, since Gopher is supposed to be independent of the
host filesystem.)

-- 
---------------------------------- personal: http://www.armory.com/~spectre/ --
 Cameron Kaiser, Floodgap Systems Ltd * So. Calif., USA * ckaiser@floodgap.com
-- FORTUNE: You learn from your mistakes. Today will be very educational. -----