[gopher] Re: Veronica-2 and robot exclusion

[Cameron Kaiser <spectre@floodgap.com>]

> In any case, itemtype=0 sel=robots.txt is what the crawler will ask for.
> (Note no leading slash, since Gopher is supposed to be independent of the
> host filesystem.)

(i.e., the request will be "robots.txt\r\n", just to make it mega-obvious.)

-- 
---------------------------------- personal: http://www.armory.com/~spectre/ --
 Cameron Kaiser, Floodgap Systems Ltd * So. Calif., USA * ckaiser@floodgap.com
-- Birth, n.: The first and direst of all disasters. -- Ambrose Bierce --------