[gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database)

[John Goerzen <jgoerzen@complete.org>]

David Allen <s2mdalle@titan.vcu.edu> writes:

> Personally, I don't see any reason not to just lift the robots.txt
> verbatim and add it.  Minus the User-Agent part, which gopher doesn't

I was going to suggest that after I read up on the format of it :-)

> Questions on these items though:
> 
> 1.) Should "some_directory" be a selector string, or the portion of the URL
>   after the host?  I.e. on my system, I have a selector, 
> "1/Python Stuff".  Should it be listed as that, or as the portion of
>   the URL after the host, which would be: "/11/Python%20Stuff"?

Hrm, that's a good question.

> 2.) Should gopher servers hide files by the name of robot.txt from the
>   view of the client?  (i.e. should it be possible for a human user to
>   come into a directory and see a robots.txt entry, or should it be
>   automagically hidden?)

That should be at the discretion of the admin.  UMN gopherd has a type
"X" for this purpose.

-- John


-- 
John Goerzen <jgoerzen@complete.org>                       www.complete.org
Sr. Software Developer, Progeny Linux Systems, Inc.    www.progenylinux.com
#include <std_disclaimer.h>                     <jgoerzen@progenylinux.com>