[gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database)

[John Goerzen <jgoerzen@complete.org>]

Cameron Kaiser <spectre@stockholm.ptloma.edu> writes:

> You could instead put in something like
> 
> iF1/stayout;1/dontindexFerror.hostF909
> 
> in the menu that references these selectors.

Not quite good enough, I'm afraid.  I might have many menus that
reference that, and others out there in gopherspace might also
reference it.  While I'd be able to control my own usage (albeit with
some difficulty), I have no control over how others out there link to
it.

> Mind you, I'd be happy with any approach that works on a per-menu level,
> just as long as the bot doesn't have to cache every server's particular
> robot policy and can determine the policy for a selector from the menu(s)
> that reference that selector. This is just one way I can think of.

Well, let's get back to this point.  I'm not sure that you must cache
it.  Assuming that a robot will traverse an entire server at once (am
I wrong with that assumption?), it would involve only one extra
request to ask for robots.txt before traversal.  If, OTOH, servers are
hit in a more random pattern, I can see there would be a problem.

Is it practical to cache this data then?  I might be inclined to
suggest that it is.  Given that only a few of the gopher servers out
there are both actively maintained and have a situation warranting
this sort of treatment, perhaps it is not so problematic a situation
after all?

-- 
John Goerzen <jgoerzen@complete.org>                       www.complete.org
Sr. Software Developer, Progeny Linux Systems, Inc.    www.progenylinux.com
#include <std_disclaimer.h>                     <jgoerzen@progenylinux.com>