[gopher] Re: Gopher "robots.txt"

[David Allen <s2mdalle@titan.vcu.edu>]

On Sun, Jan 14, 2001 at 03:25:37PM -0800, emanuel at heatdeath organisation wrote:
> 
> On Sun, Jan 14, 2001 at 05:28:31PM -0500, David Allen wrote:
> > That's not a bad idea...I'd prefer personally to put whatever
> > information that this null statement has in the gopher+ fields to
> > avoid overloading the meaning of the other fields (type, name, host,
> > port).  So maybe something like this to disallow the debian
> > subdirectory:
> > 
> > iFFerror.hostF909FDisallow: selector_for_debian_directory
> 
> Why not put this information in the Gopher+ attribute information?  Have
> an attribute that means "ignore this selector" to the robot.  This way
> clients that don't know about it will never see it.  If it's set for a
> directory, the robot will ignore than directory and will never see
> anything liked by the directory.
> 
> Isn't this kind of thing what the Gopher+ attribute information is for?

I guess it depends on the robot.  It is sort of what the attribute
information is for, but should it be the case that a robot has to
fetch the attribute information for every single directory it ever
fetches?  That could cause a hit on performance/bandwidth since a
robot can't fetch just the resource and be safe with that.

-- 
David Allen
http://opop.nols.com/
----------------------------------------
"Be excellent to each other."
	- Bill and Ted's Excellent Adventure