[gopher] Re: Gopher "robots.txt"
On Sun, Jan 14, 2001 at 03:25:37PM -0800, emanuel at heatdeath organisation wrote:
>
> On Sun, Jan 14, 2001 at 05:28:31PM -0500, David Allen wrote:
> > That's not a bad idea...I'd prefer personally to put whatever
> > information that this null statement has in the gopher+ fields to
> > avoid overloading the meaning of the other fields (type, name, host,
> > port). So maybe something like this to disallow the debian
> > subdirectory:
> >
> > iFFerror.hostF909FDisallow: selector_for_debian_directory
>
> Why not put this information in the Gopher+ attribute information? Have
> an attribute that means "ignore this selector" to the robot. This way
> clients that don't know about it will never see it. If it's set for a
> directory, the robot will ignore than directory and will never see
> anything liked by the directory.
>
> Isn't this kind of thing what the Gopher+ attribute information is for?
I guess it depends on the robot. It is sort of what the attribute
information is for, but should it be the case that a robot has to
fetch the attribute information for every single directory it ever
fetches? That could cause a hit on performance/bandwidth since a
robot can't fetch just the resource and be safe with that.
--
David Allen
http://opop.nols.com/
----------------------------------------
"Be excellent to each other."
- Bill and Ted's Excellent Adventure
Reply to: