[gopher] Re: Veronica-2 and robot exclusion
I'm on a train just now and away from the net (horrors!) so I can't check
your URLs, but a quick question:
Is this the same robots.txt format that is used for the Web? If so, could
one potentially use a single robots.txt file in the root of a site that is
served up both as Gopher and over HTTP? (As quux.org is, for instance)
That would be a very nice feature.
On Wed, Jun 18, 2003 at 06:24:18AM -0700, Cameron Kaiser wrote:
> Also, I was thinking of making the file ".robots.txt" since many Unix
> gophers don't serve dot-files, although there are a growing number of
> Windows-hosted gophers and I don't know if it will break these (I don't
> do x86 myself).
AFAIK, all Unix gophers *will* serve dot files, many just won't include them
in directory listings. (Which I think is what you meant anyway)
Reply to: