Re: please use robots.txt for your gopher apps

[Alex Schröder <kensanata@gmail.com>]

I'm interested in the details since my wiki is also available via
gopher. I'm trying to understand what I should be adding exactly.

First, the selector. Is it "robots.txt" or "/robots.txt"? On my site,
the selectors don't start with a slash but I'm assuming we're going
with a slash? Thus the correct place would be
gopher://alexschroeder.ch:70/0/robots.txt

Next, the content. I have some patterns I'd like to disallow, but I
guess I made some choices regarding the selectors that will come back
to haunt me when I look at how robots.txt works. for example, page
history. That is not something that needs to be indexed.

Here are some selectors for my About page:
gopher://alexschroeder.ch:70/1About/menu (the entry point)
gopher://alexschroeder.ch:70/0About (the plain text)
gopher://alexschroeder.ch:70/1About/history (the list of old
revisions, if available)
gopher://alexschroeder.ch:70/1About/10/menu (revision 10)
gopher://alexschroeder.ch:70/0About/10 (the plain text of revision 10)

It was more or less designed with the idea that "up" would return you
to a usable URL. Thus gopher://alexschroeder.ch:70/1About/history
seemed more reasonable than
gopher://alexschroeder.ch:70/1history/About. It doesn't quite work
that way right now because "gopher://alexschroeder.ch:70/1About has
the wrong item type (but surely something could be added) where as
gopher://alexschroeder.ch:70/1history loses the context of the About
page.

Anyway, what I'm trying to say is that I'd like to have patterns such as these:

Disallow: */history
Disallow: */\d+

What do you think? It definitely doesn't match what the WWW robots.txt
does, I know.

Cheers
Alex