[gopher] Re: New Gopher Wayback Machine Bot
On Wed, Oct 12, 2005 at 02:23:42PM -0500, Benn Newman wrote:
> I know that Veronica-2 obeys robots.txt. Does the bot have its own client name (or whatever it is that
> robots.txt calls it? I have content on my server that is easily replaceable (a mirror) and there
> is no reason to do a full text of it. Last time I updated [my not publically avaliable] JUGHEAD, the
> 'gophermap' file was already eight megabytes and I have added content since then.
OK. I can do robots.txt as well. I'll use user agent name gopherbot.
For the moment, I have told it to pick a new host at random before each
connection, which should pretty much keep most people from even noticing
that there is a bot going on.
Reply to: