[gopher] Re: New Gopher Wayback Machine Bot
I know that Veronica-2 obeys robots.txt. Does the bot have its own client name (or whatever it is that
robots.txt calls it? I have content on my server that is easily replaceable (a mirror) and there
is no reason to do a full text of it. Last time I updated [my not publically avaliable] JUGHEAD, the
'gophermap' file was already eight megabytes and I have added content since then.
On Wed, Oct 12, 2005 at 01:51:41PM -0500, John Goerzen wrote:
> On Wed, Oct 12, 2005 at 01:01:32PM -0500, John Goerzen wrote:
> > This bot is now running on my laptop and is spidering away. The IP that
> > you see connections from will vary depending on where my laptop is at a
> > given time ;-)
> Speaking of this, do we have any equivolent of robots.txt? Cameron, do
> you have some sort of exclude list or anything?
> John Goerzen
> Author, Foundations of Python Network Programming
email@example.com | gopher://igneous-rock.homeunix.net
If computers get too powerful, we can organize them into a
committee -- that will do them in.