[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

[gopher] Re: New Gopher Wayback Machine Bot

On Wed, Oct 12, 2005 at 01:01:32PM -0500, John Goerzen wrote:
> This bot is now running on my laptop and is spidering away.  The IP that
> you see connections from will vary depending on where my laptop is at a
> given time ;-)

Speaking of this, do we have any equivolent of robots.txt?  Cameron, do
you have some sort of exclude list or anything?

John Goerzen
Author, Foundations of Python Network Programming

Reply to: