Re: apache - bots

[Johann Spies <jspies@sun.ac.za>]

On Tue, Dec 04, 2001 at 05:04:09PM +0200, Johann Botha wrote:

> > I am also interested in this. I experience frequent visits from a web
> > robot on our ftp-site (with a web front).  Going to the home page of
> > this robot at www.inktomi.com there was a remark that the robot respects
> > robot.txt-files.
> > 
> > How do you set up those files?
> 
> http://www.robotstxt.org/wc/robots.html

Thanks Johann!


-- 
Johann Spies          Telefoon: 021-808 4036
Informasietegnologie, Universiteit van Stellenbosch

     "Go ye therefore, and teach all nations, baptizing them
      in the name of the Father, and of the Son, and of the 
      Holy Ghost; Teaching them to observe all things  
      whatsoever I have commanded you; and, lo, I am with 
      you alway, even unto the end of the world. Amen."     
            Matthew 28:19,20