[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: LinkWalker

<quote who="Russell Coker">

> > Why don't you just update your robots.txt to explicitly specify which
> > files you don't or do, allow spiders access to. If it's a rule-obiding
> > spider, that will be the end of it.
> I wasn't aware that there was any format to robots.txt, I thought that the 
> mere presense of such a file would prevent robots from visiting.


- Jeff

     "Funny, I have no trouble distinguishing my mobile phone from the      
       others because it's in my _own fucking pocket_!" - Mobile Rage       

Reply to: