Re: LinkWalker

[wagnerc@plebeian.com (Chris Wagner)]

You should be able to tell if it cares about robots.txt by looking in the
logs to see if it's downloading /robots.txt.  If it is then something like:
User-agent: LinkWalker
Disallow: /

will keep it off your site.  If it doesn't, then iptables will keep it away.
Robots info:
http://www.global-positioning.com/robots_text_file/index.html

The fact that it downloads binaries too makes me think it's a site sucker
and not a legit spider.


At 12:30 PM 12/23/01 -0800, Nick Jennings wrote:
>On Sun, Dec 23, 2001 at 09:17:54PM +0100, Russell Coker wrote:
>> 
>> I wasn't aware that there was any format to robots.txt, I thought that the 
>> mere presense of such a file would prevent robots from visiting.





                    ---=<REMEMBER THE WORLD TRADE CENTER>=---
                ___/`<               WTC 911               >`\___

00000100