[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: LinkWalker

You should be able to tell if it cares about robots.txt by looking in the
logs to see if it's downloading /robots.txt.  If it is then something like:
User-agent: LinkWalker
Disallow: /

will keep it off your site.  If it doesn't, then iptables will keep it away.
Robots info:

The fact that it downloads binaries too makes me think it's a site sucker
and not a legit spider.

At 12:30 PM 12/23/01 -0800, Nick Jennings wrote:
>On Sun, Dec 23, 2001 at 09:17:54PM +0100, Russell Coker wrote:
>> I wasn't aware that there was any format to robots.txt, I thought that the 
>> mere presense of such a file would prevent robots from visiting.

                    ---=<REMEMBER THE WORLD TRADE CENTER>=---
                ___/`<               WTC 911               >`\___


Reply to: