[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: LinkWalker

On Sun, 23 Dec 2001 20:28, Nick Jennings wrote:
> Why don't you just update your robots.txt to explicitly specify which
> files you don't or do, allow spiders access to. If it's a rule-obiding
> spider, that will be the end of it.

I wasn't aware that there was any format to robots.txt, I thought that the 
mere presense of such a file would prevent robots from visiting.

As for rule-abiding spiders, such programs will not download files ending in 
.wav, .mp3, .gz, .tgz, or .zip so I won't even see them.

That's why I usually don't even notice responsible web spiders such as google 
when browsing my web logs!

> On Sun, Dec 23, 2001 at 05:41:47PM +0100, Russell Coker wrote:
> > I have a nasty web spider with an agent name of "LinkWalker" downloading
> > everything on my site (including .tgz files).  Does anyone know anything
> > about it?
> >
> > I've added the following to my firewall setup to stop further attacks...

http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page

Reply to: