[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: LinkWalker

On Sun, Dec 23, 2001 at 09:17:54PM +0100, Russell Coker wrote:
> I wasn't aware that there was any format to robots.txt, I thought that the 
> mere presense of such a file would prevent robots from visiting.

Here is an example of my robots.txt

User-agent: *
Disallow: /webalizer/
Disallow: contacts.txt
Disallow: /dl/

> As for rule-abiding spiders, such programs will not download files ending in 
> .wav, .mp3, .gz, .tgz, or .zip so I won't even see them.
> That's why I usually don't even notice responsible web spiders such as google 
> when browsing my web logs!

 Hmm, I have had spiders grab .tgz's from me before, but not anymore.

 User-agent can be set to a specific spider agent-name, or * for all spiders.

  Nick Jennings

Reply to: