On Sun, Dec 23, 2001 at 09:17:54PM +0100, Russell Coker wrote:
> I wasn't aware that there was any format to robots.txt, I thought that the
> mere presense of such a file would prevent robots from visiting.
Here is an example of my robots.txt
> As for rule-abiding spiders, such programs will not download files ending in
> .wav, .mp3, .gz, .tgz, or .zip so I won't even see them.
> That's why I usually don't even notice responsible web spiders such as google
> when browsing my web logs!
Hmm, I have had spiders grab .tgz's from me before, but not anymore.
User-agent can be set to a specific spider agent-name, or * for all spiders.