Re: Missing index permission
-----BEGIN PGP SIGNED MESSAGE-----
On 06/03/2006 09:30 AM, Martin Schulze wrote:
> I noticed that the robots.txt file on www.debian.org contains
> | User-agent: *
> | Disallow: /security/
> | Disallow:
> What's the issue of not allowing /security/ to be indixed by search engines?
Hmmm, I really don't know, but I searched the CVS checkout on my
machine and, AFAIK, we do not handle robots.txt in the CVS. I also did
some research in the maillist archives and found this:
Looks like that it is historical and has some relation with DoS
made by search engine bots. Hope this helps.
Felipe Augusto van de Wiel (faw)
"Debian. Freedom to code. Code to freedom!"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Using GnuPG with Debian - http://enigmail.mozdev.org
-----END PGP SIGNATURE-----