Re: Missing index permission
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 06/03/2006 09:30 AM, Martin Schulze wrote:
> Hi,
>
> I noticed that the robots.txt file on www.debian.org contains
>
> | User-agent: *
> | Disallow: /security/
> | Disallow:
>
> What's the issue of not allowing /security/ to be indixed by search engines?
Hmmm, I really don't know, but I searched the CVS checkout on my
machine and, AFAIK, we do not handle robots.txt in the CVS. I also did
some research in the maillist archives and found this:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=67637
Looks like that it is historical and has some relation with DoS
made by search engine bots. Hope this helps.
> Regards,
> Joey
Kind regards,
- --
Felipe Augusto van de Wiel (faw)
"Debian. Freedom to code. Code to freedom!"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Using GnuPG with Debian - http://enigmail.mozdev.org
iD8DBQFEkiE6CjAO0JDlykYRAuBhAKC/9HgpCfbjyKbDCMmjaouMB7FWlQCgmMH+
+qdfTuV9VPcrBea43PNQzdg=
=2cj+
-----END PGP SIGNATURE-----
Reply to: