Bug#949173: packages.debian.org: robots.txt doesn't actually block anything

["Adam D. Barratt" <adam@adam-barratt.org.uk>]

Package: www.debian.org
User: www.debian.org@packages.debian.org
Usertags: packages
Severity: important

Hi,

While looking at blocking some web spiders that were causing
significant load on picconi (packages.d.o master), I noticed that the
robots.txt currently doesn't attempt to block them.

Specifically, the current file is:

<quote>
User-agent: *
Disallow:
</quote>

which is effectively the same as allowing everything. "Disallow: /"
might be more logical, unless there is a desire / requirement to allow
crawling and indexing of (parts of) the site.


Regards,

Adam