> > Current /robots.txt prohibits indexing of many resources that should be > > indexed. > Probably because search engines would overload www.debian.org otherwise. This can be easily checked.. are there any log analisis that has shown this?