Dear all, While browsing the stats of our small debian mirror [1] today I noticed that according to awstats, search engines produce more load (size-wise) than users of the mirror. Interestingly the ratio between search engines and users is totally different if you compare IPv4 and IPv6. The IPv6 side of the mirror has no search engine traffic worth mentioning, not even google pays a visit ;) Not only do I think this overhead in traffic in favor of search engines is annoying, I also think it's useless (at least from my personal experience in using debian). So I am curious how others handle this issue and/or if anyone has a policy regarding search engines. I know that not all robots do respect the robots.txt file, but maybe it's of use here. [1] http://debian.mur.at/ Regards, j. -- j.hofmüller http://users.mur.at/thesix/
Attachment:
signature.asc
Description: Digital signature