[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Debian mirrors and search engines policy

Dear all,

While browsing the stats of our small debian mirror [1] today I noticed
that according to awstats, search engines produce more load (size-wise)
than users of the mirror.  Interestingly the ratio between search
engines and users is totally different if you compare IPv4 and IPv6.
The IPv6 side of the mirror has no search engine traffic worth
mentioning, not even google pays a visit ;)

Not only do I think this overhead in traffic in favor of search engines
is annoying, I also think it's useless (at least from my personal
experience in using debian).  So I am curious how others handle this
issue and/or if anyone has a policy regarding search engines.  I know
that not all robots do respect the robots.txt file, but maybe it's of
use here.

[1]  http://debian.mur.at/

j.hofmüller                                http://users.mur.at/thesix/

Attachment: signature.asc
Description: Digital signature

Reply to: