[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Debian mirrors and search engines policy



Hi,

On Wed, Feb 08, 2012 at 10:49:03PM +0100, Carsten Otto wrote:

> On Wed, Feb 08, 2012 at 10:40:44PM +0100, Jogi Hofmüller wrote:
> > While browsing the stats of our small debian mirror [1] today I noticed
> > that according to awstats, search engines produce more load (size-wise)
> > than users of the mirror.  Interestingly the ratio between search
> > engines and users is totally different if you compare IPv4 and IPv6.
> > The IPv6 side of the mirror has no search engine traffic worth
> > mentioning, not even google pays a visit ;)
> 
> Are you sure awstats reports the amount of transferred data? I guess
> Google just tries to download the file and stops after a few bytes,
> which might be logged as a full download. At least this behaviour is
> something I remember from using awstats (not on my mirror, though).

Good point.  But google is not on top of the list.  It's Yahoo slurp.

> We block:
> http://ftp2.de.debian.org/robots.txt

Thanks for the tip!

Regards,
j.
-- 
j.hofmüller                                http://users.mur.at/thesix/

Attachment: signature.asc
Description: Digital signature


Reply to: