Re: mirrors.kernel.org (Re: rsync and debian mirror consistency)

[Josip Rodin <joy@entuzijast.net>]

On Wed, Feb 27, 2008 at 04:54:54PM -0500, Justin Pryzby wrote:
> > > He seems to say that debian.org pushes archive content to the kernel.org
> > > mirrors (rather than running an ssh trigger causing them to immediately
> > > do a "pull").  It seems that consistency between the .deb files and the
> > > Packages.gz files isn't maintained by this special push, as it is with
> > > the anonftpsync template script.  Is this correct?
> > 
> > No, it's not, I fixed that a few months ago. The script rsyncs over the
> > index files first, doesn't delete,  and then in the second run rsyncs
> > everything else with --delete-after. Can you demonstrate a problem, what
> > error messages did you get that led you to the conclusion that this is
> > broken on .k.o sites?
> By adding a cronjob, the following was caused within a few hours.
> 
> */5 * * * * root nice apt-get -q -q update && apt-get -q -q -d -y dist-upgrade
> 
> Email1:
> W: Failed to fetch http://mirrors.kernel.org/debian/dists/unstable/contrib/binary-i386/Packages.bz 2  Hash Sum mismatch
> W: Failed to fetch http://mirrors.kernel.org/debian/dists/unstable/non-free/binary-i386/PackagesIndex  MD5Sum mismatch
> W: Failed to fetch http://mirrors.kernel.org/debian/dists/unstable/main/source/SourcesIndex MD5Sum mismatch
> W: Failed to fetch http://mirrors.kernel.org/debian/dists/unstable/main/binary-i386/Packages.bz2 Hash Sum mismatch
> 
> Email2, started by cron 5 minutes later:
> Failed to fetch http://mirrors.kernel.org/debian/pool/main/g/glibc/locales_2.7-9_all.deb  404 Not Found [IP: 204.152.191.7 80]
> Failed to fetch http://mirrors.kernel.org/debian/pool/main/c/cdebootstrap/cdebootstrap_0.4.9_i386.deb  404 Not Found [IP: 204.152.191.7 80]

Hm. You're using their rotation, which means you're exposed to the race
condition that can happen because the two machines may update at different
speeds - that sounds like the problem in the latter case; not sure about
the first batch of checksum errors. Another person has been reporting these,
but I haven't been able to reproduce them :(

I've been meaning to fix their rotation's problem, too, but never got
around to it. Sounds like a good time to do that...

BTW, I hope you only ran that cron job as a test...

-- 
Josip Rodin
mirrors@debian.org