Re: Intelligent mirroring

[Petr Cech <cech@atrey.karlin.mff.cuni.cz>]

On Thu, Aug 09, 2001 at 05:56:03PM +0200 , Wessel Dankers wrote:
> hi!
> 
> While we're at this topic, there's something to be said for structuring
> mirroring into four phases:
> 
>  1) Download all new files in the tree
>  2) Download all changed files in the tree to a temporary directory
>  3) Quickly replace all changed files with the new versions at once
>     (not one at a time as rsync does)
>  4) Delete all files that have disappeared from the master archive.
> 
> This would prevent the typical inconsistencies that occur in a mirror
> during updating and which are the #1 reason for people to use some sort of
> slower master site: what good is a fast mirror if you get a bunch of 404's?

so wait 30 mins for push to complete :))
Anyway, the "problem" is, that dists/ comes before pool/, so Packages are
updated earlier than files are uploaded. So a 2way rsync could do it

1) rsync --exclude Packages --exclude Packages.gz --exclude Packages.bz2
2) rsync --delete-after <- even this can help. so if people have old
packages, they don't notice till *after* the mirror, so they do
apt-get update, which fetches the new Packages and everyone is happy.
There is still a gap, but it's usually only 30mins/day. In europe it's like
23-24 for primary ftp.se.d.o and 1-3 for other, so not many people will
notice

				Petr Cech
-- 
Debian GNU/Linux maintainer - www.debian.{org,cz}
           cech@atrey.karlin.mff.cuni.cz

Try: cat /dev/urandom | perl