Re: Apt & rsync
On Sat, 16 Oct 1999, Dylan Thurston wrote:
> Let me see if I understand the issue. rsync is optimized to make a single
> transfer run as quickly as possible, doing all sorts of bandwidth and CPU
> hogging things like doing many transfers at once. For Debian package
Not so much bandwidth, primarily CPU and IO. Bandwidth is generally on par
with ftp/http.
> distribution to end-users, all the business with correctly dealing with
> directories is irrelevant; the only thing we might care about is the base
> algorithm for transferring files, one or maybe two at a time. So the
> rsync daemon is not really the right starting point. Is that right?
Yes, this is pretty much exactly bang on.
> It occurred to me after I wrote this that with tools like dpkg-repack it's
> unnecessary to keep around the original compressed archives. This creates
> more client-side load, which is acceptable. Note that it's not necessary
> to recreate the initial archive exactly.
Yep, that would be the same thing the PIK does for CD images. Its a neat
scheme.
> I'm not sure we're talking about the same thing here. Let me summarize
> the algorithm as I currently see it:
>
> a) Debian developer creates the .deb file, using a modified rsync-friendly
> gzip. (Let's call it 'rzip' for now.)
** This step needs to have the 'last' .deb to pass to rzip to allow it to
fragment it properly.
> b) Developer uploads the .deb to incoming using ftp.
> c) Archive is distributed to the mirrors using the current rsync setup,
> which would run faster because of the friendly compression.
This isn't true because rsync can't handle renaming a file. A change of
contents always involves a rename so without rsync changes to be aware of
our archive structure there is no win.
> d) User uses dpkg-repack to create a .deb close to the .deb currently on
> the mirror.
I don't think this would work so well in the case of rzip as Andrew
described it.
> e) User uses the rsync algorithm (but not the current rsync protocol, for
> the reasons you point out) to update .deb to current .deb.
> Do you think it's unfeasible to get developers to switch to the new
> compression technique?
Probably yes.
As Gary suggested, rsyncing over the ungziped .deb is probably ultimately
better, but complicated and expensive server+client side.
Jason
Reply to: