Re: A success story with apt and rsync
Koblinger Egmont <egmont@uhulinux.hu> writes:
> Hi,
>
> >From time to time the question arises on different forums whether it is
> possible to efficiently use rsync with apt-get. Recently there has been a
> thread here on debian-devel and it was also mentioned in Debian Weekly News
> June 24th, 2003. However, I only saw different small parts of a huge and
> complex problem set discussed at different places, I haven't find an
> overview of the whole situation anywhere.
...
I worked on an rsync patch for apt-get some years ago and raised some
design questions, some the same as you did in the deleted parts. Lets
summarize what I still remember:
1. debs are gziped so any change (even change in time) results in a
different gzip. The rsyncable patch for gzip helps a lot there. So
lets consider that fixed.
2. most of the time you have no old file to rsync against. Only
mirrors will have an old file and they already use rsync.
3. rsyncing against the previous version is only possible via some
dirty hack as apt module. apt would have to be changed to provide
modules access to its cache structure or at least pass any previous
version as argument. Some mirror scripts alreday use older versions as
templaes for new versions.
4. (and this is the knockout) rsync support for apt-get is NO
WANTED. rsync uses too much resources (cpu and more relevant IO) on
the server side and a widespread use of rsync for apt-get would choke
the rsync mirrors and do more harm than good.
> conclusion
> ----------
>
> The good news is that it is working perfectly.
>
> The bad news is that you can't hack it on your home computer as long as your
> distribution doesn't provide rsync-friendly packages. Maybe one could set up
> a public rsync server with high bandwidth that keeps syncing the official
> packages and repacks them with rsync-friendly gzip/zlib and sorting the
> files.
There is a growing lobby to use gzip --rsyncable for debian packages
per default. Its coming.
So what can be done?
====================
Doogie is thinking about extending the Bittorrent protocol for use as
apt-get method. I talked with him on irc about some design ideas and
so far it looks realy good if he can get some mirrors to host it.
The bittorrent protocol organises multiple downloaders so that they
also upload to each other and thereby reduces the traffic on the main
server. The extension of the protocol should also utilise http/ftp
mirrors as sources for the files thereby spreading the load over
multiple servers evenly.
Bittorrent calculates a hash for each block of a file very similar to
what rsync needs to work. Via another small extension rolling
checksums for each block could be included in the protocol and a
client side rsync can be done. (I heard this variant of rsync would be
patented in US but never saw real proof of it.)
All together I think a extended bittorrent module for apt-get is by
far the better sollution but it will take some more time and designing
before it can be implemented.
MfG
Goswin
Reply to: