Re: A success story with apt and rsync
Hi,
On 6 Jul 2003, Goswin Brederlow wrote:
> 2. most of the time you have no old file to rsync against. Only
> mirrors will have an old file and they already use rsync.
This is definitely true if you install your system from CD's and then
upgrade it. However, if you keep on upgrading from testing/unstable then
you'll have more and more packages under /var/cache/apt/archives so it
will have more and more chance that an older version is found there. Or,
alternatively, if you are sitting behind a slow modem and "apt-get
upgrade" says it will upgrade "extremely-huge-package", then you can still
easily insert your CD and copy the old version of "extremely-huge-package"
to /var/cache/apt/archives and hit ENTER to apt-get afterwards.
> 3. rsyncing against the previous version is only possible via some
> dirty hack as apt module. apt would have to be changed to provide
> modules access to its cache structure or at least pass any previous
> version as argument. Some mirror scripts alreday use older versions as
> templaes for new versions.
Yes, this is what I've hacked together based on other people's great work.
It is (as I've said too) a dirty hack. If a more experienced apt-coder can
replace my hard-coded path with a mechanism that tells this path to the
module, then this hack won't even be dirty.
> 4. (and this is the knockout) rsync support for apt-get is NO
> WANTED. rsync uses too much resources (cpu and more relevant IO) on
> the server side and a widespread use of rsync for apt-get would choke
> the rsync mirrors and do more harm than good.
It might be no wanted for administrators, however, I guess it is wanted to
many of the users (at least for me :-)) I don't see the huge load of the
server (since I'm the only one rsyncing from it), but I see the huge
difference in the download time. If my download wasn't faster because of
an overloaded server, I would switch back to FTP or anything which is
better to me as an end user.
I understand that rsync causes a high load on the server when several
users are connected, and so it is not suitable as a general replacement
for ftp, however I think it is suitable as an alternative. I also don't
expect the Debian team itself to set up a public rsync server for the
packages. However, some mirrors might want to set up an rsync server
either for the public or for example a university for its students.
Similar hack could be simply used by people who have account to a machine
with high bandwidth. For example if I used Debian and Debian had rsyncable
packages, but no public rsync server was available, I'd personally mirror
Debian to a machine at the university using FTP and would use rsync from
that server to my home machine to save traffic where the bandwidth is a
bottleneck.
So I don't think it's a bad idea to set up some public rsync servers
worldwide. The maximum number of connections can be set up so that cpu
usage is limited somehow. It's obvious that if a user often gets the
connection refused then he will switch back to ftp or http. Hence I guess
that the power of the public rsync servers and the users using rsync would
somehow be automatically balanced, it doesn't have to be coordinated
centrally. So IMHO let anybody set up an rsync server if he wants to, and
let the users use rsync if they want to (but don't put an rsync:// line
in the default sources.list).
> All together I think a extended bittorrent module for apt-get is by
> far the better sollution but it will take some more time and designing
> before it can be implemented.
It is very promising and I really hope that it will be a good protocol
with a good implementation and integration to apt. But until this is
realized, we still could have rsync as an alternative, if Debian packages
were packed in a slightly different way.
bye,
Egmont
Reply to: