Re: rsync in apt sources.list?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Tuesday 17 June 2003 17:35, you wrote:
> Is there a link to just the changelog so I can see if they added this
> before I spend the modem time downloading? http://packages.debian.org
> doesn't seem to have links to changelogs, not are they in separate
> files on the mirrors.
Dunno, if it isn't available then it sounds like a good thing for a feature
request. Though at modem speeds, 12MB only takes one hour.
> Cor> There are technical solutions to precomputing the diffs used by
> Cor> rsync, as well as solutions for diffing .gz files. E.g. I was
> Cor> able to perform apt-get update, upgrade in about half the time,
> Cor> but nobody has put in the work required to get these nicely
> Cor> integrated into the current tool set,
>
> is it in apt now?
No, I manually executed it.
> Cor> set up on the servers with simple HOWTOs for mirrors.
>
> or you mean apt can do it, but not all the servers run an rsync
> daemon?
No, and no.
On machine a) I copiled /var/cache/apt/archives
I then repackaged every archive on machine a)
(dpkg-split -s, rgzip, dpkg-split -j)
For Packages.gz, gunzip Packages.gz, rgzip Packages
One week later I ran apt-get update, apt-get upgrade on machine b)
I then repackaged all the archives on machine b)
I also renamed them so the names were the same as on machine a)
Next I did rproxy (rsync via http) from b) to a)
I timed this with both the repackaged and non repackaged versions. The
repackaged version used much less network traffic (though not as much less as
I would have liked).
So you see, it was a proof of concept, not a reimplementation that users can
use.
In order to make it usable:
get the rgzip package into debian. This patch is 100% stable (read it, it is
short). The only problem with it is it can only be used in 99% of
situations, so to avoid unexpected bugs it probably should be off by default
(e.g. it cannot be used in creating the dictionaries used by dict). It can
always be used in creating .deb files. When I proposed this to the gzip
maintainer (Bdale Garbee IIRC) the only response I got is that it might go in
upstream. Well, Jean Loup is rather busy, and hasn't put it in upstream in
the last year, so I think we shouldn't wait.
Modify dpkg to use rgzip (currently dpkg uses zlib if it is installed and
calls gzip if it isn't) so by deleting the use of zlib this is trivial.
Write a program that precomputes the rolling checksums used by rproxy, and tie
it into debuild. These checksums can then be uploaded with dupload. This
seems technically feasable to me, and the whole system works without this
file as well.
Modify rproxy to use the precomputed checksums if present rather than
generating them.
Modify rproxy to cope with our version numbering so it knows two files are
different versions if their version number differs. I expect this is quite
easy.
Do some stress/security checks on rproxy, mirrors won't appreciate us giving
them security holes. I don't know the code, so I don't know how hard this
will be.
Modify the standard mirror script to also copy the rolling checksum files.
Convince some mirrors to install rproxy.
> What about http://home.tiscali.cz:8080/~cz210552/aptrsync.html
> is it now obsolete?
That looks like it does what you were looking for above (update via rsync,
upgrade via http).
What do you think?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)
iD8DBQE+7ty0i5A0ZsG8x8cRAu3tAJoD3KoxCL6/Fyh9w4yjZeZ89ZtORACdGzO1
ZwrHUSf2eXVFkfhfeHUJE2s=
=u+Tr
-----END PGP SIGNATURE-----
Reply to: