apt with index diff support
Dear Friends,
the idea to have some sort of incremental update support for the
archive index files (Packages,Sources) in the archive and in apt has
been around for some time now [1].
Anthony Towns analysed the problem in [2] and came up with the idea to
use ed-style diffs to solve the problem.
Andreas Barth implemented the server side of the index diffs
generation and has a test-repository (with only the index-files) at
[3].
I'm happy to tell you that apt is able to use those index files
now. Robert Lemmen and I implmeneted the needed support.
This massively speeds up a "apt-get update". If you e.g. update daily
you will have to get only ~15kb-30kb worth of update information per
day (instead of 2,7mb for the complete Packages.bz2 as it is now).
We believe that the code is now "good" enough for wider testing. I
have setup a repository for the patched version of apt with the pdiff
support. Just add this line to your sources.list file:
"deb http://people.debian.org/~mvo/apt/pdiffs/ /"
When fully supported by the archive, the diff support will be
completely transparent, no changes on your side necessary. In the
moment however, the archive does not carry the diff files, so a little
hack is needed to test this. The diffs are on merkel.debian.org, so change your
sources.list to point there instead of your usual mirror:
"http://merkel.debian.org/~aba/debian/";
E.g. from:
deb http://ftp.de.debian.org/debian sid main
to
deb http://merkel.debian.org/~aba/debian sid main
There are two issues with using merkel:
1. There is no Release/Release.gpg file on merkel so you get "not
authenticated" warnings
2. merkel does not have the actual package files
To work around (2) and make it possible to still download packages
(even with merkel as the only entry in sources.list) a hack was added
to apt called: "APT::URL-Remap::". This allows one to remap a
URI. Example: apt-get install 3dchess -o
APT::URL-Remap::http://merkel.debian.org/~aba/debian/=http://ftp.de.debian.org/d
+ebian/"
will ensure that archives pointing to merkel are actually fetched from
ftp.de.debian.org. Please note that this hack will be removed once
there are servers available with both index diffs and packages (it's
just to make testing easier).
To work around the authentication issue you can use the
"--allow-unauthenticated" switch in apt, this will suppress the
authentication checking. Again once we have full support for the diffs
in the archive the authentification will work (and be checked).
Known issues:
- the progress-bar is a bit jumpy
- the total transfered data information (at the end of the update)
is totally incorrect
Please report success/failure/problems directly to me. There are two
debug switches that should be used if you are having trouble:
-o Debug::pkgAcquire::Diffs=true
-o Debug::pkgAcquire::RRed=true
The source code of the patch is availab in my:
michael.vogt@ubuntu.com--2005/apt--pdiff--0 [4]
baz repository.
Cheers,
Michael
[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128818
[2] http://azure.humbug.org.au/~aj/blog/2003/12/02
[3] http://merkel.debian.org/~aba/debian/
[4] at http://people.ubuntu.com/~mvo/arch/ubuntu
--
Linux is not The Answer. Yes is the answer. Linux is The Question. - Neo
Reply to: