Re: Package metadata server
On Fri, 5 Apr 2002, Glenn McGrath wrote:
> (picked up from http://www.debianplanet.org/article.php?sid=633)
>
> The scalability problems of the Packages file is a recognised problem that
> has been discused many times on this list, i think the following idea
> could go a long way to solving it.
>
> The current method of checking for updates is to retrieve a new
> Packages.gz file and discard the old Packages.gz file. The problem with
> this method is that commonly less than 1% of the Packages.gz file has
> changed. A number of solutions have been proposed to overcome this
> problem, these include - Compressing the Packages.gz in an rsync friendly
> manner. - Making diffs from older Packages files available.
> - Splitting the Packages.gz into multiple files.
> - Reducing the size of the metadata for each package.
Or, just appending new gzipped data to the end of the current Packages.gz.
I'm looking at doing that, along with using xdelta to encode the modified
changes on the uncompressed data.
For 54 packages files(54 days of changes), for sid/main/i386, it would take
900k more(xdelta files are compressed themselves). I consider that small.
Especially considering the compressed Packages.gz file is 1.6megs.
This can work over normal http, and ftp, with no modifications to the server.
Creating these xdeltas is easy. I'm currently working on a proof of concept
client, using shell, wget, and friends. I'll then modify apt. The goal is to
have the apt download driver make use of this, so the user can use whatever
method he/she desires(http/https/ftp/scp).
Also, this is a work-sanctioned idea, so I can work on it on company time,
instead of only volunteer time.
If I were to sit down, and do nothing else, I'd probably have my
proof-of-concept client done within a day. However, even tho work approves of
me doing this, there are lots of other things to do.
Watch this list for more details.
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Reply to: