Re: New method for Packages/Sources file updates
Robert Lemmen <robertle@semistable.com> writes:
> On Thu, Nov 25, 2004 at 06:59:15PM +0100, Frank Küster wrote:
>> There was also an ITP for a client-side rsync method that puts the load
>> on the client side, and which was intended to solve the same problem,
>> eventually. I didn't bother to read the details, though.
>
> that's zsync [0], packages are in NEW right now.
Hmm, coevolution. You seem to be nearer to a stable release though.
Ever thought about bzip2 support? It is probably only good for huge
files with minimal changes since bzip2 has a blocksize of ~1Mb
(uncompressed).
> short story: you generate a file on the server side (cheap, small) and place
> it alongside your Packages file. an client that is capable of understanding
> it can retrieve it and then calculate on the client side (=for free) which
> parts of the original file it needs and retrieve thos over http range
> requests.
I also provide a cgi to compute it on the fly. Haven't done a compute
and save flavour yet.
> i thinks it's kinda perfect for debian pacakages files, and already works
> pretty well. that said it's a pretty new piece of software and has some
> issues that need to be ironed out and of course needs wider testing. but
> in general it doesn't cause much load on the servers (like rsync) doesn't
> need huge amounts of files on the server sides (like incremental patches)
> and has almost no dependencies, so it would be easy to integrate into apt.
> and as a bonus it would be easy to modify it so that you don't need to put
> the delta/checksum files on the same server as the actual file you want to
> download -- cool for testing purposes.
For Packages files, instead of doing a blockwise checksum, I did a
stanza-wise checksum and noted the size of the stanza instead of the
rolling checksum in the checksum file. This results in ~300K checksums
for main Packages (which could easily be halved by smaller checksums)
and has probably more block hits than a fixed blocksize. But that
is just a feeling since I haven't done comparisons yet.
> cu robert
>
> [0] http://zsync.moria.org.uk
>
> --
> Robert Lemmen http://www.semistable.com
MfG
Goswin
Reply to: