Re: Solving the compression dilemma when rsync-ing Debian versions
Reference to the old thread:
"http://lists.debian.org/debian-devel-0101/msg00765.html";
> Did you see this message?
> http://lists.debian.org/debian-devel-0101/msg01277.html
>
Yes, but I didn't know this thread went on. Anyway it peered out in january.
I've gotten the gzip patch from Rusty Russell ("rusty@rustcorp.com.au"),
it is located at
"http://netfilter.filewatcher.org/diary/gzip.rsync.patch2";.
Beat Rubischon ("beat@rubis.ch"), administrator of "debian.ethz.ch"
applied this patch and uses it to regularly gzip the Packages file of testing.
The first report of rsync'ed Packages.gz and Packages gives the
following figures (full report see below):
Packages.gz read 518446, matched 783300, total 1301746
Packages read 70260, matched 4430932, total 4501192
The figures are not as good as rsync'ing uncompressed but still more
than halving the download when compressed. I think this is about the
range which could be gained if used on all packages.
Of course there has to be an old package with the new name in place else
it won't have any effect. For partial mirrors anyone can use the scripts
at "http://www.problemlos.ch/~otti/";. Full mirrors could use these
scripts as well, but I'd suggest to write one who is more suited for
that task. Anyway it would be nice to get any feedback how well my
scripts would do.
Jean-loup Gailly ("jloup@gzip.org") said he doesn't have enough time to
take care of gzip, so he is looking for helpers. I don't know much of
compression and I'm just a novice Linux programmer, so maybe there is
anyone else to take this job. Jean-loup Gailly's might be reached
through the addresses listed in the README.recent of gzip.
I think there are other solutions for the correspondence-ratio of
compression and this might well lead to new compression algorithms.
O. Wyss
Full report:
*****************
rsync -P --stats
debian.ethz.ch::debian/dists/woody/main/binary-i386/Packages.gz .
Number of files: 1
Number of files transferred: 1
Total file size: 1301746 bytes
Total transferred file size: 1301746 bytes
Literal data: 518446 bytes
Matched data: 783300 bytes
File list size: 30
Total bytes written: 11274
Total bytes read: 523188
wrote 11274 bytes read 523188 bytes 16444.98 bytes/sec
total size is 1301746 speedup is 2.44
rsync -P --stats
debian.ethz.ch::debian/dists/woody/main/binary-i386/Packages .
Synch Packages
Number of files: 1
Number of files transferred: 1
Total file size: 4501192 bytes
Total transferred file size: 4501192 bytes
Literal data: 70260 bytes
Matched data: 4430932 bytes
File list size: 27
Total bytes written: 38649
Total bytes read: 95891
wrote 38649 bytes read 95891 bytes 11699.13 bytes/sec
total size is 4501192 speedup is 33.46
*****************
rsync -P --stats
debian.ethz.ch::debian/dists/woody/main/binary-i386/Packages.gz .
Number of files: 1
Number of files transferred: 1
Total file size: 1295084 bytes
Total transferred file size: 1295084 bytes
Literal data: 514584 bytes
Matched data: 780500 bytes
File list size: 30
Total bytes written: 11286
Total bytes read: 519318
wrote 11286 bytes read 519318 bytes 16844.57 bytes/sec
total size is 1295084 speedup is 2.44
rsync -P --stats
debian.ethz.ch::debian/dists/woody/main/binary-i386/Packages .
Number of files: 1
Number of files transferred: 1
Total file size: 4477323 bytes
Total transferred file size: 4477323 bytes
Literal data: 86731 bytes
Matched data: 4390592 bytes
File list size: 27
Total bytes written: 38709
Total bytes read: 112174
wrote 38709 bytes read 112174 bytes 15882.42 bytes/sec
total size is 4477323 speedup is 29.67
Reply to: