Re: .deb format: let's use 0.939, zstd, drop bzip2
On Wed, May 08, 2019 at 07:38:26PM +0200, Adam Borowski wrote:
>...
> So let's pick compressors to enable. For compression ratio, xz still wins
> (at least among popular compressors). But there's a thing to say about
> zstd: firefox.deb zstd -19 takes to unpack:
> * 2.644s .xz, stock dpkg
> * 2.532s .xz, my tool (libarchive based)
> * 0.290s .zst, my tool
> * 0.738s .gz, stock dpkg
> * 0.729s .gz 0.939, stock dpkg
> * 0.729s .gz 0.939, stock dpkg
> File sizes being 60628216 gz, 47959544 zstd, 44506304 xz.
>
> XFCE install total: 723M xz, 773M zstd, 963M gzip.
>
> Thus, even though we'd want to stick with xz for the official archive, speed
> gains from zstd are so massive that it's tempting to add support for it,
> at least for non-official uses, possibly also for common Build-Depends.
>...
Is this single-threaded or parallel?
pbzip2 decompression speed scales nicely with the number of CPUs,
and in general for anyone interested in massive speed gains the
way forward would be towards parallel decompression.
> But, the dlopen idea shows a potential victim: bzip2. Let's kill it.
>
> Stats for Buster's packages:
>
> .deb format:
> 2.0: 100%
>
> control:
> gz 11671
> xz 45210
>
> data:
> gz 966
> xz 55915
>
> With not a single package in the archive still using bz2,
You were only looking at binary packages,
for source packages bz2 is still pretty common.
> removing support
> would be reasonable. It'd be okay to give a clear error message telling the
> user to install libbz2-1.0 (dlopen) or bzip2 (pipe) -- so folks can still
> unpack historic .debs if need be.
It would be neither reasonable nor okay to create such hassle for users
for no benefits at all.
And if the tiny 75 kB libbz2 would be considered a problem,
the huge 650 kB libzstd would obviously never be an option
for packages in the archive.
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Reply to: