Re: [idea]: Switch default compression from "xz" to "zstd" for .deb packages

To: debian-devel@lists.debian.org
Subject: Re: [idea]: Switch default compression from "xz" to "zstd" for .deb packages
From: Leandro Cunha <leandrocunha016@gmail.com>
Date: Mon, 18 Sep 2023 16:34:29 -0300
Message-id: <[🔎] CAGEayXOu2hKDEhB1D0uion48Cp98Sj3A7Oh7rAenDk8P46j++g@mail.gmail.com>
In-reply-to: <[🔎] 20230918080346.GA2238505@subdivi.de>
References: <[🔎] 20230916103120.3491013bec37c1fd07ab482c@iijmio-mail.jp> <[🔎] 20230918080346.GA2238505@subdivi.de>

Hi,

On Mon, Sep 18, 2023 at 4:20 PM Helmut Grohne <helmut@subdivi.de> wrote:
>
> Hi,
>
> On Sat, Sep 16, 2023 at 10:31:20AM +0530, Hideki Yamane wrote:
> >  Today I want to propose you to change default compression format in .deb,
> >  {data,control}.tar."xz" to ."zst".
> >
> >  I want to hear your thought about this.
>
> I am not very enthusiastic about this idea. I skip over those arguments
> already raised by others and add one that I haven't seen thus far. zstd
> is quite optimized for 64bit CPUs and for amd64 in particular. amd64 is
> the only architecture for which zstd provides a hufmann implementation
> in assembly.
>
> > ## More CPUs
> >
> >  2012: ThinkPad L530 has Core i5-3320M (2 cores, 4 threads)
> >  2023: ThinkPad L15 has Core i5-1335U (10 cores, 12 threads)
> >
> >  https://www.cpubenchmark.net/compare/817vs5294/Intel-i5-3320M-vs-Intel-i5-1335U
> >   - i5-3320M: single 1614, multicore 2654
> >   - i5-1335U: single 3650, multicore 18076 points.
>
> While the majority of CPUs in active deployments is amd64, I'd also like
> to see numbers for 32bit CPUs and non-x86 ones. While I personally find
> the trade-off by zstd fit for a number of my use cases, I was also
> surprised just how slow it decompresses on armhf.
>
> I found some arm board with some linux kernel package sized 36MB.
>
>  algo     | compressed size | decompression time
> ----------+-----------------+-------------------
>  xz       |     36MB        |  14.7s
>  zstd     |     52MB        |   5.2s
>  zstd -9  |     48MB        |   5.2s
>  zstd -11 |     47MB        |   5.4s
>  zstd -19 |     41MB        |   5.7s
>
> Not as slow as I remembered apparently, but it still has a more than 10%
> size overhead. The size ratio is consistent with Robert Edmond's
> numbers, but we no longer see that 10-fold speedup. And this did not
> look at decompression memory requirements.
>
> I am decompressing a *lot* of .debs (dedup.d.n, multiarch hinter,
> crossqa.d.n, dumat). All of these applications would benefit from zstd
> compressed .debs in terms of decompression speed. Yet, that has never
> been the bottleneck to me. To me, download speed matters more and
> swapping out a 1GBit link for a faster one isn't that easy.
>
> I'd vote against this given the data we have now.
>
> Can we defer the discussion until there are more convincing numbers?
>
> Helmut
>

If even Helmut voted against, I also vote and I package both for Arch
and Debian, I didn't notice that much of a difference in performance
for compression/decompression. But I use an NVME on the machine (it's
usually very fast along with a processor that isn't slow either) I use
to compile the packages I work with. But when you say that the numbers
are not that convincing, I agree.

-- 
Cheers,
Leandro Cunha

Reply to:

References:
- [idea]: Switch default compression from "xz" to "zstd" for .deb packages
  - From: Hideki Yamane <henrich@iijmio-mail.jp>
- Re: [idea]: Switch default compression from "xz" to "zstd" for .deb packages
  - From: Helmut Grohne <helmut@subdivi.de>

Prev by Date: Re: [idea]: Switch default compression from "xz" to "zstd" for .deb packages
Next by Date: Re: lpr/lpd
Previous by thread: Re: [idea]: Switch default compression from "xz" to "zstd" for .deb packages
Next by thread: Re: [idea]: Switch default compression from "xz" to "zstd" for .deb packages
Index(es):
- Date
- Thread