Re: [proposal] use xz compression for Debian package by default

[Guillem Jover <guillem@debian.org>]

Hi!

On Tue, 2012-08-28 at 12:10:18 +0900, Hideki Yamane wrote:
>  In DebConf12, I talked about xz compression for Debian packages(*).
>  Now I'll talk about next step, suggestion for use xz with with result
>  from some experiment.

> ------------------------------------------------------------------------------
> conclusion (half)
> ------------------------------------------------------------------------------
>  We should use xz compression instead of bzip2 at least. bzip is harmful for
>  compressing debian package, so should drop it from support to check easier.

Removing bzip2 uncompression support is not an option, there's packages
on the wild compressed with it that should remain extractable from
dpkg-deb. Removing compression support could be considered, but then I
think right now bzip2 availability is more widespread than xz, so some
might want to use it for just that reason, even if it might be slower
or produce bigger packages.

> ------------------------------------------------------------------------------
> conclusion (rest)
> ------------------------------------------------------------------------------
>  I recommend to use xz ***by default*** (with appropriate option) on not only
>  i386/amd64 but on ANY architectures. Increasing extract time can be ignore by
>  decreasing download time and its only part of installation as Mike Hommey 
>  suggested "I/O is still more time consuming than CPU", and nothing worse than
>  high cpu usage.
> 
>  We know some packages are better to use gzip, but it's an exception. Using xz
>  is best choice for rest 99.99% of packages. We can deal with such exception
>  by specifying gzip for that (e.g. openclipart-png).

I thought this was already the consensus, and the only dissenting
opinion was that the base system should still be using gzip so that
foreign non-Debian systems can unpack it w/o requiring to build or
install xz beforehand.

Given the recent flurry of several packaging helpers and packages
switching to use xz, I think for jessie what makes most sense is
to switch all base packages to explicitly compress with gzip and then
switch dpkg-deb to default to xz; which I think would have made more
sense for wheezy too, but it seems too late now, given that non-base
packages have already been switched, because it might imply reverting
stuff and having to modify all base right now.

So if there's still consensus on this by then, I'll be switching
the default dpkg-deb compression to xz, *after* all base has been
switched to gzip. I've already queued a tiny patch for 1.17.x that
allows changing the default dpkg-deb compressor when building dpkg.
I've also some code already which avoids at least two copies of the
data.tar through pipes, which should speed up the unpacking.

>  *) tiny pseudo code
> 
> arch=`dpkg-architecture -qDEB_HOST_ARCH`
> 
> if [ arch = arm | armel | armhf | aarch64 ] // maybe
>     set on_arch --arm
> elsif [ arch = powerpc | ppc64 | powerpcspe ] // maybe
>     set on_arch --powerpc
> elsif [ arch = sparc | sparc64 ]    // maybe
>     set on_arch --sparc
> elsif [ arch = ia64 ]
>     set on_arch --ia64
> elsif [ arch = i386 | amd64 ]
>     set --x86
> fi

I don't think this is a good idea, and I'm not really planning on
making the dpkg-deb compression code conditional on the being built
package architecture.

In any case, thanks for the testing and comparisons!

thanks,
guillem