Re: [proposal] use xz compression for Debian package by default
On Tue, Aug 28, 2012 at 12:10:18PM +0900, Hideki Yamane wrote:
> In DebConf12, I talked about xz compression for Debian packages(*).
> Now I'll talk about next step, suggestion for use xz with with result
> from some experiment.
> ------------------------------------------------------------------------------
> conclusion (rest)
> ------------------------------------------------------------------------------
> I recommend to use xz ***by default*** (with appropriate option) on not only
> i386/amd64 but on ANY architectures. Increasing extract time can be ignore by
> decreasing download time and its only part of installation as Mike Hommey
> suggested "I/O is still more time consuming than CPU", and nothing worse than
> high cpu usage.
>
> We know some packages are better to use gzip, but it's an exception. Using xz
> is best choice for rest 99.99% of packages. We can deal with such exception
> by specifying gzip for that (e.g. openclipart-png).
There's a better compressor here, it's name is "cat" (meow!). PNG files are
already deflate-compressed, so gzip can't help (higher settings for an
infinitessimal benefit aside).
XZ is smart enough to detect uncompressible files and use "cat" for them,
except for one issue: PNGs are not strictly uncompressible, and xz can often
cut another percent or more. This means, it will try to compress them which
wastes time from our point of view. So it not using any compression here
can save CPU for negligible costs.
> *** what's the best compress option for default? ***
> *** how to find appropriate compression rate(1, 6 or 9) for xz? ***
I'd say, let's not go there. The benefits of -9 compression are small and
can break tiny systems (with less than ~100MB of memory) if you're not
careful; -1 produces negligible CPU savings at the cost of often significant
disk space, if the data is incompressible one may want to disable
compression altogether -- but only for packages big enough to bother.
Micromanaging compression levels costs human time, and increases complexity.
> [BCJ filters]
> arch=`dpkg-architecture -qDEB_HOST_ARCH`
>
> if [ arch = arm | armel | armhf | aarch64 ] // maybe
> set on_arch --arm
If this can be applied blindly to non-code files without a noticeable loss,
that could be good if placed in dpkg-dev. If not, we're entering the
micromanaging land again.
--
Copyright and patents were never about promoting culture and innovations;
from the very start they were legalized bribes to give the king some income
and to let businesses get rid of competition. For some history, please read
https://en.wikipedia.org/wiki/Statute_of_Monopolies_1623
Reply to: