[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: CD sizes again (and BoF reminder!)

On Sat, Jul 07, 2012 at 04:22:58PM -0600, Stefano Zacchiroli wrote:
> On Sun, Jul 08, 2012 at 12:03:44AM +0200, Cyril Brulebois wrote:
> > Ansgar Burchardt <ansgar@debian.org> (07/07/2012):
> > > Another question is if the release team would consider unblocking
> > > updated packages (with just the switch to xz for binaries).  I think
> > > it would be nice to be able to get a useful desktop system using just
> > > the first CD, but I'm not sure if they would make an exception for
> > > this.
> > 
> > You may want to actually ask the release team at some point, if you want
> > to know. Just saying…
> Thanks for the brilliant idea! :-)
> Oh all mighty release team,
>   Ansgar has been experimenting with .deb sizes to make the packages
> needed for a minimal desktop installation fit in the first CD. It looks
> like that's doable by switching to xz compression for the involved
> binaries. Would you grant freeze exceptions for packages that only
> changes that?

Note that has been raised in LOOOONG threads twice in less than a year.
So let's rehash the findings:

* XZ is almost always better:
  • xz -0 is twice as fast as gzip, with 78% the size (tested on a random
    92MB unstripped amd64 executable)
  • xz -3 is at par with gzip -9's compression speed
  • xz -6 (the default) is a lot slower when compressing, fast when
    decompressing, needs only 10MB memory, 58% size
  • xz -9 has very slow compression, takes gobs of memory, 56% size
    (Obviously, the "size" numbers are dragged down by uncompressible files
    when you look at the whole archive.)
  • It has somewhat slow start: small files may compress better with gzip,
    but never to a degree that would justify the complexity of switching
    compression algorithms.

* After recompressing the whole archive, it turns out compressing only
  biggest packages is not a good idea: the bulk of space is taken by
  medium-sized packages, which recompress almost as good.  Thus, for best
  effects, xz should be dpkg-deb's default.

* Any native ways to install support xz:
  • dpkg does (squeeze but not lenny)
  • d-i does (via busybox)
* Debootstrap relies on unxz being available on the running system.  Which
  might be not there for ancient ones.

Ie, the only show-stopper is debootstrap on foreign systems.

Here, there are two approaches:

* making sure all .debs debootstrap has to unpack use gzip.
  • explicitely forcing gzip in the packages affected would be error-prone
  • dpkg-deb doesn't know which packages are base, as transitive
    dependencies make prority itself not enough
  On the other hand, checking this by hand and rejecting the package only
  during/after upload could work as a short-term solution.

* letting debootstrap handle xz files.
  • using arch-dependent executables is probably a bad idea
  • unxz would be hard to re-implement in Perl or something else typically
    available on bare systems, Mirabilos tells me some BSDs don't even have
    Perl.  No other reasonable language is likely to be there.
  • having a bunch of unxz executables for every architecture is doable, but
    would be damn hard to build in an automated way, especially if we want
    debootstrap to handle unofficial ports.  Currently, gcc needs libc6-dev
    for the target system to cross-compile.
  • alternatively, one could use stand-alone binaries with some heavy
    hackery.  Unxz-embedded can be whipped into working with only three
    syscalls: read(), write() and _exit(), perhaps also some way to fetch
    memory.  This could be built without any headers, for any architecture
    gcc supports.  For an extra bonus, such ultra-static executables would
    reduce the number of variants: one file is needed for i386, x32 and
    amd64; armel and armhf, etc.  Sadly, the ELF header restricts
    architectures, or with a proper trampoline you could even have one
    executable work on twenty or more archs :p
  I did not hear Joey Hess or others chime in: would you accept such an
  universal binary into debootstrap?  It already ships a tarball (one with
  required /dev nodes).

* Repacking existing .debs is not a good idea for the main archive (even if
  it works "at home").  I doubt no-recompilation binNMUs would be safe, too.

Also, switching to XZ debs is not just a CD issue, doing so would
drastically reduce the resource usage of mirrors.

Copyright and patents were never about promoting culture and innovations;
from the very start they were legalized bribes to give the king some income
and to let businesses get rid of competition.  For some history, please read

Attachment: signature.asc
Description: Digital signature

Reply to: