xz support in dpkg (was Re: dpkg plans for the squeeze cycle)

[Jonathan Nieder <jrnieder@gmail.com>]

Hi!

d-i maintainers: If you’re impatient, please skip to "a." below.

Guillem Jover wrote re dpkg plans for squeeze:

>  * XZ compression support (to deprecate lzma).
> 
>    The lzma format will still be supported, though, always for
>    extraction (as xz tools support it in a backward manner), probably
>    disable lzma creation after a while.

I’ve been looking into using liblzma to provide this.  liblzma is a
little more low level than the piece of zlib/libbz2 we use, so I
didn’t look into this for a while, but now I’ve learned enough to offer
a question.

LZMA decompressors can use large amounts of memory when processing
archives built with a large dictionary size.  Running out of memory
can have bad effects, so the decompressor takes an argument
representing maximum memory usage and (1) does not even bother if it
is obvious that limit will be exceeded and (2) aborts decompression
if the limit was exceeded anyway.  What should that limit be for dpkg?

I asked the XZ Utils author for advice.  Here’s what he said:

> What you need to do:
>
> 1. Find out how much RAM it is OK to use for decompressing packages.
>    This may vary between architectures. I hope this would be at
>    least 10 MiB, but 20 MiB would be very nice.
>
> 2. Make sure that all packages will be compressed with settings
>    that will keep the decompressor memory usage below this limit.
>
> 3. Enforce this limit in dpkg so that installing a third-party
>    package on a phone doesn't cause a nasty surprise of too high
>    memory usage.

The xz command (by default) refuses to use more than 40% of installed RAM
when decompressing.  That’s not appropriate for dpkg, since dpkg should
not refuse to unpack policy-compliant packages just because unpacking
them would be uncomfortable.  The default dictionary size is 8 MiB.  If
we were to standardize on that, a 10 MiB memory usage limit would be
enough.

I am very uncomfortable picking a number here, since there are I don’t
know well involved:

 a. debian-installer has tight memory constraints but needs to be able
    to do a debootstrap.  Is it reasonable to assume 10 MiB of memory
    will be available just to decompress a binary package when the time
    comes?

    I think the answer should be yes, because the installer would have
    mounted swap by then.  Is this correct?  Any words of wisdom about
    memory constraints for unpacking .debs in the installer?  How much
    memory should we assume is available and use for this?

    Related question: If an LZMA-based file format might ever be used
    for udebs, what are the memory constraints for unpacking those?

 b. Some users might build their own packages that are compressed with
    too large a dictionary size.  How should dpkg deal with these?

    Usually, reporting the error and a suggested memlimit and providing
    a command-line option to increase the limit would be good enough,
    but what about when dpkg is invoked through a front-end?  Should
    there be an environment variable to set the memory limit, or do the
    front-ends all have easy-to-find facilities for passing extra options
    to dpkg?

Thoughts?

Thanks,
Jonathan