Re: xz support in dpkg (was Re: dpkg plans for the squeeze cycle)
Hi!
On Tue, 2009-09-22 at 01:40:20 -0500, Jonathan Nieder wrote:
> LZMA decompressors can use large amounts of memory when processing
> archives built with a large dictionary size. Running out of memory
> can have bad effects, so the decompressor takes an argument
> representing maximum memory usage and (1) does not even bother if it
> is obvious that limit will be exceeded and (2) aborts decompression
> if the limit was exceeded anyway. What should that limit be for dpkg?
>
> I asked the XZ Utils author for advice. Here’s what he said:
>
> > What you need to do:
> >
> > 1. Find out how much RAM it is OK to use for decompressing packages.
> > This may vary between architectures. I hope this would be at
> > least 10 MiB, but 20 MiB would be very nice.
I guess a better question is, how much benefit a bigger dictionary size
would give us?
> > 2. Make sure that all packages will be compressed with settings
> > that will keep the decompressor memory usage below this limit.
We can try to specify it, and codify it in the tools, but there's people
out there building packages with ar and tar... (this should not affect
Debian though).
> > 3. Enforce this limit in dpkg so that installing a third-party
> > package on a phone doesn't cause a nasty surprise of too high
> > memory usage.
>
> The xz command (by default) refuses to use more than 40% of installed RAM
> when decompressing. That’s not appropriate for dpkg, since dpkg should
> not refuse to unpack policy-compliant packages just because unpacking
> them would be uncomfortable. The default dictionary size is 8 MiB. If
> we were to standardize on that, a 10 MiB memory usage limit would be
> enough.
>
> I am very uncomfortable picking a number here, since there are I don’t
> know well involved:
>
> a. debian-installer has tight memory constraints but needs to be able
> to do a debootstrap. Is it reasonable to assume 10 MiB of memory
> will be available just to decompress a binary package when the time
> comes?
>
> I think the answer should be yes, because the installer would have
> mounted swap by then. Is this correct? Any words of wisdom about
> memory constraints for unpacking .debs in the installer? How much
> memory should we assume is available and use for this?
>
> Related question: If an LZMA-based file format might ever be used
> for udebs, what are the memory constraints for unpacking those?
Well this is outside the scope of dpkg itself, and more a project wide
decision, but I'm not sure we'd want any package in the base system
built with anything but gzip, as that's shared by derivatives,
embedded distros, etc. xz should probably be used for big packages that
are guaranteed to be used on desktops or huge boxes (think games,
openoffice.org, etc).
> b. Some users might build their own packages that are compressed with
> too large a dictionary size. How should dpkg deal with these?
>
> Usually, reporting the error and a suggested memlimit and providing
> a command-line option to increase the limit would be good enough,
> but what about when dpkg is invoked through a front-end? Should
> there be an environment variable to set the memory limit, or do the
> front-ends all have easy-to-find facilities for passing extra options
> to dpkg?
This does not seem very nice. Is there no way to know how much memory
will we need from the file headers? If we could do that dynamically
that'd be great. Is there a maximum dictionary size?
OTOH if the package is out of spec we can do whatever we want, but I'd
rather make dpkg cope with such packages gracefully.
thanks,
guillem
Reply to: