Re: xz support in dpkg (was Re: dpkg plans for the squeeze cycle)
On Fri, 2009-09-25 at 05:58 +0200, Guillem Jover wrote:
> > > 3. Enforce this limit in dpkg so that installing a third-party
> > > package on a phone doesn't cause a nasty surprise of too high
> > > memory usage.
> >
> > The xz command (by default) refuses to use more than 40% of
> installed RAM
> > when decompressing. That’s not appropriate for dpkg, since dpkg
> should
> > not refuse to unpack policy-compliant packages just because
> unpacking
> > them would be uncomfortable. The default dictionary size is 8 MiB.
> If
> > we were to standardize on that, a 10 MiB memory usage limit would be
> > enough.
> >
> > I am very uncomfortable picking a number here, since there are I
> don’t
> > know well involved:
> >
> > a. debian-installer has tight memory constraints but needs to be
> able
> > to do a debootstrap. Is it reasonable to assume 10 MiB of
> memory
> > will be available just to decompress a binary package when the
> time
> > comes?
> >
> > I think the answer should be yes, because the installer would
> have
> > mounted swap by then. Is this correct? Any words of wisdom
> about
> > memory constraints for unpacking .debs in the installer? How
> much
> > memory should we assume is available and use for this?
> >
> > Related question: If an LZMA-based file format might ever be
> used
> > for udebs, what are the memory constraints for unpacking those?
>
> Well this is outside the scope of dpkg itself, and more a project wide
> decision, but I'm not sure we'd want any package in the base system
> built with anything but gzip, as that's shared by derivatives,
> embedded distros, etc. xz should probably be used for big packages
> that
> are guaranteed to be used on desktops or huge boxes (think games,
> openoffice.org, etc).
I'd think this is the sort of thing where a reasonably sane default
should be chosen, with the ability to override it with a distro- or
user-tweakable knob (maybe an environment variable, persistent
configuration option, a command-line configuration option or any
combination of the above).
If the default were something like 8 MB or 5% of RAM, whichever was
larger, that would pretty much cover the idea that packages would be
encouraged not to use a huge dictionary size for maximum compatibility.
That way, distributors, administrators, and users can expect that 8 MB
will be used pretty much regardless, but desktop system packages could
in theory be able to use a dictionary of 512MB * 5% = 25.6 MB of memory
if there was some significant gain in compression. (Assuming that 512MB
is a reasonable presumption for the memory in a desktop system; I think
that it is today, since new systems have sold with more than that for a
while.)
Then if someone really does need to limit it (or increase it) for a
particular reason, it's easily tweakable so that corner cases can be
covered, too.
--- Mike
--
Blog: http://mike.trausch.us/blog/
Misc. Software: http://mike.trausch.us/software/
“The greater danger for most of us lies not in setting our aim too
high and falling short; but in setting our aim too low, and achieving
our mark.” —Michelangelo
Reply to: