[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: xz support in dpkg (was Re: dpkg plans for the squeeze cycle)

On Fri, 2009-09-25 at 05:58 +0200, Guillem Jover wrote:
> > > 3. Enforce this limit in dpkg so that installing a third-party
> > >    package on a phone doesn't cause a nasty surprise of too high
> > >    memory usage.
> > 
> > The xz command (by default) refuses to use more than 40% of
> installed RAM
> > when decompressing.  That’s not appropriate for dpkg, since dpkg
> should
> > not refuse to unpack policy-compliant packages just because
> unpacking
> > them would be uncomfortable.  The default dictionary size is 8 MiB.
> If
> > we were to standardize on that, a 10 MiB memory usage limit would be
> > enough.
> > 
> > I am very uncomfortable picking a number here, since there are I
> don’t
> > know well involved:
> > 
> >  a. debian-installer has tight memory constraints but needs to be
> able
> >     to do a debootstrap.  Is it reasonable to assume 10 MiB of
> memory
> >     will be available just to decompress a binary package when the
> time
> >     comes?
> > 
> >     I think the answer should be yes, because the installer would
> have
> >     mounted swap by then.  Is this correct?  Any words of wisdom
> about
> >     memory constraints for unpacking .debs in the installer?  How
> much
> >     memory should we assume is available and use for this?
> > 
> >     Related question: If an LZMA-based file format might ever be
> used
> >     for udebs, what are the memory constraints for unpacking those?
> Well this is outside the scope of dpkg itself, and more a project wide
> decision, but I'm not sure we'd want any package in the base system
> built with anything but gzip, as that's shared by derivatives,
> embedded distros, etc. xz should probably be used for big packages
> that
> are guaranteed to be used on desktops or huge boxes (think games,
> openoffice.org, etc). 

I'd think this is the sort of thing where a reasonably sane default
should be chosen, with the ability to override it with a distro- or
user-tweakable knob (maybe an environment variable, persistent
configuration option, a command-line configuration option or any
combination of the above).

If the default were something like 8 MB or 5% of RAM, whichever was
larger, that would pretty much cover the idea that packages would be
encouraged not to use a huge dictionary size for maximum compatibility.
That way, distributors, administrators, and users can expect that 8 MB
will be used pretty much regardless, but desktop system packages could
in theory be able to use a dictionary of 512MB * 5% = 25.6 MB of memory
if there was some significant gain in compression.  (Assuming that 512MB
is a reasonable presumption for the memory in a desktop system; I think
that it is today, since new systems have sold with more than that for a

Then if someone really does need to limit it (or increase it) for a
particular reason, it's easily tweakable so that corner cases can be
covered, too.

	--- Mike

Blog:  http://mike.trausch.us/blog/
Misc. Software:  http://mike.trausch.us/software/

“The greater danger for most of us lies not in setting our aim too
high and falling short; but in setting our aim too low, and achieving
our mark.” —Michelangelo

Reply to: