Bug#592300: XZ Utils 5 status?
On 2010-09-26 Thorsten Glaser wrote:
> Lasse Collin dixit:
> >default memory usage limit for compression (but not decompression),
> >but I don't know what that should be (40 %, 80 %, 95 % of RAM?
> >max(80 % of RAM, RAM - 256 MiB)?). Some default limit may be needed
> >in the future
>
> RAM is irrelevant, datasize ulimit (soft, maybe hard if you
> manipulate it) are the things to look out for.
Resource limits could be worth checking too. Usability of resource
limits varies between kernels, e.g. Linux ignores the data segment size
limit. So many Linux systems don't have such limits set.
> I’d rather suggest to abort if the memory for the compression level
> selected cannot be allocated, but that may just be me. It’s uncon-
> venient but unsurprising.
That's what xz does now. It breaks some scripts on older boxes. The old
method of having a default memory usage limit for both compression and
decompression broke different scripts on older boxes. Breaking things
generally annoys people, even though some users don't mind too much that
they need to fix it either by editing the script(s) or setting a memory
usage limit via XZ_DEFAULTS or XZ_OPT.
The problem can be phrased this way: Should "xz -9" work by default on a
system with e.g. 64 MiB RAM? For many uses, a preset level higher than
the default 6 is waste of memory, but even that needs 94 MiB of memory,
which is no fun on old 64 MiB box.
Some possible solutions:
- Keep the current way and let xz fail on older systems by default
if the user hasn't set a memory usage limit.
- Have a default memory usage limit for compression (but not
decompression) based on total RAM and possibly also what
resource usage limits are. Scale too high settings down to
meet the limit.
- Scale settings down until malloc() succeeds. In certain
situations together with bad luck it can still fail by
running out of memory in the middle of the file, but
it is unlikely.
Personally I can live with the current way, since I set limits via
XZ_DEFAULTS anyway. But maybe some other way would reduce the number of
bug reports.
--
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
Reply to: