Re: lzma running out of memory on qemu-user (#846564)

[Guillem Jover <guillem@debian.org>]

Hi!

On Thu, 2024-09-26 at 10:31:55 +0200, John Paul Adrian Glaubitz wrote:
> I'm using qemu-user with a Debian unstable sh4 chroot on a 40-core Xeon machine
> and running dpkg-deb fails with the same issue reported in #846564 [1]:
> 
> dpkg-deb: building package 'gcc-snapshot' in '../gcc-snapshot_20240922-1+sh4.1_sh4.deb'.
> dpkg-deb: building package 'gcc-snapshot-dbgsym' in '../gcc-snapshot-dbgsym_20240922-1+sh4.1_sh4.deb'.
> dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate memory
> dpkg-deb: error: <compress> from tar -cf subprocess returned error exit status 2
> dh_builddeb: error: dpkg-deb --root-owner-group --build debian/.debhelper/gcc-snapshot/dbgsym-root .. returned exit code 2
> dh_builddeb: error: Aborting due to earlier error

Hmm, the heuristics in the code should in theory prevent this, so it
would be nice to know what is going wrong with that, in case it could
be improved. See the filter_xz_get_memlimit() and
filter_xz_get_cputhreads() functions in lib/dpkg/.

> Is there a way to tune lzma such that it reduces memory consumption in this case?

You should be able to globally reduce the amount of threads used with
the DPKG_DEB_THREADS_MAX envvar (as the --threads-max option would
need to be passed from debian/rules or debhelper or similar).

Thanks,
Guillem