Bug#1023870: dpkg: Problems in buildds due to slow compression
Package: dpkg
Version: 1.21.9
Severity: normal
X-Debbugs-Cc: mafm@debian.org, debian-wb-team@lists.debian.org
(Setting initial severity as "normal" because it affects packages built in
buildd machines).
Hi,
The origins of this bug report are because there are sometimes problems building
packages in buildds, the compression phase is very slow and sometimes the build
is aborted due to inactivity:
E: Build killed with signal TERM after 300 minutes of inactivity [1]
After some investigation by aurel32 and myself, this was traced back to the
commit f8d254943051e085040367d689048c00f31514c3 [2], in which the calculation of
the memory that can be used, to determine the number of threads to use, was
changed from half of the physical mem to be based on the memory available.
For example in buildds with not-so-large amount of RAM, using part of it for
tempfs (which is how buildds are set-up), the reported available memory maybe is
not very large. For example it could be MemAvailable=2GB for 16GB of physical
RAM in the machine. In the dpkg code, for the purposes of the calculation of
how much memory can be used, the result appears to be to be half of the
MemAvailable [3], so only 1GB.
According to the tables in xz man page, for compression the algorithm can use
674MB. So as a result, dpkg-deb uses single-threaded compression, even if it
could easily use 2-3 threads and still use only RAM; or use the full number of
threads in the machine (e.g. 4 or 8) even if it means swapping out some of the
content of tempfs -- which is not a problem in most cases for buildds, specially
if using fast disks.
As a result of all this, it is taking upwards of 600 mins of CPU time to
compress recent linux-image-*-dbg packages in the buildds of riscv64
architecture at the moment, so when using 2 threads of less, it's guaranteed to
be aborted.
But this also affects other arches and other packages in other ways, at least by
making the build needlessly slow in many cases.
It's not very clear what could be the solution for this, as the default could be
OK for desktops and many machines in which there's plenty of available RAM, but
this is not the case of all of our buildds. It might not be possible to detect
which is the best from within dpkg.
Maybe it could be through a new variable DPKG_DEB_THREADS (similar to the
recently added DPKG_DEB_MAX_THREADS) in which the buildds explicitly set the
amount of threads that they want.
PS: Bug #501456 [4] is possibly a duplicate, or at least the intersection is
largish :)
[1] https://buildd.debian.org/status/fetch.php?pkg=linux&arch=riscv64&ver=6.1%7Erc3-1%7Eexp1&stamp=1667646869&raw=0
[2] https://git.dpkg.org/cgit/dpkg/dpkg.git/commit/?id=f8d254943051e085040367d689048c00f31514c3
[3] At least that's what I got in my tests as value for mt_memlimit, half of the
MemAvailable at the time, even if I expected the full amount of MemAvailable
when quickly reading the code. But I didn't pay much attention because this
detail is not very important in principle: MemAvailable could be very low if
tempfs is using more than the physical RAM, it doesn't matter if it's full
or half in many cases with low memory.
[4] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=501456
Thanks and cheers.
--
Manuel A. Fernandez Montecelo <mafm@debian.org>
Reply to: