Re: Bug#1023870: dpkg: Problems in buildds due to slow compression
On Fri, Nov 11, 2022 at 07:15:59PM +0100, Manuel A. Fernandez Montecelo wrote:
>...
> The origins of this bug report are because there are sometimes problems building
> packages in buildds, the compression phase is very slow and sometimes the build
> is aborted due to inactivity:
>
> E: Build killed with signal TERM after 300 minutes of inactivity [1]
>
> After some investigation by aurel32 and myself, this was traced back to the
> commit f8d254943051e085040367d689048c00f31514c3 [2], in which the calculation of
> the memory that can be used, to determine the number of threads to use, was
> changed from half of the physical mem to be based on the memory available.
>
> For example in buildds with not-so-large amount of RAM, using part of it for
> tempfs (which is how buildds are set-up), the reported available memory maybe is
> not very large. For example it could be MemAvailable=2GB for 16GB of physical
> RAM in the machine. In the dpkg code, for the purposes of the calculation of
> how much memory can be used, the result appears to be to be half of the
> MemAvailable [3], so only 1GB.
>
> According to the tables in xz man page, for compression the algorithm can use
> 674MB. So as a result, dpkg-deb uses single-threaded compression, even if it
> could easily use 2-3 threads and still use only RAM; or use the full number of
> threads in the machine (e.g. 4 or 8) even if it means swapping out some of the
> content of tempfs -- which is not a problem in most cases for buildds, specially
> if using fast disks.
I don't see src:linux changing the compression from the default level 6,
so that should be < 100 MB per core.
I am wondering whether these buildds are for some reason running into
the 128 MiB default at the bottom of the commit when the reading from
/proc fails for some reason.
> As a result of all this, it is taking upwards of 600 mins of CPU time to
> compress recent linux-image-*-dbg packages in the buildds of riscv64
> architecture at the moment, so when using 2 threads of less, it's guaranteed to
> be aborted.
>
> But this also affects other arches and other packages in other ways, at least by
> making the build needlessly slow in many cases.
There is an even more worrysome issue, from xz(1):
If the specified memory usage limit is exceeded when decompressing, xz
will display an error and decompressing the file will fail. If the
limit is exceeded when compressing, xz will try to scale the settings
down so that the limit is no longer exceeded
Different compression of packages in the archive based on what is in
/proc on the buildd is not desirable.
> It's not very clear what could be the solution for this, as the default could be
> OK for desktops and many machines in which there's plenty of available RAM, but
> this is not the case of all of our buildds. It might not be possible to detect
> which is the best from within dpkg.
>...
Sebastian, was there any real-world problem motivating your commit,
or did it just sound more correct?
With default settings there should be < 100 MB/core RAM usage,
and even with "xz -9"[1] RAM usage should be < 700 MB/core.
These numbers shouldn't be a problem on buildds that successfully
manage to build packages large enough that multithreaded compression
is even possible.
Perhaps my understanding of xz memory usage is wrong,
or I might be missing some other problem.
> Thanks and cheers.
cu
Adrian
[1] which gives a lintian warning
Reply to: