Re: Bug#1023870: dpkg: Problems in buildds due to slow compression
Hi,
On 2022-11-12 23:04, Adrian Bunk wrote:
> On Fri, Nov 11, 2022 at 07:15:59PM +0100, Manuel A. Fernandez Montecelo wrote:
> >...
> > The origins of this bug report are because there are sometimes problems building
> > packages in buildds, the compression phase is very slow and sometimes the build
> > is aborted due to inactivity:
> >
> > E: Build killed with signal TERM after 300 minutes of inactivity [1]
> >
> > After some investigation by aurel32 and myself, this was traced back to the
> > commit f8d254943051e085040367d689048c00f31514c3 [2], in which the calculation of
> > the memory that can be used, to determine the number of threads to use, was
> > changed from half of the physical mem to be based on the memory available.
> >
> > For example in buildds with not-so-large amount of RAM, using part of it for
> > tempfs (which is how buildds are set-up), the reported available memory maybe is
> > not very large. For example it could be MemAvailable=2GB for 16GB of physical
> > RAM in the machine. In the dpkg code, for the purposes of the calculation of
> > how much memory can be used, the result appears to be to be half of the
> > MemAvailable [3], so only 1GB.
> >
> > According to the tables in xz man page, for compression the algorithm can use
> > 674MB. So as a result, dpkg-deb uses single-threaded compression, even if it
> > could easily use 2-3 threads and still use only RAM; or use the full number of
> > threads in the machine (e.g. 4 or 8) even if it means swapping out some of the
> > content of tempfs -- which is not a problem in most cases for buildds, specially
> > if using fast disks.
>
> I don't see src:linux changing the compression from the default level 6,
> so that should be < 100 MB per core.
>
> I am wondering whether these buildds are for some reason running into
> the 128 MiB default at the bottom of the commit when the reading from
> /proc fails for some reason.
No the problem they have is that the build is done in a 80GB tmpfs, with
100GB swap. This works because the kernel is slowly moving things to
swap when there is no RAM available. However for packages needing more
build space than the physical RAM, the kernel seems to start moving data
to the swap when MemAvailable is in the order of 100MB to 200MB. dpkg
therefore considers there are no memory available, that said after
moving a bit more data from the tmpfs to the swap, plenty of memory
would be available for dpkg to do the parallel compression.
To make a comparison, if GCC need 4GB of RAM, it just allocate them, and
the kernel takes care of moving things to swap. In the worst case the
OOM killer just kill GCC. On the other hand dpkg look how much memory
available without moving things to the swap, and only uses that.
> > As a result of all this, it is taking upwards of 600 mins of CPU time to
> > compress recent linux-image-*-dbg packages in the buildds of riscv64
> > architecture at the moment, so when using 2 threads of less, it's guaranteed to
> > be aborted.
> >
> > But this also affects other arches and other packages in other ways, at least by
> > making the build needlessly slow in many cases.
>
> There is an even more worrysome issue, from xz(1):
> If the specified memory usage limit is exceeded when decompressing, xz
> will display an error and decompressing the file will fail. If the
> limit is exceeded when compressing, xz will try to scale the settings
> down so that the limit is no longer exceeded
>
> Different compression of packages in the archive based on what is in
> /proc on the buildd is not desirable.
A *very* quick look at the xz source seems to show it looks at the
physical memory here, so we're good.
> > It's not very clear what could be the solution for this, as the default could be
> > OK for desktops and many machines in which there's plenty of available RAM, but
> > this is not the case of all of our buildds. It might not be possible to detect
> > which is the best from within dpkg.
> >...
>
> Sebastian, was there any real-world problem motivating your commit,
> or did it just sound more correct?
>
> With default settings there should be < 100 MB/core RAM usage,
> and even with "xz -9"[1] RAM usage should be < 700 MB/core.
I think the use case there, is for the desktop. It's preferable to use 2
threads only on a 4 core processor than sending Firefox to the swap.
That said that heuristics is not necessary the best for the build
daemon.
> These numbers shouldn't be a problem on buildds that successfully
> manage to build packages large enough that multithreaded compression
> is even possible.
Yep, we always have at least 1GB per core on all our buildds.
Regards
Aurelien
--
Aurelien Jarno GPG: 4096R/1DDD8C9B
aurelien@aurel32.net http://www.aurel32.net
Reply to: