Bug#1071272: marked as done (linux: building the bookworm-backports armhf kernel causes OOM on buildds)
Your message dated Mon, 27 May 2024 21:30:15 +0200
with message-id <ZlTfRzcLwMeCa0Tf@aurel32.net>
and subject line Re: Bug#1071272: linux: building the bookworm-backports armhf kernel causes OOM on buildds
has caused the Debian Bug report #1071272,
regarding linux: building the bookworm-backports armhf kernel causes OOM on buildds
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)
--
1071272: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1071272
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems
--- Begin Message ---
Source: linux
Version: 6.7.12-1~bpo12+1
Severity: serious
X-Debbugs-CC: debian-arm@lists.debian.org
X-Debbugs-CC: dsa@debian.org
Hi,
armhf builds of the bookworm-backports kernel appear to have led to
outages on several buildds recently.
Each of arm-ubc-04, arm-ubc-05 and arm-ubc-06 (QEMU ganeti guests)
stopped responding after starting to build the kernel, and had to be
rebooted. The build logs stop are various points - two at different
points during drivers/net, and one during the dpkg-deb runs at the end
of the build. The one common factor appears to be that the system logs
on each machine show the OOM killer being invoked during the build,
initially killing syslog but subsequently schroot and many system
processes.
Each buildd has 12GB of RAM and 120GB of swap available.
The issue also seems to be specific to the armhf build - arm-ubc-06
recently successfully built the armel build of the same kernel version.
Please let us know if you need any further information.
Regards,
Adam
--- End Message ---
--- Begin Message ---
Hi,
On 2024-05-17 14:46, Adam D. Barratt wrote:
> Source: linux
> Version: 6.7.12-1~bpo12+1
> Severity: serious
> X-Debbugs-CC: debian-arm@lists.debian.org
> X-Debbugs-CC: dsa@debian.org
>
> Hi,
>
> armhf builds of the bookworm-backports kernel appear to have led to
> outages on several buildds recently.
>
> Each of arm-ubc-04, arm-ubc-05 and arm-ubc-06 (QEMU ganeti guests)
> stopped responding after starting to build the kernel, and had to be
> rebooted. The build logs stop are various points - two at different
> points during drivers/net, and one during the dpkg-deb runs at the end
> of the build. The one common factor appears to be that the system logs
> on each machine show the OOM killer being invoked during the build,
> initially killing syslog but subsequently schroot and many system
> processes.
This is a known issue on arm32, this is known as low memory starvation.
Simplistically, the kernel needs low memory to be able to manage the
high memory, and when there is not enough low memory available (even if
there is plenty of high memory available), the kernel can OOM.
I have reinstalled the arm-ubc-04/05/06 buildds as arm64, that should
fix the issue, and at the same time fix building chromium and firefox.
For the release team, this means that the armel and armhf architectures
are now exclusively built from a 64-bit installation. In practice this
is similar to what is being done on i386 for many years.
Regards
Aureien
--
Aurelien Jarno GPG: 4096R/1DDD8C9B
aurelien@aurel32.net http://aurel32.net
--- End Message ---
Reply to: