Bug#934752: libc6: SEGFAULTs caused by tcache after upgrade to Buster
Hi,
On 2019-08-14 14:50, Pavel Matěja wrote:
> Package: glibc
> Version: 2.28-10:amd64
>
> Dear Maintainer,
>
> We are running manually compiled Apache and OpenSSL on Debian servers in
> Debian-based chroots.
> After chroot upgrade from Stretch to Buster we started to see strange
> SEGFAULTs.
> The strange means they appear only on 2 servers out of 6.
> Servers with Xeon E5606 and Pentium G6950 were running fine while Xeon
> E3-1220 v6 produced crashes.
> It did not matter if the host Debian was Stretch or Buster.
[snip]
> SSLv3 and TLS code path looked quite distinct to cause the same problem.
> Based on info that SEGFAULTs are related to memory allocation in new libc
> and CPU performance I found
> http://51.15.138.76/patch/17499/
> where Wilco Dijkstra discuss some problems with tcache which "leads to
> various crashes in benchtests"
This patch looks an early version of the one that has been merged in
glibc 2.29 to fix tunables tcache issues:
https://sourceware.org/bugzilla/show_bug.cgi?id=24531
The patch has been backported to the upstream glibc 2.28 branch:
https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=58d2672f64176fcb323859d3bd5240fb1cf8f25c
Once we have the fix reaching unstable and then testing, I'll schedule
an upload to buster with the changes from the upstream glibc 2.28 branch.
> As workaround I tried to
> export GLIBC_TUNABLES=glibc.malloc.tcache_count=0
> in Apache startup script and I saw no SEGFAULT since.
>
> I have coredumps but they contain production private keys for Apache which I
> can't share and to make things even worse they are 1,6GB each.
>
> I understand this is heisenbug which you won't be able to reproduce. The CPU
> model dependency is beyond my comprehension.
> I'm curious if you are familiar with the new tcache and if you think if the
> patch in discussion can help.
> I'll try to build libc6 package with it to confirm final solution but I'm
> confused by the patch tree so far.
You can easily build a fixed glibc package that way (providing you have
the glibc build-dependencies, devscripts and git installed):
apt-get source glibc
cd glibc-2.28/
quilt pop -a
debian/rules update-from-upstream
dch -i + set the version you want + add a new changelog entry
debuild
Regards,
Aurelien
--
Aurelien Jarno GPG: 4096R/1DDD8C9B
aurelien@aurel32.net http://www.aurel32.net
Reply to: