Re: Serious regression in systemd 215-17+deb8u10
Thanks for your responses. One of my colleagues has been looking into this trying to get the bottom of it and we do seem to have identified a memory leak which isn't present on stretch. I note the report posted to the list https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=924060.
Here are what my colleague Alan has documented about his findings, we're going into this without much knowledge of the area, sharing in the hope it helps someone more familiar with the situation:
We have run journald under Valgrind, putting system logs through logger - which has identified several memory leaks, for example:
==9577== 1,298,665 bytes in 21,103 blocks are definitely lost in loss record 11 of 11
==9577== at 0x4C28C20: malloc (vg_replace_malloc.c:296)
==9577== by 0x1272F0: strappend (in /lib/systemd/systemd-journald)
==9577== by 0x112849: dispatch_message_real.lto_priv.175 (in /lib/systemd/systemd-journald)
==9577== by 0x131867: server_dispatch_message (in /lib/systemd/systemd-journald)
==9577== by 0x12F189: process_datagram.part.10.lto_priv.168 (in
==9577== by 0x12355F: source_dispatch (in /lib/systemd/systemd-journald)
==9577== by 0x12402E: sd_event_run (in /lib/systemd/systemd-journald)
==9577== by 0x10DFC3: main (in /lib/systemd/systemd-journald)
Note that CVE-2018-16864.patch applies to function "dispatch_message_real" in journald-server.c,
which matches the leak found by Valgrind.
This patch changes calls to "strappenda" to "strappend".
"strappenda" calls "alloca(3)" which allocates memory on the stack, to be freed on function return.
"strappend" calls internal function "new()", which ends up calling "malloc_multiply()" and then "malloc(3)"
"malloc" normally allocates memory from the heap.
There are no corresponding calls to "free(3)" in the patch.
Red Hat's initial fix for CVE-2018-16864 introduced the same bug,
which has been assigned CVE-2019-3815
See
https://access.redhat.com/security/cve/cve-2019-3815
and
https://bugzilla.redhat.com/show_bug.cgi?id=1666646 (currently "Access Denied")
Centos has applied the RH patch in
0669-journald-free-cmdline-buffers-owned-by-iovec.patch from
http://vault.centos.org/7.6.1810/updates/Source/SPackages/systemd-219-62.el7_6.5.src.rpm
although the RH/Centos code is rather different in that they have backported "set_iovec_field_free" from systemd v233
----------
From: Chris Lamb <lamby@debian.org>
Sent: 06 March 2019 19:40
To: Dan Poltawski; debian-lts@lists.debian.org
Subject: Re: Serious regression in systemd 215-17+deb8u10
Hi Dan,
> We have discovered that the latest version of systemd uploaded to
> jessie is causing systemd-journald to use an ever increasing amount of
> memory, eventually leading to all available memory being consumed. This
> has been observed on multiple different systems we use which are
> logging quite heavily and have upgraded to the latest systemd package.
> We have some systems which haven't had the new security patches applied
> and are not observing this behaviour - there is a clear correlation
> with the latest version of the systemd package.
So, this is the patch that was applied in systemd 215-17+deb8u10:
Description: journald: do not store the iovec entry for process commandline on stack
This fixes a crash (CVE-2018-16864) where we
would read the commandline, whose length is under control of the
sending program, and then crash when trying to create a stack
allocation for it.
.
This is a backport of https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsystemd%2Fsystemd%2Fcommit%2F084eeb865ca63887098e0945fb4e93c852b91b0f&data=02%7C01%7Cdan.poltawski%40tnp.net.uk%7Caa9f27e2dcc5426fff0e08d6a26b8bad%7C925e5e137a8344cbb28144e1c700f7e5%7C1%7C0%7C636874980116819527&sdata=eXxpHC6N5HvqMm%2FANrXSTgJ%2B4TrmoXFouYnT8DYHbVg%3D&reserved=0
Author: Antoine Beaupré <anarcat@debian.org>
Bug-Debian: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.debian.org%2F918841&data=02%7C01%7Cdan.poltawski%40tnp.net.uk%7Caa9f27e2dcc5426fff0e08d6a26b8bad%7C925e5e137a8344cbb28144e1c700f7e5%7C1%7C0%7C636874980116819527&sdata=Nc%2FhVL6OtNP9xDC%2FDNd13UBCbGlJQQ%2Fpc5HcTeYegyM%3D&reserved=0
Origin: Debian
Bug: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.redhat.com%2Fshow_bug.cgi%3Fid%3D1653855&data=02%7C01%7Cdan.poltawski%40tnp.net.uk%7Caa9f27e2dcc5426fff0e08d6a26b8bad%7C925e5e137a8344cbb28144e1c700f7e5%7C1%7C0%7C636874980116819527&sdata=dxW4DrP2EZkKB2KwDIGp4ujl2j9GP0aGVDFlkco6k%2FY%3D&reserved=0
Forwarded: not-needed
Last-Update: 2019-01-22
--- systemd-215.orig/src/journal/journald-server.c
+++ systemd-215/src/journal/journald-server.c
@@ -602,7 +602,10 @@ static void dispatch_message_real(
r = get_process_cmdline(ucred->pid, 0, false, &t);
if (r >= 0) {
- x = strappenda("_CMDLINE=", t);
+ /* At most _SC_ARG_MAX (2MB usually), which is
+ * too much to put on stack. Let's use a heap
+ * allocation for this one. */
+ x = strappend("_CMDLINE=", t);
free(t);
IOVEC_SET_STRING(iovec[n++], x);
}
@@ -716,7 +719,9 @@ static void dispatch_message_real(
r = get_process_comm(object_pid, &t);
if (r >= 0) {
- x = strappenda("OBJECT_COMM=", t);
+ /* See above for size limits, only ->cmdline
+ * may be large, so use a heap allocation for it. */
+ x = strappend("OBJECT_COMM=", t);
free(t);
IOVEC_SET_STRING(iovec[n++], x);
}
I can't immediately see what's up, nor the direct relevance to the
upstream changes listed on #920018, however.
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` lamby@debian.org 🍥 chris-lamb.co.uk
`-
________________________________
The Networking People (NorthWest) Limited. Registered office: c/o Hanleys, Spring Court, Hale, Cheshire, WA14 2UQ. Registered in England & Wales with company number: 07667393
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
Reply to: