Bug#931111: linux-image-4.9.0-9: Memory "leak" caused by CGroup as used by pam_systemd
Hello,
Am 24.07.19 um 16:41 schrieb Roman Gushchin:
> On Wed, Jul 24, 2019 at 09:12:50AM +0200, Philipp Hahn wrote:
>> Am 24.07.19 um 00:03 schrieb Ben Hutchings:
...
>>> I would say this is a kernel bug. I think it's the same problem that
>>> this patch series is trying to solve:
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lwn.net_ml_linux-2Dkernel_20190611231813.3148843-2D1-2Dguro-40fb.com_&d=DwIDaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=jJYgtDM7QT-W-Fz_d29HYQ&m=xNLFAB3gBGB1NCKmQZN-6JNEj_AXfJ3-wYK7IDWJAx4&s=YfWWnoW-zJdTN0hd1tzzZQlUIUtjv-iBN9Co5rNP5J0&e=
>>>
>>> Does the description there seem to match what you're seeing?
>>
>> Yes, Roman Gushchin replied to me by private mail, which I will quote
>> here to get his response archived in Debian's BTS as well:
...
>>> I've spent lot of time working on this problem, and the final patchset
>>> has been merged into 5.3. It implements reparenting of the slab memory
>>> on cgroup deletion. 5.3 should be much better in reclaiming dying cgroups.
>>>
>>> Unfortunately, the patchset is quite invasive and is based on some
>>> vmstats changes from 5.2, so it's not trivial to backport it to
>>> older kernels.
>>>
>>> Also, there is no good workaround, only manually dropping kernel
>>> caches or disable the kernel memory accounting as a whole.
...
>> So should someone™ bite the bullet and try to backport Romans change to
>> 4.19 (and 4.9)? (those are the kernel versions used by Debian).
>> I'm not a kernel expert myself, especially no mm/cg expert, but have
>> done some work myself in the past, but I would happily pass on the
>> chalice to someone more experienced.
>
> It's doable from the technical point of view, but I really doubt it's suitable
> for the official stable. The backport will consist of at least 20+ core
> mm/memcontrol patches, so it really feels excessive.
>
> If you still want to try, you need to backport 205b20cc5a99 first (and the rest
> of the patchset), but it may also depend on some other vmstat changes.
I haven't yet started on trying the backport, but is there some process
to force free those dying cgroups manually?
I have found yet another report of this issue at
<https://github.com/moby/moby/issues/29638#issuecomment-514287415> and
there a cron-job
> 6 */12 * * * root echo 3 > /proc/sys/vm/drop_caches
is recommended. I tried that manually on one of our affected systems and
the number of memory cgroups only dropped marginally from 211_620 to
210_396 after doing the `drop_caches` multiple times and waiting for 10
minutes by now. On that idle system a lot of RAM is gone:
> # free -h
> total used free shared buff/cache available
> Mem: 141G 60G 80G 15M 755M 80G
Thanks again for all your help.
Philipp
Reply to: