Re: Bug#945879: linux: slab memory leak around 30MB/hour causing OOM on OVH VPS
control: reassign -1 openssh/1:7.9p1-10+deb10u1
control: retitle -1 openssh-server: cgroup leftovers with socket activation when key exchange fails
On 2019-11-30 13:52, Aurelien Jarno wrote:
> Package: src:linux
> Version: 4.19.67-2+deb10u2
> Severity: important
>
> Dear Maintainer,
>
> Since the latest Debian stable release, I observe a slab memory leak of
> about 30MB/hour when running the kernel 4.19.67-2+deb10u2 on an OVH VPS,
> which causes an all applications to slowly move to swap after a few days,
> and eventually an OOM. You'll find a typical munin memory plot attached
> to the bug report.
[...]
> The problem has been introduced in Debian in kernel 4.19.67-1. I have
> found that the problem has been introduced upstream in the 4.19.66
> release.
It happens that the original problem is due to SSH, just that this new
kernel version makes things way more visible.
When using systemd socket activation the OpenSSH daemon sometimes does
not remove the cgroup created for the connection after the key exchange
algorithm has failed. This usually happens relatively rarely, less than
1% of the time. However on a single CPU system (e.g. VM with a single
vCPU), the problem happens 100% of the time.
To reproduce the problem using a VM:
- Reduce the number of vCPU to 1
- Switch the OpenSSH daemon to systemd socket activation using systemctl
enable ssh.socket followed by a reboot
- Try to connect to the system with a key exchange algorithm not
supported on buster. For example
ssh -o KexAlgorithms=diffie-hellman-group-exchange-sha1 host
- Look at /sys/fs/cgroup/memory/system.slice/system-ssh.slice. Each
connection leaves an entry in that directory. Each entry takes some
kernel memory.
- Depending on the available memory and available swap, after a few
thousands connections the OOM killer kills all the processes making
the system unusable.
--
Aurelien Jarno GPG: 4096R/1DDD8C9B
aurelien@aurel32.net http://www.aurel32.net
Reply to: