Re: Inexplicable memory usage after move to Debian9
On Thu, Apr 26, 2018 at 03:09:15PM +0200, Simon Beirnaert wrote:
> Hi Recently I've started moving a fleet of Debian 7, 32-bit machines over to
> Debian 9, 64-bit. This migration is done by creating a fresh Debian 9 image
> with the necessary services, moving over user data (some wars and the
> content of /home) and rebooting into the new OS.
>
> Relevant services (ones we manage and use) are:
>
> - Jetty- Puppet - SSH - AutoSSH - NewRelic Infrastructure Through Puppet, we
> enforce system configuration is pretty much identical, save for stuff like
> host names and SSH keys. Now, we notice that on some systems, the RAM usage
> is way higher than expected, to the point where system memory is exhausted
> and processes (are) terminate(d). Investigation into what is causing this,
> leaves me at a dead end. I can't figure out where the memory is being
> consumed. Even after quitting all services we manage (leaving a "clean"
> installation), RAM usage on the system hovers just over 600MB, half a gig
> over what the same exact image consumes just after boot. The only fix I
> found so far is to reboot the system. The systems have ~1.8GB of system
> memory available. We don't use swap. Enabling swap gives the system some
> breathing room. On one system I enabled a swap volume of 512MB, which the
> kernel fills up and leaves filled up indefinitely. This points me to unused
> memory being allocated by mistake. A memory leak in the kernel, maybe? Below
> I've posted the output of some of the things I checked (with all services
> online). I also searched the internet and reached a topic about slab
> allocation (1). However, that didn't seem to solve anything. Can anyone here
> point me to some more stuff I can check out or try to debug this? Thanks!
> Simon root@mysystem:~# uname -a Linux mysystem 4.9.0-6-amd64 #1 SMP Debian
> 4.9.82-1+deb9u2 (2018-02-21) x86_64 GNU/Linux root@mysystem:~# free -m total
> used free shared buff/cache available Mem: 1831 1091 238 16 501 520 Swap:
> 511 511 0 root@mysystem:~# vmstat 1 procs -----------memory----------
> ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si
> so bi bo in cs us sy id wa st 2 0 524268 244480 46044 467248 1 2 2137 36 97
> 35 9 3 85 3 0 0 0 524268 244092 46052 467244 0 0 0 16 1125 2406 3 3 94 1 0 0
> 0 524268 244092 46060 467284 0 0 4 208 1230 3906 4 3 93 0 0 0 0 524268
> 244084 46068 467256 0 0 0 32 1003 1990 1 2 97 0 0 0 0 524268 244180 46068
> 467256 0 0 0 0 1099 2121 4 1 95 0 0 0 0 524268 244204 46076 467272 0 0 0 20
> 1000 1978 1 2 97 1 0 0 0 524268 244080 46076 467272 0 0 0 0 1135 2315 2 2 96
> 0 0 0 0 524268 244080 46084 467264 0 0 0 16 1079 2103 1 3 96 1 0 0 0 524268
> 244080 46092 467272 0 0 0 56 1002 1973 2 2 96 0 0 1 0 524268 244080 46100
> 467264 0 0 0 16 997 1979 1 2 97 1 0 0 0 524268 244144 46100 467268 0 0 0 0
> 988 1957 1 2 97 0 0 0 1 524268 244228 46108 467260 0 0 0 980 1292 2700 4 5
> 81 9 0 root@mysystem:~# smem PID User Command Swap USS PSS RSS 528 root
> /sbin/agetty -f /etc/issue. 148 4 4 8 554 myuser /usr/lib/autossh/autossh -o
> 80 24 40 184 220 root /lib/systemd/systemd-udevd 552 108 140 688 10665 root
> /usr/lib/autossh/autossh -o 0 104 143 648 432 root /usr/sbin/cron -f 164 120
> 147 500 434 root /usr/sbin/irqbalance --fore 252 156 172 508 12042 mail
> /usr/sbin/nullmailer-send - 120 144 219 1228 382 systemd-timesync
> /lib/systemd/systemd-timesy 456 152 301 1060 439 messagebus
> /usr/bin/dbus-daemon --syst 272 280 348 1116 430 root
> /lib/systemd/systemd-logind 380 192 448 1224 557 myuser /usr/bin/ssh -o
> StrictHostK 564 360 515 1472 216 root /sbin/lvmetad -f 188 484 517 1028
> 10668 root /usr/bin/ssh -o StrictHostK 0 736 825 1112 14225 ntp
> /usr/sbin/ntpd -p /var/run/ 0 808 849 1404 435 root /usr/sbin/rsyslogd -n
> 716 896 980 2036 11612 root /usr/sbin/sshd -D 0 856 991 1560 500 root
> /sbin/dhclient -4 -v -pf /r 840 928 1010 2068 1330 root sudo -i 0 920 1375
> 3548 1234 myuser -bash 0 600 1407 3684 1331 root -bash 0 640 1447 3712 1233
> myuser sshd: myuser@pts/0 0 300 1697 4568 1 root /sbin/init 524 1568 1935
> 3524 1227 root sshd: posios [priv] 0 1372 2984 6380 193 root
> /lib/systemd/systemd-journa 312 4304 4521 5828 8885 root /usr/bin/python
> /usr/bin/sm 0 9100 9452 11292 14574 root /usr/bin/newrelic-infra 1440 17348
> 17378 17828 520 root /opt/puppetlabs/puppet/bin/ 20636 40140 40168 40592 566
> jetty /usr/lib/jvm/java-8-openjdk 493896 958124 958381 959804 [1] https://unix.stackexchange.com/questions/244735/why-are-slab-objects-not-reclaimed-automatically
Hi,
The last line from smem sticks out with high usage figures:
566 jetty /usr/lib/jvm/java-8-openjdk 493896 958124 958381 959804
I don't use jetty (or java) so don't know if this normal but
googling "openjdk memory usage for jetty" (or "openjdk 8 jetty memory")
suggests a few ideas.
In particular, some people wrote about "an issue in Java8/9 that
manifests itself in Jetty due to the annotations module that scans the
jars and has a memory leak."
https://stackoverflow.com/questions/33104516/java-8-with-jetty-on-linux-memory-issue
Suggested solutions are to disable the annotations module completely,
or to restrict annotation scanning using a pattern in an xml file.
Let us know if you find a fix...
Reply to: