Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)
Hello,
On Tue, Apr 16, 2019 at 11:49:57AM +0300, Reco wrote:
> I see nothing unusual short of somewhat low amount of RAM by today's
> standards.
> It's no excuse for the kernel to behave that way, for obvious reasons.
yes, 1 GB is not that much - but should be more than enough for the two
small RADIUSd instances (some 80 MB). Of course, if the workload
requires it, we can assign more RAM. In fact, that was what we first did
when the problem started to show, but this only lasted so long until the
additional RAM was consumed as well - it just takes a bit longer.
> slabtop from "procps" package, definitely.
> Should've thought of it earlier.
I did take a look at slaptop (and at /proc/slabinfo) before. But since the
values for "SReclaimable" and "SUnreclaim" from /proc/meminfo are not high
enough to explain the memory consumption, I didn't investigate any further in
that direction.
Looks unsuspicious to me:
root@rad-wgv-srv01:~# slabtop --once --sort=s | head -15
Active / Total Objects (% used) : 144152 / 186846 (77,2%)
Active / Total Slabs (% used) : 11205 / 11206 (100,0%)
Active / Total Caches (% used) : 66 / 116 (56,9%)
Active / Total Size (% used) : 40726,60K / 46301,70K (88,0%)
Minimum / Average / Maximum Object : 0,02K / 0,25K / 4096,00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
0 0 0% 4096,00K 0 1 0K kmalloc-4194304
0 0 0% 4096,00K 0 1 0K dma-kmalloc-4194304
0 0 0% 2048,00K 0 1 0K kmalloc-2097152
0 0 0% 2048,00K 0 1 0K dma-kmalloc-2097152
0 0 0% 1024,00K 0 1 0K kmalloc-1048576
0 0 0% 1024,00K 0 1 0K dma-kmalloc-1048576
0 0 0% 512,00K 0 1 0K kmalloc-524288
0 0 0% 512,00K 0 1 0K dma-kmalloc-524288
root@rad-wgv-srv01:~#
> > Only I can't unload modules that are in use, of course. Unloading
> > vmw_balloon, vmw_vmci, and vmw_vsock_vmci_transport didn't help.
>
> I doubt that these are the problem. Unless you're changing VM's RAM at
> runtime (and you wrote you don't).
We sometimes do increase a VM's RAM while it is running. But most of the VMs
showing the problem still have their initial memory size from the template they
were deployed from.
['perf top' and 'perf record']
> It's very indirect method. Basically it shows internal kernel functions
> used, which may or may not be the source of the leak.
I'm afraid 'perf' is beyond my knowledge. In its default invocation, 'perf_4.9
top' seems to be more focussed on CPU usage? And 'perf_4.9 record' is used to
profile a certain command? Tried with "sleep 30" as command, but not sure how
to interpret the recording.
So I'm really unsure how to use them to further drill into kernel or module
memory usage. Any hints?
> There's this SystemTap thing that can presumably do it better, but last
> time I've checked it required a debug version of the kernel (and that's
> unsuitable for just about anything short of kernel development).
By then I should probably go to debian-kernel? ;-)
Thanks again for all your help!
Martin
--
Martin Schwarz * Karlsruhe, Germany * http://kuroi.de/
Reply to: