Re: Need help analyzing (kernel?) memory usage and reclaiming RAM (Debian Stretch)
Hi.
On Tue, Apr 16, 2019 at 10:22:16AM +0200, Martin Schwarz wrote:
> Hello,
>
> On Mon, Apr 15, 2019 at 07:03:13PM +0300, Reco wrote:
> [...]
> > What I suspect is happening here is runaway memory allocation by a
> > kernel module (at least one of them), and said kernel module is likely
> > to be VMWare-specific.
> > It could be vmxnet3 (network). It could be that LSI kernel module or
> > whatever they're using for SCSI these days (vmw_pvscsi?).
>
> sounds interesting.
>
> That would explain why I haven't seen this problem on one of my (few)
> personal Stretch installations running as Xen DomU. But then, I guess
> we're not the only ones who use Debian Stretch on VMware ESXi ;-)
> but haven't found any mention of this problem. I wonder what makes our
> setup so special ...
I see nothing unusual short of somewhat low amount of RAM by today's
standards.
It's no excuse for the kernel to behave that way, for obvious reasons.
> Here's the outout from lsmod:
Nothing unexpected here.
> Is there a way to show the memory consumed by each module? (besides the
> 'perf' tool you recommend below)
slabtop from "procps" package, definitely.
Should've thought of it earlier.
> Would memory consumed by a module be released when the module is
> unloaded? I guess so.
Barring kernel memory leaks - yes. Yep, they "invented" them too.
A price to pay if you write a kernel in C.
> Only I can't unload modules that are in use, of course. Unloading
> vmw_balloon, vmw_vmci, and vmw_vsock_vmci_transport didn't help.
I doubt that these are the problem. Unless you're changing VM's RAM at
runtime (and you wrote you don't).
If you can unload it - it's not used, hence no kernel memory allocations
worthy of speaking.
> > And that means - 'perf top', or better yet - 'perf record'.
>
> I have never used perf before, will look into it.
It's very indirect method. Basically it shows internal kernel functions
used, which may or may not be the source of the leak.
There's this SystemTap thing that can presumably do it better, but last
time I've checked it required a debug version of the kernel (and that's
unsuitable for just about anything short of kernel development).
Reco
Reply to: