Re: swap maxed out when plenty of RAM available
On Fri, Mar 25, 2022 at 04:51:51PM +0000, Adam Weremczuk wrote:
> [Tue Mar 22 00:24:10 2022] Tasks state (memory values in pages):
> [Tue Mar 22 00:24:10 2022] [ pid ] uid tgid total_vm rss
> pgtables_bytes swapents oom_score_adj name
> [Tue Mar 22 00:24:10 2022] [ 2211] 0 2211 14228 228 159744
> 127 0 systemd
> [Tue Mar 22 00:24:10 2022] [ 2622] 0 2622 93208 59485
> 753664 73 0 systemd-journal
Well, at this point systemd-journald (I assume the name is truncated)
was using more memory than anything else.
> [Tue Mar 22 00:24:10 2022] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0-1,oom_memcg=/lxc/101,task_memcg=/lxc/101/ns,task=systemd-journal,pid=2622,uid=0
> [Tue Mar 22 00:24:10 2022] Memory cgroup out of memory: Killed process 2622
And the kernel killed it.
But there still wasn't enough memory, so it continued:
> [Tue Mar 22 00:24:10 2022] pgsteal 9855948
> [Tue Mar 22 00:24:10 2022] Tasks state (memory values in pages):
> [Tue Mar 22 00:24:10 2022] [ pid ] uid tgid total_vm rss
> pgtables_bytes swapents oom_score_adj name
> [Tue Mar 22 00:24:10 2022] [ 2211] 0 2211 14228 228 159744
> 127 0 systemd
[...]
> [Tue Mar 22 00:24:10 2022] [ 11695] 0 11695 10289 2685
> 114688 0 0 dhcpd
[...]
> [Tue Mar 22 00:24:10 2022] [ 21053] 0 21053 1069 17 53248
> 0 0 apt.systemd.dai
> [Tue Mar 22 00:24:10 2022] [ 21057] 0 21057 1069 31 53248
> 0 0 apt.systemd.dai
> [Tue Mar 22 00:24:10 2022] [ 21065] 0 21065 17753 1712
> 180224 0 0 apt-get
> [Tue Mar 22 00:24:10 2022] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0-1,oom_memcg=/lxc/101,task_memcg=/lxc/101/ns,task=dhcpd,pid=11695,uid=0
> [Tue Mar 22 00:24:10 2022] Memory cgroup out of memory: Killed process 11695
So, next it killed dhcpd. And it still wasn't done.
> [Tue Mar 22 00:24:10 2022] [ 21057] 0 21057 1069 31 53248
> 0 0 apt.systemd.dai
> [Tue Mar 22 00:24:10 2022] [ 21065] 0 21065 17753 2552
> 180224 0 0 apt-get
> [Tue Mar 22 00:24:10 2022] [ 21068] 0 21068 9475 110
> 110592 0 0 systemd-journal
> [Tue Mar 22 00:24:10 2022] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0-1,oom_memcg=/lxc/101,task_memcg=/lxc/101/ns,task=apt-get,pid=21065,uid=0
> [Tue Mar 22 00:24:10 2022] Memory cgroup out of memory: Killed process 21065
At this point, it killed apt-get.
Looks like this system doesn't have enough memory to perform its daily
tasks (including what I'm guessing are unattended upgrades, triggering
calls to apt-get from a systemd timer). You'll either need to stop
letting it run those daily tasks, or add more memory, or add more swap,
or get rid of some of the other programs that are using memory.
If you really want the unattended upgrades, adding more swap would be
the easiest solution, but be warned that this could mean the system
will run extremely slowly during those unattended upgrades. That could
be something you don't care about, or something that matters a lot. Only
you would know.
Reply to: