On Fri, Mar 25, 2022 at 04:51:51PM +0000, Adam Weremczuk wrote:
> [Tue Mar 22 00:24:10 2022] Tasks state (memory values in pages):
> [Tue Mar 22 00:24:10 2022] [ pid ] uid tgid total_vm rss
> pgtables_bytes swapents oom_score_adj name
> [Tue Mar 22 00:24:10 2022] [ 2211] 0 2211 14228 228 159744
> 127 0 systemd
> [Tue Mar 22 00:24:10 2022] [ 2622] 0 2622 93208 59485
> 753664 73 0 systemd-journal
Well, at this point systemd-journald (I assume the name is truncated)
was using more memory than anything else.
.....
> [Tue Mar 22 00:24:10 2022] Memory cgroup out of memory: Killed process 11695
So, next it killed dhcpd. And it still wasn't done.
> [Tue Mar 22 00:24:10 2022] [ 21057] 0 21057 1069 31 53248
> 0 0 apt.systemd.dai
> [Tue Mar 22 00:24:10 2022] [ 21065] 0 21065 17753 2552
> 180224 0 0 apt-get
> [Tue Mar 22 00:24:10 2022] [ 21068] 0 21068 9475 110
> 110592 0 0 systemd-journal
> [Tue Mar 22 00:24:10 2022] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0-1,oom_memcg=/lxc/101,task_memcg=/lxc/101/ns,task=apt-get,pid=21065,uid=0
> [Tue Mar 22 00:24:10 2022] Memory cgroup out of memory: Killed process 21065
At this point, it killed apt-get.
Looks like this system doesn't have enough memory to perform its daily
tasks (including what I'm guessing are unattended upgrades, triggering
calls to apt-get from a systemd timer). You'll either need to stop
letting it run those daily tasks, or add more memory, or add more swap,
or get rid of some of the other programs that are using memory.
If you have re-configured your apt repositories but made a mistake, or lost contact with them for
other network reasons, you will see those automated apt-get commands stack-up over time.
They reach a point in execution where they acquire a lock that blocks all other updates, so just
keep piling up. Each consumes swap. If the CPU workload is the local bottleneck you may notice
a CPU or two pegged at 100% as the first symptom.
If you really want the unattended upgrades, adding more swap would be
the easiest solution, but be warned that this could mean the system
will run extremely slowly during those unattended upgrades. That could
be something you don't care about, or something that matters a lot. Only
you would know.
If it's a server, servers should not swap and they should not get upgraded without purpose.