Re: Out of memory killer misconfigured?
On Wed 20 Apr 2022 at 18:45:47 (+0100), piorunz wrote:
> On 20/04/2022 17:11, Jonathan Dowland wrote:
> > On Wed, Apr 20, 2022 at 04:23:38PM +0100, piorunz wrote:
> > > Sorry but this happened to me a few times, each tome with the same Wine
> > > program. it just loves to eat all available memory when its doing heavy
> > > computations. When I am not careful and I start too many threads, is
> > > eats all memory. My system gets killed every single time. I am not
> > > extrapolating anything. Misbehaving app should get killed, but instead,
> > > Linux kills itself.
> >
> > Ok, not one experience, but one scenario. And there are plausible
> > reasons why this might be happening as outlined in my other mail to the
> > thread (OOM adjustments to make the killer skip over the process).
> > Occam's razor suggests something about your particular setup versus
> > the OOM killer simply being as bad as you think it is. FWIW, I invoke
> > the OOM killer a lot recently (due to some scientific experiments) and
> > the sensible processes were killed in my case, every time, leaving my
> > desktop functional.
> >
> > More data about your setup is needed to fathom out what's going on.
> >
> I didn't configured OOM. I use default Debian Testing with KDE. I
> changed nothing apart from user desktop things. MY situation is not "a
> scenario". It's what everyone can reproduce, just install Wine and
> MetaTester5 program, I can guide you. I cannot guarantee this will
> happen with non-Wine programs.
With respect, the combination of testing + KDE + Wine + MetaTester5
is difficult to categorise as more than one scenario.
> So I just reproduced this again, system crashed totally because I forgot
> to apply choom --adjust 1000 to all metatester processes. Reboot. After
> reboot, I started MT again and applied the following:
>
> renice 19 `pidof metatester64.ex`
> choom -p `pidof metatester64.ex | awk {'print $1'}` --adjust 1000
> choom -p `pidof metatester64.ex | awk {'print $2'}` --adjust 1000
> choom -p `pidof metatester64.ex | awk {'print $3'}` --adjust 1000
> choom -p `pidof metatester64.ex | awk {'print $4'}` --adjust 1000
> choom -p `pidof metatester64.ex | awk {'print $5'}` --adjust 1000
> choom -p `pidof metatester64.ex | awk {'print $6'}` --adjust 1000
> choom -p `pidof metatester64.ex | awk {'print $7'}` --adjust 1000
>
> I spawned more than 8 processes to saturate RAM quickly. Meaning, some
> processes were not adjusted to 1000 score. After 20 seconds, OOM event
> happened.
When I start my window manager, I use a construction like this:
if [ -x /usr/bin/fvwm ]; then
mv -f "$HOME/.xsession-fvwm-$Displaynumber-log" "$HOME/.xsession-fvwm-$Displaynumber-log~"
exec /usr/bin/fvwm > "$HOME/.xsession-fvwm-$Displaynumber-log" 2>&1 & Wmpid=$!
elif [ -x …
…
…
else
printf '%s\n' "Error - no window manager found"
fi
so that .xsession can avoid terminating with:
wait $Wmpid
Would it be sensible for you to use a similar construction so that you
systematically catch /all/ of these processes with their choom commands.
> And what Linux does? It has at least 8 processes with 1000 score, but
> let's see what is does:
>
> $ sudo dmesg | grep "Out of memory"
> [ 704.018238] Out of memory: Killed process 39674 (metatester64.ex)
> total-vm:2939256kB, anon-rss:250524kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:1344kB oom_score_adj:1000
> [ 708.528663] Out of memory: Killed process 25882 (QtWebEngineProc)
> total-vm:5451920kB, anon-rss:20588kB, file-rss:0kB, shmem-rss:116kB,
> UID:1000 pgtables:772kB oom_score_adj:300
> [ 709.811945] Out of memory: Killed process 23411 (krunner-keepass)
> total-vm:52680kB, anon-rss:12860kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:140kB oom_score_adj:200
> [ 710.788999] Out of memory: Killed process 23413 (pipewire-media-)
> total-vm:170052kB, anon-rss:9492kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:248kB oom_score_adj:200
> [ 712.748691] Out of memory: Killed process 23618 (kactivitymanage)
> total-vm:550696kB, anon-rss:8848kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:244kB oom_score_adj:200
> [ 714.856773] Out of memory: Killed process 23414 (pulseaudio)
> total-vm:1384772kB, anon-rss:8508kB, file-rss:0kB, shmem-rss:260kB,
> UID:1000 pgtables:320kB oom_score_adj:200
> [ 716.051274] Out of memory: Killed process 23560 (kglobalaccel5)
> total-vm:282368kB, anon-rss:6728kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:256kB oom_score_adj:200
> [ 716.082495] Out of memory: Killed process 23412 (pipewire)
> total-vm:53124kB, anon-rss:4484kB, file-rss:0kB, shmem-rss:52kB,
> UID:1000 pgtables:104kB oom_score_adj:200
> [ 717.965930] Out of memory: Killed process 23867 (ksystemstats)
> total-vm:166540kB, anon-rss:3844kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:176kB oom_score_adj:200
> [ 718.775772] Out of memory: Killed process 23703 (kscreen_backend)
> total-vm:227996kB, anon-rss:3060kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:164kB oom_score_adj:200
> [ 718.792339] Out of memory: Killed process 23417 (dbus-daemon)
> total-vm:10552kB, anon-rss:1332kB, file-rss:0kB, shmem-rss:0kB, UID:1000
> pgtables:60kB oom_score_adj:200
> [ 719.422552] Out of memory: Killed process 23714 (obexd)
> total-vm:45836kB, anon-rss:732kB, file-rss:0kB, shmem-rss:0kB, UID:1000
> pgtables:76kB oom_score_adj:200
> [ 719.423354] Out of memory: Killed process 23555 (dconf-service)
> total-vm:157600kB, anon-rss:544kB, file-rss:0kB, shmem-rss:0kB, UID:1000
> pgtables:64kB oom_score_adj:200
> [ 719.424525] Out of memory: Killed process 23397 ((sd-pam))
> total-vm:169688kB, anon-rss:3724kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:92kB oom_score_adj:100
> [ 719.425356] Out of memory: Killed process 23396 (systemd)
> total-vm:18600kB, anon-rss:1912kB, file-rss:0kB, shmem-rss:0kB, UID:1000
> pgtables:76kB oom_score_adj:100
> [ 719.426070] Out of memory: Killed process 55808 (metatester64.ex)
> total-vm:5510844kB, anon-rss:3909364kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:8516kB oom_score_adj:0
>
> I lost Metatester (as expected), but I also lost sound and shortcuts and
> KDE widgets. I need to reboot.
>
> Look at memory sizes for last process and other processes killed. Last one:
> Out of memory: Killed process 55808 (metatester64.ex)
> total-vm:5510844kB, anon-rss:3909364kB, file-rss:0kB, shmem-rss:0kB,
> UID:1000 pgtables:8516kB oom_score_adj:0
>
> Why this is not killed right after 39674 (metatester64.ex)?
We don't see all the process numbers listed for these tester programs,
but the 55808 could indicate that it missed being choomed. It might
help to be more systematic with starting the processes and with the
logged information.
> Why other 7 metatester64.ex processes for which I adjusted priority to
> 1000 were not killed before KDE processes were killed?
I've no idea. I don't run a DE, for starters.
Cheers,
David.
Reply to: