Hi,
Hope you can help me with a rather serious failure on one my machines last
night. When I went to access it this morning it was as dead as a door nail.
No network activity, not even local keyboard or video - totally dead. I
initially thought a kernel oops but having a look at the logs makes me think
it might be an out of memory issue. Here's a snippet from the logs:
########### LOG SNIPPET ###########
Dec 6 01:30:47 compost kernel: oom-killer: gfp_mask=0x1d2
Dec 6 01:30:47 compost kernel: DMA per-cpu:
Dec 6 01:30:47 compost kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 6 01:30:47 compost kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 6 01:30:47 compost kernel: Normal per-cpu:
Dec 6 01:30:47 compost kernel: cpu 0 hot: low 62, high 186, batch 31
Dec 6 01:30:47 compost kernel: cpu 0 cold: low 0, high 62, batch 31
Dec 6 01:30:47 compost kernel: HighMem per-cpu: empty
Dec 6 01:30:48 compost kernel:
Dec 6 01:30:48 compost kernel: Free pages: 3688kB (0kB HighMem)
Dec 6 01:30:48 compost kernel: Active:36614 inactive:36018 dirty:0
writeback:0 unstable:0 free:922 slab:3999 mapped:70710 pagetables:1086
Dec 6 01:30:48 compost kernel: DMA free:1328kB min:112kB low:140kB high:168kB
active:5624kB inactive:5196kB present:16384kB pages_scanned:14375
all_unreclaimable? yes
Dec 6 01:30:48 compost kernel: lowmem_reserve[]: 0 304 304
Dec 6 01:30:48 compost kernel: Normal free:2360kB min:2172kB low:2712kB
high:3256kB active:140832kB inactive:138876kB present:311296kB
pages_scanned:235254 all_unreclaimable? no
Dec 6 01:30:48 compost kernel: lowmem_reserve[]: 0 0 0
Dec 6 01:30:48 compost kernel: HighMem free:0kB min:128kB low:160kB
high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Dec 6 01:30:48 compost kernel: lowmem_reserve[]: 0 0 0
Dec 6 01:30:48 compost kernel: DMA: 0*4kB 2*8kB 0*16kB 1*32kB 2*64kB 1*128kB
0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1328kB
Dec 6 01:30:48 compost kernel: Normal: 88*4kB 7*8kB 12*16kB 7*32kB 0*64kB
0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2360kB
Dec 6 01:30:48 compost kernel: HighMem: empty
Dec 6 01:30:48 compost kernel: Swap cache: add 15070980, delete 15070823,
find 13016180/15808735, race 8+53
Dec 6 01:30:48 compost kernel: Free swap = 0kB
Dec 6 01:30:48 compost kernel: Total swap = 393584kB
Dec 6 01:30:48 compost kernel: Out of Memory: Killed process 15517
(postmaster).
########### END LOG SNIPPET ###########
I've never seen anything like that before and there is a lot of it after about
1:30 (the snippet above is the first occurrence). It seems to have
systematically killed off processes presumably to acquire more free memory
until it killed some vital ones at which point the box keeled over an died.