Why was oom-killer invoked?
Hi,
Please help me understand why oom-killer was invoked?
From the SAR logs we can see that there is lots of memory in use by the system cache, but free is low. How can we ensure there is more memory available in free to avoid triggering oom-killer?
From Kernel Log:
Mar 16 23:30:06 kernel: [49173666.630999] postgres invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
Mar 16 23:30:06 kernel: [49173666.631008] Pid: 14865, comm: postgres Not tainted 2.6.26-2-686-bigmem #1
Mar 16 23:30:06 kernel: [49173666.631019] [<c016004e>] oom_kill_process+0x4f/0x195
Mar 16 23:30:06 kernel: [49173666.631028] [<c0160478>] out_of_memory+0x14e/0x17f
Mar 16 23:30:06 kernel: [49173666.631032] [<c01623e0>] __alloc_pages_internal+0x2b8/0x34e
Mar 16 23:30:06 kernel: [49173666.631037] [<c0162482>] __alloc_pages+0x7/0x9
Mar 16 23:30:06 kernel: [49173666.631041] [<c028cdd1>] tcp_sendmsg+0x41b/0x8d9
Mar 16 23:30:06 kernel: [49173666.631048] [<c029468c>] tcp_current_mss+0xaa/0xc8
Mar 16 23:30:06 kernel: [49173666.631052] [<c025fc13>] sock_sendmsg+0xc7/0xe1
Mar 16 23:30:06 kernel: [49173666.631058] [<c0138850>] autoremove_wake_function+0x0/0x2d
Mar 16 23:30:06 kernel: [49173666.631062] [<c029ad8e>] tcp_v4_rcv+0x545/0x597
Mar 16 23:30:06 kernel: [49173666.631066] [<c028368f>] ip_local_deliver_finish+0xe8/0x183
Mar 16 23:30:06 kernel: [49173666.631069] [<c028358a>] ip_rcv_finish+0x286/0x2a3
Mar 16 23:30:06 kernel: [49173666.631075] [<c02604f7>] sys_sendto+0x105/0x130
Mar 16 23:30:06 kernel: [49173666.631077] [<c013e5e2>] clocksource_get_next+0x39/0x3f
Mar 16 23:30:06 kernel: [49173666.631081] [<c013d5da>] update_wall_time+0x519/0x68f
Mar 16 23:30:06 kernel: [49173666.631087] [<c013aaa4>] hrtimer_forward+0xe4/0x100
Mar 16 23:30:06 kernel: [49173666.631090] [<c013cfc4>] getnstimeofday+0x37/0xbc
Mar 16 23:30:06 kernel: [49173666.631095] [<c026053b>] sys_send+0x19/0x1d
Mar 16 23:30:06 kernel: [49173666.631098] [<c0260dd3>] sys_socketcall+0xed/0x19e
Mar 16 23:30:06 kernel: [49173666.631102] [<c010afdb>] do_IRQ+0x52/0x63
Mar 16 23:30:06 kernel: [49173666.631106] [<c0108857>] sysenter_past_esp+0x78/0xb1
Mar 16 23:30:06 kernel: [49173666.631111] =======================
Mar 16 23:30:06 kernel: [49173666.631113] Mem-info:
Mar 16 23:30:06 kernel: [49173666.631114] DMA per-cpu:
Mar 16 23:30:06 kernel: [49173666.631116] CPU 0: hi: 0, btch: 1 usd: 0
Mar 16 23:30:06 kernel: [49173666.631117] CPU 1: hi: 0, btch: 1 usd: 0
Mar 16 23:30:06 kernel: [49173666.631119] CPU 2: hi: 0, btch: 1 usd: 0
Mar 16 23:30:06 kernel: [49173666.631120] CPU 3: hi: 0, btch: 1 usd: 0
Mar 16 23:30:06 kernel: [49173666.631122] Normal per-cpu:
Mar 16 23:30:06 kernel: [49173666.631123] CPU 0: hi: 186, btch: 31 usd: 181
Mar 16 23:30:06 kernel: [49173666.631125] CPU 1: hi: 186, btch: 31 usd: 179
Mar 16 23:30:06 kernel: [49173666.631126] CPU 2: hi: 186, btch: 31 usd: 174
Mar 16 23:30:06 kernel: [49173666.631128] CPU 3: hi: 186, btch: 31 usd: 166
Mar 16 23:30:06 kernel: [49173666.631129] HighMem per-cpu:
Mar 16 23:30:06 kernel: [49173666.631131] CPU 0: hi: 186, btch: 31 usd: 160
Mar 16 23:30:06 kernel: [49173666.631132] CPU 1: hi: 186, btch: 31 usd: 15
Mar 16 23:30:06 kernel: [49173666.631134] CPU 2: hi: 186, btch: 31 usd: 126
Mar 16 23:30:06 kernel: [49173666.631135] CPU 3: hi: 186, btch: 31 usd: 150
Mar 16 23:30:06 kernel: [49173666.631138] Active:1648305 inactive:211079 dirty:222 writeback:0 unstable:0
Mar 16 23:30:06 kernel: [49173666.631139] free:8709 slab:11962 mapped:539697 pagetables:188949 bounce:0
Mar 16 23:30:06 kernel: [49173666.631142] DMA free:3544kB min:68kB low:84kB high:100kB active:8kB inactive:4kB present:16256k
B pages_scanned:0 all_unreclaimable? no
Mar 16 23:30:06 kernel: [49173666.631144] lowmem_reserve[]: 0 873 9128 9128
Mar 16 23:30:06 kernel: [49173666.631148] Normal free:3652kB min:3744kB low:4680kB high:5616kB active:320kB inactive:184kB pr
esent:894080kB pages_scanned:1012 all_unreclaimable? yes
Mar 16 23:30:06 kernel: [49173666.631151] lowmem_reserve[]: 0 0 66040 66040
Mar 16 23:30:06 kernel: [49173666.631155] HighMem free:27640kB min:512kB low:9368kB high:18228kB active:6592892kB inactive:84
4128kB present:8453120kB pages_scanned:0 all_unreclaimable? no
Mar 16 23:30:06 kernel: [49173666.631157] lowmem_reserve[]: 0 0 0 0
Mar 16 23:30:06 kernel: [49173666.631160] DMA: 387*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0
*4096kB = 3588kB
Mar 16 23:30:06 kernel: [49173666.631167] Normal: 21*4kB 6*8kB 2*16kB 3*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB
0*4096kB = 3780kB
Mar 16 23:30:06 kernel: [49173666.631173] HighMem: 2228*4kB 2061*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*
2048kB 0*4096kB = 27432kB
Mar 16 23:30:06 kernel: [49173666.631182] 1086618 total pagecache pages
Mar 16 23:30:06 kernel: [49173666.631184] Swap cache: add 28233, delete 28233, find 12822/15537
Mar 16 23:30:06 kernel: [49173666.631186] Free swap = 995184kB
Mar 16 23:30:06 kernel: [49173666.631187] Total swap = 995988kB
Mar 16 23:30:06 kernel: [49173666.669460] 2359296 pages of RAM
Mar 16 23:30:06 kernel: [49173666.669463] 2129920 pages of HIGHMEM
Mar 16 23:30:06 kernel: [49173666.669465] 281785 reserved pages
Mar 16 23:30:06 kernel: [49173666.669466] 46131460 pages shared
Mar 16 23:30:06 kernel: [49173666.669467] 0 pages swap cached
Mar 16 23:30:06 kernel: [49173666.669468] 222 pages dirty
Mar 16 23:30:06 kernel: [49173666.669470] 0 pages writeback
Mar 16 23:30:06 kernel: [49173666.669471] 539697 pages mapped
Mar 16 23:30:06 kernel: [49173666.669472] 11962 pages slab
Mar 16 23:30:06 kernel: [49173666.669473] 188949 pages pagetables
Mar 16 23:30:06 kernel: [49173666.669476] Out of memory: kill process 27302 (postgres) score 337392 or a child
Mar 16 23:30:06 kernel: [49173666.669567] Killed process 27304 (postgres)
Mar 16 23:30:06 kernel: [49173666.712469] postgres invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
Mar 16 23:30:06 kernel: [49173666.712474] Pid: 14865, comm: postgres Not tainted 2.6.26-2-686-bigmem #1
Mar 16 23:30:06 kernel: [49173666.712489] [<c016004e>] oom_kill_process+0x4f/0x195
Mar 16 23:30:06 kernel: [49173666.712500] [<c0160478>] out_of_memory+0x14e/0x17f
Mar 16 23:30:06 kernel: [49173666.712505] [<c01623e0>] __alloc_pages_internal+0x2b8/0x34e
Mar 16 23:30:06 kernel: [49173666.712510] [<c0162482>] __alloc_pages+0x7/0x9
Mar 16 23:30:06 kernel: [49173666.712513] [<c028cdd1>] tcp_sendmsg+0x41b/0x8d9
Mar 16 23:30:06 kernel: [49173666.712521] [<c029468c>] tcp_current_mss+0xaa/0xc8
Mar 16 23:30:06 kernel: [49173666.712525] [<c025fc13>] sock_sendmsg+0xc7/0xe1
Mar 16 23:30:06 kernel: [49173666.712532] [<c0138850>] autoremove_wake_function+0x0/0x2d
Mar 16 23:30:06 kernel: [49173666.712537] [<c029ad8e>] tcp_v4_rcv+0x545/0x597
Mar 16 23:30:06 kernel: [49173666.712541] [<c028368f>] ip_local_deliver_finish+0xe8/0x183
Mar 16 23:30:06 kernel: [49173666.712544] [<c028358a>] ip_rcv_finish+0x286/0x2a3
Mar 16 23:30:06 kernel: [49173666.712547] [<c0268d31>] netif_receive_skb+0x31c/0x349
Mar 16 23:30:06 kernel: [49173666.712553] [<c02604f7>] sys_sendto+0x105/0x130
Mar 16 23:30:06 kernel: [49173666.712556] [<c013e5e2>] clocksource_get_next+0x39/0x3f
Mar 16 23:30:06 kernel: [49173666.712561] [<c013d5da>] update_wall_time+0x519/0x68f
Mar 16 23:30:06 kernel: [49173666.712567] [<c013aaa4>] hrtimer_forward+0xe4/0x100
Mar 16 23:30:06 kernel: [49173666.712570] [<c013cfc4>] getnstimeofday+0x37/0xbc
Mar 16 23:30:06 kernel: [49173666.712575] [<c026053b>] sys_send+0x19/0x1d
Mar 16 23:30:06 kernel: [49173666.712578] [<c0260dd3>] sys_socketcall+0xed/0x19e
Mar 16 23:30:06 kernel: [49173666.712582] [<c010afdb>] do_IRQ+0x52/0x63
Mar 16 23:30:06 kernel: [49173666.712586] [<c0108857>] sysenter_past_esp+0x78/0xb1
Mar 16 23:30:06 kernel: [49173666.712598] =======================
Mar 16 23:30:06 kernel: [49173666.712600] Mem-info:
Mar 16 23:30:06 kernel: [49173666.712601] DMA per-cpu:
Mar 16 23:30:06 kernel: [49173666.712603] CPU 0: hi: 0, btch: 1 usd: 0
Mar 16 23:30:06 kernel: [49173666.712604] CPU 1: hi: 0, btch: 1 usd: 0
Mar 16 23:30:06 kernel: [49173666.712606] CPU 2: hi: 0, btch: 1 usd: 0
Mar 16 23:30:06 kernel: [49173666.712607] CPU 3: hi: 0, btch: 1 usd: 0
Mar 16 23:30:06 kernel: [49173666.712609] Normal per-cpu:
Mar 16 23:30:06 kernel: [49173666.712610] CPU 0: hi: 186, btch: 31 usd: 156
Mar 16 23:30:06 kernel: [49173666.712612] CPU 1: hi: 186, btch: 31 usd: 155
Mar 16 23:30:06 kernel: [49173666.712614] CPU 2: hi: 186, btch: 31 usd: 177
Mar 16 23:30:06 kernel: [49173666.712615] CPU 3: hi: 186, btch: 31 usd: 180
Mar 16 23:30:06 kernel: [49173666.712616] HighMem per-cpu:
Mar 16 23:30:06 kernel: [49173666.712618] CPU 0: hi: 186, btch: 31 usd: 159
Mar 16 23:30:06 kernel: [49173666.712620] CPU 1: hi: 186, btch: 31 usd: 13
Mar 16 23:30:06 kernel: [49173666.712621] CPU 2: hi: 186, btch: 31 usd: 126
Mar 16 23:30:06 kernel: [49173666.712623] CPU 3: hi: 186, btch: 31 usd: 149
Mar 16 23:30:06 kernel: [49173666.712625] Active:1647934 inactive:211078 dirty:3 writeback:0 unstable:0
Mar 16 23:30:06 kernel: [49173666.712626] free:9053 slab:11962 mapped:537653 pagetables:188949 bounce:0
Mar 16 23:30:06 kernel: [49173666.712629] DMA free:3544kB min:68kB low:84kB high:100kB active:0kB inactive:44kB present:16256
kB pages_scanned:33 all_unreclaimable? no
Mar 16 23:30:06 kernel: [49173666.712632] lowmem_reserve[]: 0 873 9128 9128
Mar 16 23:30:06 kernel: [49173666.712636] Normal free:3852kB min:3744kB low:4680kB high:5616kB active:332kB inactive:140kB pr
esent:894080kB pages_scanned:917 all_unreclaimable? yes
Mar 16 23:30:06 kernel: [49173666.712638] lowmem_reserve[]: 0 0 66040 66040
Mar 16 23:30:06 kernel: [49173666.712642] HighMem free:28816kB min:512kB low:9368kB high:18228kB active:6591432kB inactive:84
4128kB present:8453120kB pages_scanned:0 all_unreclaimable? no
Mar 16 23:30:06 kernel: [49173666.712644] lowmem_reserve[]: 0 0 0 0
Mar 16 23:30:06 kernel: [49173666.712647] DMA: 388*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0
*4096kB = 3592kB
Mar 16 23:30:06 kernel: [49173666.712654] Normal: 83*4kB 6*8kB 2*16kB 3*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB
0*4096kB = 4028kB
Mar 16 23:30:06 kernel: [49173666.712661] HighMem: 2565*4kB 2063*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*
2048kB 0*4096kB = 28796kB
Mar 16 23:30:06 kernel: [49173666.712668] 1086618 total pagecache pages
Mar 16 23:30:06 kernel: [49173666.712670] Swap cache: add 28233, delete 28233, find 12822/15537
Mar 16 23:30:06 kernel: [49173666.712671] Free swap = 995184kB
Mar 16 23:30:06 kernel: [49173666.712673] Total swap = 995988kB
Mar 16 23:30:06 kernel: [49173666.739537] 2359296 pages of RAM
Mar 16 23:30:06 kernel: [49173666.739537] 2129920 pages of HIGHMEM
Mar 16 23:30:06 kernel: [49173666.739537] 281785 reserved pages
Mar 16 23:30:06 kernel: [49173666.739537] 45818784 pages shared
Mar 16 23:30:06 kernel: [49173666.739537] 0 pages swap cached
Mar 16 23:30:06 kernel: [49173666.739537] 3 pages dirty
Mar 16 23:30:06 kernel: [49173666.739537] 0 pages writeback
Mar 16 23:30:06 kernel: [49173666.739537] 536777 pages mapped
Mar 16 23:30:06 kernel: [49173666.739537] 11962 pages slab
Mar 16 23:30:06 kernel: [49173666.739537] 188949 pages pagetables
Mar 16 23:30:06 kernel: [49173666.739537] Out of memory: kill process 27302 (postgres) score 335971 or a child
Mar 16 23:30:06 kernel: [49173666.739537] Killed process 27305 (postgres)
Output from SAR:
22:05:01 28904 8281140 99.65 10552 4357216 995184 804 0.08 0
22:15:02 27056 8282988 99.67 11428 4357428 995184 804 0.08 0
22:25:01 26740 8283304 99.68 10404 4359928 995184 804 0.08 0
22:35:01 27020 8283024 99.67 11748 4357312 995184 804 0.08 0
22:45:01 29172 8280872 99.65 10872 4355748 995184 804 0.08 0
22:55:02 29092 8280952 99.65 8352 4357176 995184 804 0.08 0
23:05:01 94768 8215276 98.86 13384 4283616 995184 804 0.08 0
23:15:01 23720 8286324 99.71 10372 4360112 995184 804 0.08 0
23:25:01 34256 8275788 99.59 6896 4352652 995184 804 0.08 0
23:35:01 3809472 4500572 54.16 1652 4346532 995184 804 0.08 0
23:45:01 3220276 5089768 61.25 29880 4891584 995184 804 0.08 0
23:55:01 3219504 5090540 61.26 30656 4891720 995184 804 0.08 0
23:59:01 3219276 5090768 61.26 30996 4891776 995184 804 0.08 0
00:00:01 3218864 5091180 61.27 31092 4891808 995184 804 0.08 0
Average: 172503 8137541 97.92 16271 4345654 995204 784 0.08 4
total used free shared buffers cached
Mem: 8115 7968 146 0 23 5575
-/+ buffers/cache: 2369 5745
Swap: 972 127 845
TIA,
Mit
Reply to: