Bug#549532: kernel: invoked oom-killer: gfp_mask=0x1201d2
Package: kernel
Severity: important
My system (RHEL 5.4 KVM guest) got unaccessible (*very* high cpu load). After rebooting it and checking the logs, I found out that a kernel issue appears:
ct 4 02:02:14 mubuntos3 mount.davfs: open files exceed max cache size by 685 MiBytes
Oct 4 02:07:38 mubuntos3 mount.davfs: open files exceed max cache size by 685 MiBytes
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] pop3-login invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] Pid: 15273, comm: pop3-login Not tainted 2.6.26-2-amd64 #1
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241]
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] Call Trace:
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff802738c0>] oom_kill_process+0x57/0x1dc
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff8023b551>] __capable+0x9/0x1c
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff80273beb>] badness+0x188/0x1c7
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff80273e1f>] out_of_memory+0x1f5/0x28e
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff80276b70>] __alloc_pages_internal+0x31d/0x3bf
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff80278826>] __do_page_cache_readahead+0x79/0x183
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff802730d5>] filemap_fault+0x15d/0x33c
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff8027e554>] __do_fault+0x50/0x3e6
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff8022f0e8>] hrtick_set+0x9e/0xf7
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff802818b7>] handle_mm_fault+0x3f4/0x867
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff8023d020>] del_timer_sync+0xc/0x16
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff8042916a>] schedule_timeout+0x92/0xad
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff80221fbc>] do_page_fault+0x5d8/0x9c8
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff802710da>] sync_page+0x0/0x41
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff8024acb6>] getnstimeofday+0x39/0x98
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff8022c264>] default_wake_function+0x0/0xe
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] [<ffffffff8042a599>] error_exit+0x0/0x60
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241]
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] Mem-info:
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] Node 0 DMA per-cpu:
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] CPU 0: hi: 0, btch: 1 usd: 0
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] CPU 1: hi: 0, btch: 1 usd: 0
Oct 4 02:07:38 mubuntos3 kernel: [372811.164241] Node 0 DMA32 per-cpu:
[...]
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] CPU 0: hi: 186, btch: 31 usd: 181
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] CPU 1: hi: 186, btch: 31 usd: 158
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] Active:46691 inactive:46797 dirty:0 writeback:0 unstable:0
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] free:1189 slab:8849 mapped:18 pagetables:12688 bounce:0
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] Node 0 DMA free:2000kB min:60kB low:72kB high:88kB active:988kB inactive:1568kB present:10792kB pages_scanned:3873 all_unreclaimable? yes
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] lowmem_reserve[]: 0 489 489 489
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] Node 0 DMA32 free:2756kB min:2796kB low:3492kB high:4192kB active:185776kB inactive:185620kB present:500896kB pages_scanned:723643 all_unreclaimable? yes
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] lowmem_reserve[]: 0 0 0 0
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] Node 0 DMA: 114*4kB 11*8kB 1*16kB 1*32kB 0*64kB 1*128kB 1*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 2000kB
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] Node 0 DMA32: 379*4kB 4*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2764kB
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] 99 total pagecache pages
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] Swap cache: add 6478897, delete 6478897, find 372825/1106998
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] Free swap = 0kB
Oct 4 02:07:42 mubuntos3 kernel: [372811.164241] Total swap = 684024kB
Oct 4 02:07:42 mubuntos3 kernel: [372811.168289] 131056 pages of RAM
Oct 4 02:07:42 mubuntos3 kernel: [372811.168289] 3384 reserved pages
Oct 4 02:07:42 mubuntos3 kernel: [372811.168289] 2480 pages shared
Oct 4 02:07:42 mubuntos3 kernel: [372811.168289] 0 pages swap cached
Oct 4 02:07:42 mubuntos3 kernel: [372811.168289] Out of memory: kill process 16835 (mysqld) score 101365 or a child
Oct 4 02:07:42 mubuntos3 kernel: [372811.168289] Killed process 16835 (mysqld)
Oct 4 02:07:42 mubuntos3 kernel: >] system_call_after_swapgs+0x8a/0x8f
Oct 4 02:07:42 mubuntos3 kernel: [372927.778448]
Oct 4 02:07:42 mubuntos3 kernel: [372927.778451] INFO: task bzip2:18310 blocked for more than 120 seconds.
Oct 4 02:07:42 mubuntos3 kernel: [372927.779017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 4 02:07:42 mubuntos3 kernel: [372927.779915] bzip2 D 00007fbfcd336000 0 18310 18309
Oct 4 02:07:42 mubuntos3 kernel: [372927.779918] ffff810001d1be90 0000000000000086 00003ffffffff000 ffff810001d15600
Oct 4 02:07:42 mubuntos3 kernel: [372927.779921] ffff810001a33570 ffff810000d93430 ffff810001a337f8 0000000101018880
Oct 4 02:07:42 mubuntos3 kernel: [372927.779923] ffff8100038a53f8 0000000000000168 ffff8100197b4330 0000000000000001
Oct 4 02:07:42 mubuntos3 kernel: [372927.779926] Call Trace:
Oct 4 02:07:42 mubuntos3 kernel: [372927.779934] [<ffffffff8042952d>] __mutex_lock_slowpath+0x64/0x9b
Oct 4 02:07:42 mubuntos3 kernel: [372927.779938] [<ffffffff80429392>] mutex_lock+0xa/0xb
Oct 4 02:07:42 mubuntos3 kernel: [372927.779943] [<ffffffffa028cfa8>] :fuse:fuse_direct_write+0x2e/0x76
Oct 4 02:07:42 mubuntos3 kernel: [372927.779947] [<ffffffff8029b4e5>] vfs_write+0xad/0x156
Oct 4 02:07:42 mubuntos3 kernel: [372927.779950] [<ffffffff8029ba87>] sys_write+0x45/0x6e
Oct 4 02:07:42 mubuntos3 kernel: [372927.779954] [<ffffffff8020beca>] system_call_after_swapgs+0x8a/0x8f
Oct 4 02:07:42 mubuntos3 kernel: [372927.779959]
I have been running a backup script which bzips some important files to a WebDAV server today, but it had been working flawlessly a few times before.
-- System Information:
Debian Release: 5.0.3
APT prefers stable
APT policy: (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.26-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Reply to: