[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#625217: xen-linux-system-2.6.32-5-xen-amd64: Heavy load on domU causes dom0 to run out of memory

Package: xen-linux-system-2.6.32-5-xen-amd64
Version: 2.6.32-31
Severity: critical
Justification: breaks the whole system

Hi,

I have 64 bit xen kernel from squeeze installed on a dual xeon maschine. Usually everything runs fine until it comes to heavy load on a domU with high I/O and memory consumption.
This causes the dom0 to run out of memory and to kill several processes (see log below). As a consequence of this, the whole system becomes unusable.

I tried several things like assign dedicated memory to dom0, disable balloning, increase scheduler domain weights and assigned dedicated CPUs to dom0 as described in 
http://wiki.xensource.com/xenwiki/XenBestPractices but had no success.

I think a domU should never break the whole system, so this might be a bug. Please let me know if you need further information.

Thanks
Sebastian


May  2 16:05:26 hercules kernel: [ 1768.319877] nrpe invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
May  2 16:05:26 hercules kernel: [ 1768.319883] nrpe cpuset=/ mems_allowed=0
May  2 16:05:26 hercules kernel: [ 1768.319886] Pid: 2118, comm: nrpe Not tainted 2.6.32-5-xen-amd64 #1
May  2 16:05:26 hercules kernel: [ 1768.319889] Call Trace:
May  2 16:05:26 hercules kernel: [ 1768.319899]  [<ffffffff810b7104>] ? oom_kill_process+0x7f/0x23f
May  2 16:05:26 hercules kernel: [ 1768.319903]  [<ffffffff810b7628>] ? __out_of_memory+0x12a/0x141
May  2 16:05:26 hercules kernel: [ 1768.319906]  [<ffffffff810b777f>] ? out_of_memory+0x140/0x172
May  2 16:05:26 hercules kernel: [ 1768.319912]  [<ffffffff81042ace>] ? pick_next_task_fair+0xca/0xd6
May  2 16:05:26 hercules kernel: [ 1768.319916]  [<ffffffff810bb520>] ? __alloc_pages_nodemask+0x4e5/0x5f5
May  2 16:05:26 hercules kernel: [ 1768.319922]  [<ffffffff810d9c04>] ? read_swap_cache_async+0x5d/0xf3
May  2 16:05:27 hercules kernel: [ 1768.319926]  [<ffffffff810d9d23>] ? swapin_readahead+0x89/0x98
May  2 16:05:27 hercules kernel: [ 1768.319931]  [<ffffffff8100c2f1>] ? __raw_callee_save_xen_pte_val+0x11/0x1e
May  2 16:05:27 hercules kernel: [ 1768.319935]  [<ffffffff810cdc18>] ? handle_mm_fault+0x47f/0x80f
May  2 16:05:27 hercules kernel: [ 1768.319940]  [<ffffffff8106d41f>] ? ktime_get_ts+0x68/0xb2
May  2 16:05:27 hercules kernel: [ 1768.319945]  [<ffffffff8130f616>] ? do_page_fault+0x2e0/0x2fc
May  2 16:05:27 hercules kernel: [ 1768.319951]  [<ffffffff8130d4b5>] ? page_fault+0x25/0x30
May  2 16:05:27 hercules kernel: [ 1768.319953] Mem-Info:
May  2 16:05:27 hercules kernel: [ 1768.319955] Node 0 DMA per-cpu:
May  2 16:05:27 hercules kernel: [ 1768.319958] CPU    0: hi:    0, btch:   1 usd:   0
May  2 16:05:27 hercules kernel: [ 1768.319960] CPU    1: hi:    0, btch:   1 usd:   0
May  2 16:05:27 hercules kernel: [ 1768.319961] Node 0 DMA32 per-cpu:
May  2 16:05:27 hercules kernel: [ 1768.319964] CPU    0: hi:  186, btch:  31 usd:  18
May  2 16:05:27 hercules kernel: [ 1768.319966] CPU    1: hi:  186, btch:  31 usd:   0
May  2 16:05:28 hercules kernel: [ 1768.319971] active_anon:1 inactive_anon:25 isolated_anon:0
May  2 16:05:28 hercules kernel: [ 1768.319972]  active_file:15047 inactive_file:196012 isolated_file:128
May  2 16:05:28 hercules kernel: [ 1768.319973]  unevictable:5 dirty:196071 writeback:143 unstable:0
May  2 16:05:28 hercules kernel: [ 1768.319974]  free:2030 slab_reclaimable:14633 slab_unreclaimable:4593
May  2 16:05:28 hercules kernel: [ 1768.319975]  mapped:3592 shmem:0 pagetables:957 bounce:0
May  2 16:05:28 hercules kernel: [ 1768.319977] Node 0 DMA free:4016kB min:52kB low:64kB high:76kB active_anon:0kB inactive_anon:0kB active_file:196kB inactive_file:9112kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13828kB mlocked:0kB dirty:9128kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:592kB slab_unreclaimable:60kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10656 all_unreclaimable? no
May  2 16:05:28 hercules kernel: [ 1768.319986] lowmem_reserve[]: 0 994 994 994
May  2 16:05:28 hercules kernel: [ 1768.319991] Node 0 DMA32 free:4104kB min:4004kB low:5004kB high:6004kB active_anon:4kB inactive_anon:100kB active_file:59992kB inactive_file:774936kB unevictable:20kB isolated(anon):0kB isolated(file):512kB present:1018080kB mlocked:20kB dirty:775156kB writeback:572kB mapped:14368kB shmem:0kB slab_reclaimable:57940kB slab_unreclaimable:18312kB kernel_stack:1376kB pagetables:3828kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:441765 all_unreclaimable? no
May  2 16:05:28 hercules kernel: [ 1768.320001] lowmem_reserve[]: 0 0 0 0
May  2 16:05:28 hercules kernel: [ 1768.320005] Node 0 DMA: 4*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4016kB
May  2 16:05:28 hercules kernel: [ 1768.320019] Node 0 DMA32: 28*4kB 265*8kB 115*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4104kB
May  2 16:05:28 hercules kernel: [ 1768.320030] 211215 total pagecache pages
May  2 16:05:28 hercules kernel: [ 1768.320032] 25 pages in swap cache
May  2 16:05:28 hercules kernel: [ 1768.320034] Swap cache stats: add 19348, delete 19323, find 514/1053
May  2 16:05:28 hercules kernel: [ 1768.320036] Free swap  = 1894804kB
May  2 16:05:28 hercules kernel: [ 1768.320037] Total swap = 1959800kB
May  2 16:05:28 hercules kernel: [ 1768.322969] 262144 pages RAM
May  2 16:05:28 hercules kernel: [ 1768.322972] 22887 pages reserved
May  2 16:05:28 hercules kernel: [ 1768.322973] 218032 pages shared
May  2 16:05:28 hercules kernel: [ 1768.322974] 27151 pages non-shared
May  2 16:05:28 hercules kernel: [ 1768.322978] Out of memory: kill process 2640 (sshd) score 21829 or a child
May  2 16:05:28 hercules kernel: [ 1768.323109] Killed process 2656 (bash)



-- System Information:
Debian Release: 6.0.1
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-5-xen-amd64 (SMP w/2 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages xen-linux-system-2.6.32-5-xen-amd64 depends on:
ii  linux-image-2.6.32-5-xen-amd6 2.6.32-31  Linux 2.6.32 for 64-bit PCs, Xen d
ii  xen-hypervisor-4.0-amd64 [xen 4.0.1-2    The Xen Hypervisor on AMD64

xen-linux-system-2.6.32-5-xen-amd64 recommends no packages.

xen-linux-system-2.6.32-5-xen-amd64 suggests no packages.


-- no debconf information
Reply to: