Bug#700755: marked as done (huge slab_unreclaimable in Xen domU)

To: Ben Hutchings <benh@debian.org>
Subject: Bug#700755: marked as done (huge slab_unreclaimable in Xen domU)
From: owner@bugs.debian.org (Debian Bug Tracking System)
Date: Sun, 29 May 2016 20:28:11 +0000
Message-id: <[🔎] handler.700755.D700755.1464553499911.ackdone@bugs.debian.org>
References: 6b4b485b-8381-4de4-bee3-82a89519ae57@decadent.org.uk <20130216232240.GA9728@entuzijast.net>

Your message dated Sun, 29 May 2016 21:24:52 +0100
with message-id 6b4b485b-8381-4de4-bee3-82a89519ae57@decadent.org.uk
and subject line Closing bugs assigned to linux-2.6 package
has caused the Debian Bug report #700755,
regarding huge slab_unreclaimable in Xen domU
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
700755: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700755
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

--- Begin Message ---

To: submit@bugs.debian.org
Subject: huge slab_unreclaimable in Xen domU
From: Josip Rodin <joy@debbugs.entuzijast.net>
Date: Sun, 17 Feb 2013 00:22:40 +0100
Message-id: <20130216232240.GA9728@entuzijast.net>

Package: linux-image-2.6.32-5-xen-amd64
Version: 2.6.32-46

Moved out of #519586 because duplicating e-mails is easy :p

----- Forwarded message from Josip Rodin <joy@debbugs.entuzijast.net> -----

Date: Tue, 22 Jan 2013 10:59:17 +0100
From: Josip Rodin <joy@debbugs.entuzijast.net>
To: 519586@bugs.debian.org
Cc: control@bugs.debian.org, 519586-submitter@bugs.debian.org
Subject: Re: Huge "Slab Unreclaimable" and continually growing

unarchive 519586
reopen 519586
reassign 519586 linux-image-2.6.32-5-xen-amd64
found 519586 2.6.32-46
retitle 519586 huge slab_unreclaimable in Xen domU
thanks

Hi,

I appear to be experiencing a serious problem with a 768 MB RAM Xen domU
machine running an NFS client - every now and then (for months now), often
in the middle of the night, it enters some kind of a broken state where a
few semi-random processes (mainly apache2's and vsftpd's which are told to
serve files from the NFS mount) start battling it out for the memory, and
everything including sshd starts invoking the OOM killer, over and over
again. Nothing seems to halt the downward spiral; manual invocation of the
OOM killer does nothing of any use. Terminating all processes is the only
thing that makes it go quiet, but then that's effectively the same as a
reboot.

This is the SysRq+M output on the machine once it's been in the broken state
for a while:

SysRq : Show Memory
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd:  35
CPU    1: hi:  186, btch:  31 usd:   0
active_anon:394 inactive_anon:3197 isolated_anon:0
 active_file:25 inactive_file:176 isolated_file:32
 unevictable:2659 dirty:1 writeback:0 unstable:0
 free:21456 slab_reclaimable:16177 slab_unreclaimable:143165
 mapped:677 shmem:76 pagetables:455 bounce:0
Node 0 DMA free:3432kB min:1212kB low:1512kB high:1816kB active_anon:0kB inactive_anon:0kB active_file:0kB inac
lowmem_reserve[]: 0 741 741 741
Node 0 DMA32 free:82392kB min:64320kB low:80400kB high:96480kB active_anon:1576kB inactive_anon:12788kB active_
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 42*4kB 42*8kB 45*16kB 15*32kB 5*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 3432kB
Node 0 DMA32: 9560*4kB 4201*8kB 365*16kB 41*32kB 15*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 8
906 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 0kB
Total swap = 0kB
196608 pages RAM
5319 pages reserved
3301 pages shared
168474 pages non-shared

While this is when it's working normally:

SysRq : Show Memory
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 123
CPU    1: hi:  186, btch:  31 usd:  60
active_anon:13098 inactive_anon:72 isolated_anon:0
 active_file:10401 inactive_file:58662 isolated_file:0
 unevictable:1902 dirty:8 writeback:0 unstable:0
 free:92863 slab_reclaimable:9038 slab_unreclaimable:1166
 mapped:2560 shmem:75 pagetables:597 bounce:0
Node 0 DMA free:14524kB min:1212kB low:1512kB high:1816kB active_anon:0kB inactive_anon:0kB active_file:0kB inac
lowmem_reserve[]: 0 741 741 741
Node 0 DMA32 free:356928kB min:64320kB low:80400kB high:96480kB active_anon:52392kB inactive_anon:288kB active_f
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 2*64kB 2*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 3*4096kB = 14524kB
Node 0 DMA32: 32*4kB 145*8kB 84*16kB 40*32kB 22*64kB 15*128kB 12*256kB 21*512kB 10*1024kB 3*2048kB 78*4096kB = 3
69760 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 0kB
Total swap = 0kB
196608 pages RAM
5319 pages reserved
17310 pages shared
85757 pages non-shared

The thing I noticed was the slab_unreclaimable explosion, by a factor
of 122. That... doesn't sound like something that should be happenning.

Googling for slab_unreclaimable found me this old bug report about
slab_unreclaimable domU problems that was mass-closed with the switch to the
new paravirtops Xen release. Granted, our use case is not Samba like with
the original reporter, but the pattern of a file server was close enough for
me to be uncomfortable with it :|

I'm going to try to run slabtop the next time I catch it in this state,
in order to try to glean some more information.

Here's some more SysRq output from the broken state, hope it's informative:

[643359.895958] SysRq : Show State
[643359.895973]   task                        PC stack   pid father
[643359.895981] init          D ffff88000256aa60     0     1      0 0x00000000
[643359.895994]  ffff88000256aa60 0000000000000282 ffffffff810bb9e1 000000010995142e
[643359.896013]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002fdcbfd8
[643359.896029]  0000000000015780 0000000000015780 ffff88002fdd0000 ffff88002fdd02f8
[643359.896045] Call Trace:
[643359.896061]  [<ffffffff810bb9e1>] ? __set_page_dirty_no_writeback+0x0/0xc
[643359.896073]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.896083]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.896092]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.896104]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.896116]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.896126]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.896135]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.896145]  [<ffffffff810bb9e1>] ? __set_page_dirty_no_writeback+0x0/0xc
[643359.896155]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643359.896166]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643359.896176]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.896187]  [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[643359.896196]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.896205]  [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[643359.896215]  [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[643359.896225]  [<ffffffff81045e7e>] ? find_busiest_group+0x420/0xa4e
[643359.896235]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.896244]  [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[643359.896254]  [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[643359.896263]  [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[643359.896273]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643359.896283]  [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643359.896293]  [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[643359.896302]  [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[643359.896311]  [<ffffffff810660c0>] ? wake_bit_function+0x0/0x23
[643359.896321]  [<ffffffff810fa746>] ? getname+0x23/0x1a0
[643359.896330]  [<ffffffff810b54d5>] ? lock_page+0x9/0x1f
[643359.896339]  [<ffffffff810fa746>] ? getname+0x23/0x1a0
[643359.896347]  [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0
[643359.897424]  [<ffffffff810fa746>] ? getname+0x23/0x1a0
[643359.897433]  [<ffffffff810fb1a6>] ? user_path_at+0x20/0x79
[643359.897442]  [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643359.897452]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.897461]  [<ffffffff810f361a>] ? vfs_fstatat+0x2c/0x57
[643359.897470]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.897480]  [<ffffffff810f370d>] ? sys_newstat+0x11/0x30
[643359.897489]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.897498]  [<ffffffff8100b4d3>] ? xen_mc_issue+0x21/0x23
[643359.897508]  [<ffffffff8101295b>] ? device_not_available+0x1b/0x20
[643359.897518]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.897526] kthreadd      S 0000000000000000     0     2      0 0x00000000
[643359.897538]  ffffffff814a11f0 0000000000000246 0000000000000000 0000000000000001
[643359.897554]  00000020304045a0 ffffffff81012b42 000000000000f9e0 ffff88002fdd9fd8
[643359.897570]  0000000000015780 0000000000015780 ffff88002fdd0710 ffff88002fdd0a08
[643359.897586] Call Trace:
[643359.897593]  [<ffffffff81012b42>] ? kernel_thread+0x82/0xe0
[643359.897603]  [<ffffffff81065cd1>] ? kthreadd+0x71/0xec
[643359.897612]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.897621]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.897630]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.897641]  [<ffffffff810e178d>] ? do_set_mempolicy+0x128/0x13a
[643359.897650]  [<ffffffff810e178d>] ? do_set_mempolicy+0x128/0x13a
[643359.897659]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.897666] migration/0   S 0000000000000000     0     3      2 0x00000000
[643359.897679]  ffffffff814a11f0 0000000000000246 0000000000000000 ffff8800031947e8
[643359.897694]  ffff88000265b8b8 ffff88000265b8b8 000000000000f9e0 ffff88002fdddfd8
[643359.897710]  0000000000015780 0000000000015780 ffff88002fdd1530 ffff88002fdd1828
[643359.897725] Call Trace:
[643359.897733]  [<ffffffff8100f0bc>] ? xen_smp_send_reschedule+0x0/0x7
[643359.897744]  [<ffffffff8104c304>] ? migration_thread+0x161/0x236
[643359.897753]  [<ffffffff8104c1a3>] ? migration_thread+0x0/0x236
[643359.897762]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.897771]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.897779]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.897788]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.897798]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.897806] ksoftirqd/0   S ffff88000256aa60     0     4      2 0x00000000
[643359.897817]  ffff88000256aa60 0000000000000246 0000000000000000 ffff88002fddfe7c
[643359.897834]  000000000000e030 0000000000000246 000000000000f9e0 ffff88002fddffd8
[643359.897849]  0000000000015780 0000000000015780 ffff88002fdd1c40 ffff88002fdd1f38
[643359.897865] Call Trace:
[643359.897873]  [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[643359.897884]  [<ffffffff810549cd>] ? ksoftirqd+0x46/0xd3
[643359.897893]  [<ffffffff81054987>] ? ksoftirqd+0x0/0xd3
[643359.897902]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.897910]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.897919]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.897928]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.897938]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.897945] watchdog/0    S 0000000000000001     0     5      2 0x00000000
[643359.897957]  ffff88002fdd2a60 0000000000000246 ffffffff8104a1b9 ffff880003194780
[643359.897973]  ffff880003194780 ffffffff8104a1b9 000000000000f9e0 ffff88002fde9fd8
[643359.897989]  0000000000015780 0000000000015780 ffff88002fdd2350 ffff88002fdd2648
[643359.898005] Call Trace:
[643359.898013]  [<ffffffff8104a1b9>] ? switched_to_rt+0x0/0x58
[643359.898022]  [<ffffffff8104a1b9>] ? switched_to_rt+0x0/0x58
[643359.898032]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.898043]  [<ffffffff8106a53b>] ? cpu_clock+0x28/0x30
[643359.898053]  [<ffffffff810954c7>] ? watchdog+0x0/0x75
[643359.898061]  [<ffffffff8109550b>] ? watchdog+0x44/0x75
[643359.898070]  [<ffffffff810954c7>] ? watchdog+0x0/0x75
[643359.898078]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.898087]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.898096]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.898105]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.898114]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.898121] migration/1   S 0000000000000000     0     6      2 0x00000000
[643359.898133]  ffff88002fdd0e20 0000000000000246 0000000000000000 ffff8800031b27e8
[643359.898149]  ffff88000265f138 ffff88000265f138 000000000000f9e0 ffff88002fdebfd8
[643359.898165]  0000000000015780 0000000000015780 ffff88002fdd2a60 ffff88002fdd2d58
[643359.898181] Call Trace:
[643359.898189]  [<ffffffff8100f0bc>] ? xen_smp_send_reschedule+0x0/0x7
[643359.898198]  [<ffffffff8104c1a3>] ? migration_thread+0x0/0x236
[643359.898207]  [<ffffffff8104c304>] ? migration_thread+0x161/0x236
[643359.898217]  [<ffffffff8104c1a3>] ? migration_thread+0x0/0x236
[643359.898226]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.898235]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.898243]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.898252]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.898262]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.898269] ksoftirqd/1   S 0000000000000000     0     7      2 0x00000000
[643359.898281]  ffff88002fdd0e20 0000000000000246 0000000000000000 ffff88002fdf1e7c
[643359.898297]  000000000000e030 0000000000000246 000000000000f9e0 ffff88002fdf1fd8
[643359.898312]  0000000000015780 0000000000015780 ffff88002fdd3170 ffff88002fdd3468
[643359.898328] Call Trace:
[643359.898336]  [<ffffffff810549cd>] ? ksoftirqd+0x46/0xd3
[643359.898345]  [<ffffffff81054987>] ? ksoftirqd+0x0/0xd3
[643359.898353]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.898362]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.898371]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.898380]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.898389]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.898396] watchdog/1    S 0000000000000000     0     8      2 0x00000000
[643359.898408]  ffff88002fdd0e20 0000000000000246 ffffffff8104a1b9 ffff8800031b2780
[643359.898424]  ffff8800031b2780 ffffffff8104a1b9 000000000000f9e0 ffff88002fdf3fd8
[643359.898439]  0000000000015780 0000000000015780 ffff88002fdd3880 ffff88002fdd3b78
[643359.898455] Call Trace:
[643359.898462]  [<ffffffff8104a1b9>] ? switched_to_rt+0x0/0x58
[643359.898472]  [<ffffffff8104a1b9>] ? switched_to_rt+0x0/0x58
[643359.898481]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.898491]  [<ffffffff8106a53b>] ? cpu_clock+0x28/0x30
[643359.898500]  [<ffffffff810954c7>] ? watchdog+0x0/0x75
[643359.898508]  [<ffffffff8109550b>] ? watchdog+0x44/0x75
[643359.898517]  [<ffffffff810954c7>] ? watchdog+0x0/0x75
[643359.898525]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.898534]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.898542]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.898551]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.898560]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.898567] events/0      S ffff88000256aa60     0     9      2 0x00000000
[643359.898579]  ffff88000256aa60 0000000000000246 00000000000000fa 00000000000000f9
[643359.898595]  0000000109950214 0000000109950214 000000000000f9e0 ffff88002fdfdfd8
[643359.898610]  0000000000015780 0000000000015780 ffff88002fdd3f90 ffff88002fdd4288
[643359.898626] Call Trace:
[643359.898633]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.898643]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.898652]  [<ffffffff810c93a8>] ? vmstat_update+0x0/0x39
[643359.898661]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.898671]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.898680]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.898688]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] events/1      S 0000000000000000     0    10      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000000000000000 00000000000000f9
[643359.899950]  0000000109950211 0000000109950211 000000000000f9e0 ffff88002fdfffd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fdd46a0 ffff88002fdd4998
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff810c93a8>] ? vmstat_update+0x0/0x39
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] cpuset        S ffff8800031acc40     0    11      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 ffff88002fe01e00 ffff88002fe01dfc
[643359.899950]  0000000000000000 0000000000000000 000000000000f9e0 ffff88002fe01fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fdd4db0 ffff88002fdd50a8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] khelper       S 0000000000000000     0    12      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000000000000000 ffff88002fdd54c0
[643359.899950]  ffff88002e7ce248 ffff88002e7ce240 000000000000f9e0 ffff88002fe03fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fdd54c0 ffff88002fdd57b8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff81061251>] ? __call_usermodehelper+0x0/0x69
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] netns         S 0000000000000000     0    13      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000000000000000 0000000000000000
[643359.899950]  0000000000000000 0000000000000000 000000000000f9e0 ffff88002fe29fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fdd5bd0 ffff88002fdd5ec8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] async/mgr     S ffff8800031acc40     0    14      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 ffff88002fe2be50 ffff88002fe2be4c
[643359.899950]  0000000000000001 ffff8800031b2780 000000000000f9e0 ffff88002fe2bfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fdd62e0 ffff88002fdd65d8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8106afbe>] ? async_manager_thread+0xae/0xd3
[643359.899950]  [<ffffffff8104b4bc>] ? default_wake_function+0x0/0x9
[643359.899950]  [<ffffffff8106af10>] ? async_manager_thread+0x0/0xd3
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] pm            S 0000000000000001     0    15      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000000000000000 0000000000000000
[643359.899950]  0000000000000000 0000000000000000 000000000000f9e0 ffff88002fe2dfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fdd69f0 ffff88002fdd6ce8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] xenwatch      S ffff88002fdd7100     0    16      2 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000246 ffffffff8100ecf2 0000000000000001
[643359.899950]  0000000000000010 ffffffff814964f0 000000000000f9e0 ffff88002fe2ffd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fdd7100 ffff88002fdd73f8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffffa0000a18>] ? netback_changed+0x0/0x8e3 [xen_netfront]
[643359.899950]  [<ffffffff811f62b4>] ? xenwatch_thread+0x87/0x14a
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff811f622d>] ? xenwatch_thread+0x0/0x14a
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] xenbus        S 0000000000000001     0    17      2 0x00000000
[643359.899950]  ffff88002fdd7100 0000000000000246 ffff88002fdd7138 ffff88002fdd7848
[643359.899950]  0000000000000000 0000000000000000 000000000000f9e0 ffff88002fe35fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fdd7810 ffff88002fdd7b08
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[643359.899950]  [<ffffffff811f5faf>] ? xenbus_thread+0x0/0x27e
[643359.899950]  [<ffffffff811f5b1a>] ? xb_wait_for_data_to_read+0x98/0xb4
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff811f5fb8>] ? xenbus_thread+0x9/0x27e
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] sync_supers   S 0000000000000000     0    18      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000000000000000 00000020304045a0
[643359.899950]  0000000000000001 ffffffff810c971a 000000000000f9e0 ffff88002fe49fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe40000 ffff88002fe402f8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff810c971a>] ? bdi_sync_supers+0x0/0x3f
[643359.899950]  [<ffffffff810f2046>] ? sync_supers+0x82/0x99
[643359.899950]  [<ffffffff810c971a>] ? bdi_sync_supers+0x0/0x3f
[643359.899950]  [<ffffffff810c9745>] ? bdi_sync_supers+0x2b/0x3f
[643359.899950]  [<ffffffff810c971a>] ? bdi_sync_supers+0x0/0x3f
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] bdi-default   S 0000000000000000     0    19      2 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000246 0000000000000000 0000000000000170
[643359.899950]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002fe4bfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe40710 ffff88002fe40a08
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.899950]  [<ffffffff8105b9ec>] ? try_to_del_timer_sync+0x63/0x6c
[643359.899950]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.899950]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.899950]  [<ffffffff810ca1be>] ? bdi_forker_task+0x19a/0x2d8
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8100f0bc>] ? xen_smp_send_reschedule+0x0/0x7
[643359.899950]  [<ffffffff810ca024>] ? bdi_forker_task+0x0/0x2d8
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kintegrityd/0 S 0000000000000001     0    20      2 0x00000000
[643359.899950]  ffff88002fdd0000 0000000000000246 0000001000000000 ffff88002fdd0000
[643359.899950]  ffff880003194780 ffff8800031b27e8 000000000000f9e0 ffff88002fe4dfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe40e20 ffff88002fe41118
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kintegrityd/1 S 0000000000000001     0    21      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000001000000000 ffff88002fdd0000
[643359.899950]  ffff880003194780 ffff8800031b27e8 000000000000f9e0 ffff88002fe4ffd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe41530 ffff88002fe41828
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kblockd/0     S 0000000000000000     0    22      2 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000246 0000000000000000 ffffffffa000a751
[643359.899950]  000000000000e030 ffff880002110000 000000000000f9e0 ffff88002fe5dfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe41c40 ffff88002fe41f38
[643359.899950] Call Trace:
[643359.899950]  [<ffffffffa000a751>] ? do_blkif_request+0x316/0x374 [xen_blkfront]
[643359.899950]  [<ffffffff8118a7bc>] ? cfq_queue_empty+0x0/0x13
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff8118b9dc>] ? cfq_kick_queue+0x0/0x41
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kblockd/1     S 0000000000000000     0    23      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000000000000000 ffffffffa000a7a0
[643359.899950]  ffff88000255b9e0 ffff880002110000 000000000000f9e0 ffff88002fe5ffd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe42350 ffff88002fe42648
[643359.899950] Call Trace:
[643359.899950]  [<ffffffffa000a7a0>] ? do_blkif_request+0x365/0x374 [xen_blkfront]
[643359.899950]  [<ffffffff8118a7bc>] ? cfq_queue_empty+0x0/0x13
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff8118b9dc>] ? cfq_kick_queue+0x0/0x41
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kseriod       S 0000000000000000     0    24      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000000000000000 ffff8800031b2780
[643359.899950]  0000000000000001 ffffffff81041497 000000000000f9e0 ffff88002fe61fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe42a60 ffff88002fe42d58
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81041497>] ? dequeue_entity+0x18/0x135
[643359.899950]  [<ffffffff8123b986>] ? serio_thread+0x304/0x373
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8123b682>] ? serio_thread+0x0/0x373
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kondemand/0   S 0000000000000000     0    27      2 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000246 0000000000000000 ffffffff81040c8e
[643359.899950]  ffff8800031b2780 ffff8800031b27e8 000000000000f9e0 ffff88002fe99fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe43f90 ffff88002fe44288
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81040c8e>] ? update_curr+0xa6/0x147
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kondemand/1   S 0000000000000001     0    28      2 0x00000000
[643359.899950]  ffff88002fdd0000 0000000000000246 ffff88002fe446d8 ffffffff81040c8e
[643359.899950]  ffff8800031b2780 ffff8800031b27e8 000000000000f9e0 ffff88002fe9bfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe446a0 ffff88002fe44998
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81040c8e>] ? update_curr+0xa6/0x147
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] khungtaskd    S 0000000000000000     0    29      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000000000000000 0000000000000150
[643359.899950]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002dfbbfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe43170 ffff88002fe43468
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.899950]  [<ffffffff8105b9ec>] ? try_to_del_timer_sync+0x63/0x6c
[643359.899950]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.899950]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.899950]  [<ffffffff81095795>] ? watchdog+0x40/0x1ba
[643359.899950]  [<ffffffff81095755>] ? watchdog+0x0/0x1ba
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kswapd0       D 0000000000000000     0    30      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000000000000000 0000000109951433
[643359.899950]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002dfbdfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe44db0 ffff88002fe450a8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.899950]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.899950]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.899950]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643359.899950]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810c11a8>] ? kswapd+0x5f9/0x686
[643359.899950]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff810c0baf>] ? kswapd+0x0/0x686
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] ksmd          S 0000000000000001     0    31      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 ffff88002fe454c0 ffff88002fe457c0
[643359.899950]  0000000100000001 ffff88002dfbe000 000000000000f9e0 ffff88002dfbffd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe454c0 ffff88002fe457b8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff810e4673>] ? ksm_scan_thread+0x0/0xa1a
[643359.899950]  [<ffffffff810e502a>] ? ksm_scan_thread+0x9b7/0xa1a
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810e4673>] ? ksm_scan_thread+0x0/0xa1a
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] aio/0         S 0000000000000000     0    32      2 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000246 0000000000000000 ffffffff81040c8e
[643359.899950]  ffff8800031b2780 ffff8800031b27e8 000000000000f9e0 ffff88002dfe1fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe45bd0 ffff88002fe45ec8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81040c8e>] ? update_curr+0xa6/0x147
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] aio/1         S 0000000000000001     0    33      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 ffff88002fe46318 ffffffff81040c8e
[643359.899950]  ffff8800031b2780 ffff8800031b27e8 000000000000f9e0 ffff88002dfe3fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe462e0 ffff88002fe465d8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81040c8e>] ? update_curr+0xa6/0x147
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] crypto/0      S 0000000000000000     0    34      2 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000246 0000000000000000 ffffffff81040c8e
[643359.899950]  ffff8800031b2780 ffff8800031b27e8 000000000000f9e0 ffff88002dfe7fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe469f0 ffff88002fe46ce8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81040c8e>] ? update_curr+0xa6/0x147
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] crypto/1      S 0000000000000001     0    35      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 ffff88002fe47138 ffffffff81040c8e
[643359.899950]  ffff8800031b2780 ffff8800031b27e8 000000000000f9e0 ffff88002dff5fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe47100 ffff88002fe473f8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81040c8e>] ? update_curr+0xa6/0x147
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] khvcd         S 0000000000000000     0    38      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000000000000000 0000000000000000
[643359.899950]  0000000000000200 0000000000000200 000000000000f9e0 ffff88000250bfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002fe43880 ffff88002fe43b78
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8121db33>] ? khvcd+0xba/0x107
[643359.899950]  [<ffffffff8121da79>] ? khvcd+0x0/0x107
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kjournald     S ffff88000256aa60     0   114      2 0x00000000
[643359.899950]  ffff88000256aa60 0000000000000246 ffff880002122424 ffff88001c4e9060
[643359.899950]  ffff880002180000 0000000300200200 000000000000f9e0 ffff880002125fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88000256f100 ffff88000256f3f8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[643359.899950]  [<ffffffffa00201ef>] ? kjournald+0x1a6/0x226 [jbd]
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffffa0020049>] ? kjournald+0x0/0x226 [jbd]
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] udevd         S ffff88000265e9f0     0   165      1 0x00000000
[643359.899950]  ffff88000265e9f0 0000000000000286 ffff8800027b5a58 ffff8800027b5a54
[643359.899950]  dead000000200200 ffff8800027b5a98 000000000000f9e0 ffff8800027b5fd8
[643359.899950]  0000000000015780 0000000000015780 ffff880002568710 ffff880002568a08
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff81068d06>] ? hrtimer_try_to_cancel+0x3a/0x43
[643359.899950]  [<ffffffff8130d6d4>] ? schedule_hrtimeout_range+0x4a/0x112
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff812c35be>] ? unix_dgram_poll+0x0/0x139
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950]  [<ffffffff810fd3a0>] ? do_sys_poll+0x2f8/0x391
[643359.899950]  [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950]  [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950]  [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950]  [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950]  [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950]  [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff810993bc>] ? __call_rcu+0x110/0x118
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff810993bc>] ? __call_rcu+0x110/0x118
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff81051399>] ? release_task+0x3e9/0x3fa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff81154942>] ? cap_task_wait+0x0/0x3
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff81051f61>] ? do_wait+0x1dd/0x21b
[643359.899950]  [<ffffffff8105203f>] ? sys_wait4+0xa0/0xb0
[643359.899950]  [<ffffffff810fd5c6>] ? sys_poll+0x47/0xb2
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] udevd         S ffff88002f163f90     0   205    165 0x00000000
[643359.899950]  ffff88002f163f90 0000000000000282 0000000000000000 0000000000000000
[643359.899950]  0000000000000000 0000000000000000 000000000000f9e0 ffff880002703fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002f164db0 ffff88002f1650a8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8130d6d4>] ? schedule_hrtimeout_range+0x4a/0x112
[643359.899950]  [<ffffffff811549b7>] ? cap_socket_getpeersec_dgram+0x0/0x6
[643359.899950]  [<ffffffff8127bec0>] ? netlink_sendmsg+0x162/0x255
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8125a6e8>] ? datagram_poll+0x0/0xc2
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8125a6e8>] ? datagram_poll+0x0/0xc2
[643359.899950]  [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950]  [<ffffffff810fd3a0>] ? do_sys_poll+0x2f8/0x391
[643359.899950]  [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950]  [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff81154984>] ? cap_unix_may_send+0x0/0x3
[643359.899950]  [<ffffffff8125462f>] ? sock_def_readable+0x10/0x62
[643359.899950]  [<ffffffff812c352e>] ? unix_dgram_sendmsg+0x3fd/0x444
[643359.899950]  [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff810ba77e>] ? free_hot_cold_page+0x1a2/0x1af
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950]  [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
[643359.899950]  [<ffffffff8100e585>] ? xen_set_pte_at+0xa2/0xc2
[643359.899950]  [<ffffffff8125163d>] ? sockfd_lookup_light+0x1a/0x51
[643359.899950]  [<ffffffff8125285e>] ? sys_sendto+0xf5/0x138
[643359.899950]  [<ffffffff8130e002>] ? _spin_lock_irq+0x7/0x22
[643359.899950]  [<ffffffff810fd4eb>] ? sys_ppoll+0xb2/0x146
[643359.899950]  [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] udevd         S ffff88002f165bd0     0   206    165 0x00000000
[643359.899950]  ffff88002f165bd0 0000000000000286 0000000000000000 0000000000000000
[643359.899950]  0000000000000000 0000000000000000 000000000000f9e0 ffff8800025d3fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002f1654c0 ffff88002f1657b8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8130d6d4>] ? schedule_hrtimeout_range+0x4a/0x112
[643359.899950]  [<ffffffff811549b7>] ? cap_socket_getpeersec_dgram+0x0/0x6
[643359.899950]  [<ffffffff81259304>] ? __alloc_skb+0x3e/0x15a
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8125a6e8>] ? datagram_poll+0x0/0xc2
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8125a6e8>] ? datagram_poll+0x0/0xc2
[643359.899950]  [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950]  [<ffffffff810fd3a0>] ? do_sys_poll+0x2f8/0x391
[643359.899950]  [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950]  [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff81154984>] ? cap_unix_may_send+0x0/0x3
[643359.899950]  [<ffffffff8125462f>] ? sock_def_readable+0x10/0x62
[643359.899950]  [<ffffffff812c352e>] ? unix_dgram_sendmsg+0x3fd/0x444
[643359.899950]  [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff810ba77e>] ? free_hot_cold_page+0x1a2/0x1af
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950]  [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
[643359.899950]  [<ffffffff8100e585>] ? xen_set_pte_at+0xa2/0xc2
[643359.899950]  [<ffffffff8125163d>] ? sockfd_lookup_light+0x1a/0x51
[643359.899950]  [<ffffffff8125285e>] ? sys_sendto+0xf5/0x138
[643359.899950]  [<ffffffff8130e002>] ? _spin_lock_irq+0x7/0x22
[643359.899950]  [<ffffffff810fd4eb>] ? sys_ppoll+0xb2/0x146
[643359.899950]  [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] flush-202:0   S 0000000000000000     0   418      2 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000246 0000000000000000 0000000000000180
[643359.899950]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002f675fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002ea08710 ffff88002ea08a08
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.899950]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.899950]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.899950]  [<ffffffff8110a6ea>] ? bdi_writeback_task+0x7f/0xaa
[643359.899950]  [<ffffffff810ca2fc>] ? bdi_start_fn+0x0/0xd0
[643359.899950]  [<ffffffff810ca36c>] ? bdi_start_fn+0x70/0xd0
[643359.899950]  [<ffffffff810ca2fc>] ? bdi_start_fn+0x0/0xd0
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] rpciod/0      S ffff88002ea0dbd0     0   512      2 0x00000000
[643359.899950]  ffff88002ea0f100 0000000000000246 ffff88000319cf20 0000000000000000
[643359.899950]  0000000000000000 ffffffff8106609b 000000000000f9e0 ffff88002ea33fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002ea0dbd0 ffff88002ea0dec8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8106609b>] ? autoremove_wake_function+0x9/0x2e
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffffa00dc59c>] ? rpc_async_schedule+0x0/0xc [sunrpc]
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] rpciod/1      S 0000000000000000     0   513      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000000000000000 ffffffff81054915
[643359.899950]  ffff8800026a0180 ffffffffa00dc203 000000000000f9e0 ffff88002ea35fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002ea0b170 ffff88002ea0b468
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[643359.899950]  [<ffffffffa00dc203>] ? rpc_wake_up_next+0x181/0x188 [sunrpc]
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffffa00dc59c>] ? rpc_async_schedule+0x0/0xc [sunrpc]
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kslowd000     S 0000000000000000     0   514      2 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000246 0000000000000000 ffff8800031947e8
[643359.899950]  0000000000000001 ffff88002ea36000 000000000000f9e0 ffff88002ea37fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002ea0b880 ffff88002ea0bb78
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff810af145>] ? slow_work_thread+0x140/0x2c6
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810af005>] ? slow_work_thread+0x0/0x2c6
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kslowd001     S 0000000000000000     0   515      2 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000000000000000 ffff8800031b27e8
[643359.899950]  0000000000000001 ffff88002e7e0000 000000000000f9e0 ffff88002e7e1fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002ea0f810 ffff88002ea0fb08
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff810af145>] ? slow_work_thread+0x140/0x2c6
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810af005>] ? slow_work_thread+0x0/0x2c6
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] nfsiod        S 0000000000000000     0   516      2 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000246 0000000000000000 ffffffff8100e635
[643359.899950]  ffffffff8100ecf2 ffff88002b89d300 000000000000f9e0 ffff88002df7ffd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002ea0f100 ffff88002ea0f3f8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff810b6e17>] ? mempool_free_slab+0x0/0xe
[643359.899950]  [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950]  [<ffffffffa00db998>] ? rpc_async_release+0x0/0xc [sunrpc]
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] syslogd       S 0000000000000000     0   776      1 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000286 0000000000000000 ffff8800031acd10
[643359.899950]  ffff880003194780 ffff88002ea2c000 000000000000f9e0 ffff88002ea2dfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002ea0bf90 ffff88002ea0c288
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8109b022>] ? delayacct_end+0x74/0x7f
[643359.899950]  [<ffffffff81065fb3>] ? bit_waitqueue+0x10/0xa0
[643359.899950]  [<ffffffff8130d6d4>] ? schedule_hrtimeout_range+0x4a/0x112
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff812c35be>] ? unix_dgram_poll+0x0/0x139
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950]  [<ffffffff810fdb9f>] ? do_select+0x528/0x57a
[643359.899950]  [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950]  [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950]  [<ffffffff810b620e>] ? generic_file_buffered_write+0x1f5/0x278
[643359.899950]  [<ffffffff810b6642>] ? __generic_file_aio_write+0x25f/0x293
[643359.899950]  [<ffffffff8130d666>] ? mutex_lock+0xd/0x31
[643359.899950]  [<ffffffff810f6f6a>] ? pipe_write+0x48f/0x4da
[643359.899950]  [<ffffffff810b66f9>] ? generic_file_aio_write+0x83/0x9f
[643359.899950]  [<ffffffff810f6adb>] ? pipe_write+0x0/0x4da
[643359.899950]  [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950]  [<ffffffff810fdd75>] ? core_sys_select+0x184/0x21e
[643359.899950]  [<ffffffff810efd65>] ? rw_copy_check_uvector+0x6d/0xe4
[643359.899950]  [<ffffffff810f054f>] ? do_readv_writev+0xb2/0x115
[643359.899950]  [<ffffffff810fe042>] ? sys_select+0x92/0xbb
[643359.899950]  [<ffffffff8105c978>] ? sys_rt_sigprocmask+0x4c/0xc4
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] bacula-fd     S 0000000000000000     0   859      1 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000286 ffffffff8100e635 ffffffff8100ecf2
[643359.899950]  ffffffff8118c035 ffff880002101000 000000000000f9e0 ffff8800021a1fd8
[643359.899950]  0000000000015780 0000000000015780 ffff880002658000 ffff8800026582f8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8118c035>] ? cfq_set_request+0x0/0x44d
[643359.899950]  [<ffffffff8117864c>] ? elv_rb_del+0x2f/0x47
[643359.899950]  [<ffffffffa000a43b>] ? do_blkif_request+0x0/0x374 [xen_blkfront]
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffffa000a43b>] ? do_blkif_request+0x0/0x374 [xen_blkfront]
[643359.899950]  [<ffffffff8130d6d4>] ? schedule_hrtimeout_range+0x4a/0x112
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8128d2cf>] ? tcp_poll+0x0/0x14a
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8128d2cf>] ? tcp_poll+0x0/0x14a
[643359.899950]  [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950]  [<ffffffff810fdb9f>] ? do_select+0x528/0x57a
[643359.899950]  [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950]  [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950]  [<ffffffff81041497>] ? dequeue_entity+0x18/0x135
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff810baed0>] ? get_page_from_freelist+0x4bc/0x760
[643359.899950]  [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950]  [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8119934b>] ? __bitmap_weight+0x3a/0x7e
[643359.899950]  [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950]  [<ffffffff810fdd75>] ? core_sys_select+0x184/0x21e
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643359.899950]  [<ffffffff8104e864>] ? do_fork+0x254/0x31e
[643359.899950]  [<ffffffff810fe042>] ? sys_select+0x92/0xbb
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] bacula-fd     S 0000000000000000     0   861      1 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000286 0000000000000000 00007fa60c740e70
[643359.899950]  0000000085145b7e 0000000000000001 000000000000f9e0 ffff8800025f7fd8
[643359.899950]  0000000000015780 0000000000015780 ffff880002658e20 ffff880002659118
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81068f88>] ? __hrtimer_start_range_ns+0x22f/0x242
[643359.899950]  [<ffffffff81072a43>] ? futex_wait_queue_me+0xa4/0xbd
[643359.899950]  [<ffffffff81072ba1>] ? futex_wait+0x145/0x291
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff81068587>] ? hrtimer_wakeup+0x0/0x22
[643359.899950]  [<ffffffff81072a19>] ? futex_wait_queue_me+0x7a/0xbd
[643359.899950]  [<ffffffff81074799>] ? do_futex+0x9d/0x995
[643359.899950]  [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3
[643359.899950]  [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643359.899950]  [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950]  [<ffffffff810751a4>] ? sys_futex+0x113/0x131
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] cron          S 0000000000000000     0   886      1 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000282 0000000000000000 0000000000000008
[643359.899950]  00000000000007dd 0000000000000001 000000000000f9e0 ffff88002dfd5fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002ea09530 ffff88002ea09828
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81068f88>] ? __hrtimer_start_range_ns+0x22f/0x242
[643359.899950]  [<ffffffff8130d80a>] ? do_nanosleep+0x65/0xa3
[643359.899950]  [<ffffffff81069040>] ? hrtimer_nanosleep+0x8b/0x10e
[643359.899950]  [<ffffffff81068587>] ? hrtimer_wakeup+0x0/0x22
[643359.899950]  [<ffffffff8130d7ef>] ? do_nanosleep+0x4a/0xa3
[643359.899950]  [<ffffffff8106910f>] ? sys_nanosleep+0x4c/0x62
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] klogd         S ffff88002ea0bf90     0  1223      1 0x00000000
[643359.899950]  ffff88002ea0bf90 0000000000000286 ffff88000203dd80 0000000000000000
[643359.899950]  ffff880000000000 ffff88000203ded8 000000000000f9e0 ffff88000203dfd8
[643359.899950]  0000000000015780 0000000000015780 ffff880002569c40 ffff880002569f38
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff81152d27>] ? cap_capable+0x0/0x1c
[643359.899950]  [<ffffffff8104ffa1>] ? do_syslog+0x102/0x394
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8113aa1d>] ? kmsg_read+0x3a/0x45
[643359.899950]  [<ffffffff8113251d>] ? proc_reg_read+0x6f/0x8a
[643359.899950]  [<ffffffff810f0b24>] ? vfs_read+0xa6/0xff
[643359.899950]  [<ffffffff810f0c39>] ? sys_read+0x45/0x6e
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] vsftpd        S 0000000000000000     0  1248      1 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000286 0000000000000000 ffff88002f11bd78
[643359.899950]  0000000000000040 00000000000200da 000000000000f9e0 ffff88002f11bfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002ea09c40 ffff88002ea09f38
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8130e129>] ? _spin_lock_bh+0x9/0x25
[643359.899950]  [<ffffffff8128cc86>] ? inet_csk_accept+0xce/0x1e9
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff81118afc>] ? inotify_d_instantiate+0x12/0x39
[643359.899950]  [<ffffffff812ab101>] ? inet_accept+0x25/0xb1
[643359.899950]  [<ffffffff812530cc>] ? sys_accept4+0x106/0x1d7
[643359.899950]  [<ffffffff8104e864>] ? do_fork+0x254/0x31e
[643359.899950]  [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] sshd          S ffff88000256aa60     0  1281      1 0x00000000
[643359.899950]  ffff88000256aa60 0000000000000286 ffffffff8100e635 ffffffff8100ecf2
[643359.899950]  ffffffff8118c035 0000160000000000 000000000000f9e0 ffff88000209bfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88000256b170 ffff88000256b468
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8118c035>] ? cfq_set_request+0x0/0x44d
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8118c418>] ? cfq_set_request+0x3e3/0x44d
[643359.899950]  [<ffffffff8130d6d4>] ? schedule_hrtimeout_range+0x4a/0x112
[643359.899950]  [<ffffffff8128d2cf>] ? tcp_poll+0x0/0x14a
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8128d2cf>] ? tcp_poll+0x0/0x14a
[643359.899950]  [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950]  [<ffffffff810fdb9f>] ? do_select+0x528/0x57a
[643359.899950]  [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950]  [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950]  [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950]  [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950]  [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff8109b022>] ? delayacct_end+0x74/0x7f
[643359.899950]  [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643359.899950]  [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[643359.899950]  [<ffffffff810b54c6>] ? __lock_page+0x5d/0x63
[643359.899950]  [<ffffffff810660c0>] ? wake_bit_function+0x0/0x23
[643359.899950]  [<ffffffff810b54d5>] ? lock_page+0x9/0x1f
[643359.899950]  [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950]  [<ffffffff810fdd75>] ? core_sys_select+0x184/0x21e
[643359.899950]  [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff810fe042>] ? sys_select+0x92/0xbb
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] munin-node    S 0000000000000000     0  1439      1 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000282 0000000000000000 0000000000000000
[643359.899950]  ffff88002f9b6c20 0000000000000001 000000000000f9e0 ffff880002575fd8
[643359.899950]  0000000000015780 0000000000015780 ffff880002568000 ffff8800025682f8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81068f88>] ? __hrtimer_start_range_ns+0x22f/0x242
[643359.899950]  [<ffffffff81068d06>] ? hrtimer_try_to_cancel+0x3a/0x43
[643359.899950]  [<ffffffff8130d757>] ? schedule_hrtimeout_range+0xcd/0x112
[643359.899950]  [<ffffffff81068587>] ? hrtimer_wakeup+0x0/0x22
[643359.899950]  [<ffffffff8130d739>] ? schedule_hrtimeout_range+0xaf/0x112
[643359.899950]  [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950]  [<ffffffff810fdb9f>] ? do_select+0x528/0x57a
[643359.899950]  [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950]  [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950]  [<ffffffff810492e7>] ? finish_task_switch+0x44/0xaf
[643359.899950]  [<ffffffff8130cd30>] ? thread_return+0x79/0xe0
[643359.899950]  [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff8109b022>] ? delayacct_end+0x74/0x7f
[643359.899950]  [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643359.899950]  [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[643359.899950]  [<ffffffff810b54c6>] ? __lock_page+0x5d/0x63
[643359.899950]  [<ffffffff810660c0>] ? wake_bit_function+0x0/0x23
[643359.899950]  [<ffffffff810b54d5>] ? lock_page+0x9/0x1f
[643359.899950]  [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950]  [<ffffffff810fdd75>] ? core_sys_select+0x184/0x21e
[643359.899950]  [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950]  [<ffffffff8106cbaa>] ? timekeeping_get_ns+0xe/0x2e
[643359.899950]  [<ffffffff8106d74d>] ? ktime_get_ts+0x61/0x88
[643359.899950]  [<ffffffff810fe042>] ? sys_select+0x92/0xbb
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] getty         S 0000000000000000     0  1486      1 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000286 0000000000000000 ffffffff8120e775
[643359.899950]  000000000000001c ffffffff8130e0ca 000000000000f9e0 ffff8800027e3fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88000256c6a0 ffff88000256c998
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8120e775>] ? set_termios+0x3c9/0x3fb
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff811548e9>] ? cap_file_permission+0x0/0x3
[643359.899950]  [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950]  [<ffffffff811548e9>] ? cap_file_permission+0x0/0x3
[643359.899950]  [<ffffffff8130e002>] ? _spin_lock_irq+0x7/0x22
[643359.899950]  [<ffffffff8106337f>] ? flush_work+0x66/0x87
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff8105b9ec>] ? try_to_del_timer_sync+0x63/0x6c
[643359.899950]  [<ffffffff8120cbb5>] ? n_tty_read+0x498/0x774
[643359.899950]  [<ffffffff811548e9>] ? cap_file_permission+0x0/0x3
[643359.899950]  [<ffffffff8104b4bc>] ? default_wake_function+0x0/0x9
[643359.899950]  [<ffffffff81208906>] ? tty_read+0x76/0xb0
[643359.899950]  [<ffffffff810f0b24>] ? vfs_read+0xa6/0xff
[643359.899950]  [<ffffffff810f0c39>] ? sys_read+0x45/0x6e
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] apache2       S 0000000000000000     0 26160      1 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000286 0000000000000000 0000000000000000
[643359.899950]  0000000000000000 0000000000000001 000000000000f9e0 ffff8800022d3fd8
[643359.899950]  0000000000015780 0000000000015780 ffff880002048e20 ffff880002049118
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81068f88>] ? __hrtimer_start_range_ns+0x22f/0x242
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8130d757>] ? schedule_hrtimeout_range+0xcd/0x112
[643359.899950]  [<ffffffff81068587>] ? hrtimer_wakeup+0x0/0x22
[643359.899950]  [<ffffffff8130d739>] ? schedule_hrtimeout_range+0xaf/0x112
[643359.899950]  [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950]  [<ffffffff810fdb9f>] ? do_select+0x528/0x57a
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8100c1a7>] ? xen_mc_flush+0x159/0x185
[643359.899950]  [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950]  [<ffffffff810492e7>] ? finish_task_switch+0x44/0xaf
[643359.899950]  [<ffffffff8130cd30>] ? thread_return+0x79/0xe0
[643359.899950]  [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff8109b022>] ? delayacct_end+0x74/0x7f
[643359.899950]  [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643359.899950]  [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[643359.899950]  [<ffffffff810b54c6>] ? __lock_page+0x5d/0x63
[643359.899950]  [<ffffffff810660c0>] ? wake_bit_function+0x0/0x23
[643359.899950]  [<ffffffff810b54d5>] ? lock_page+0x9/0x1f
[643359.899950]  [<ffffffff8100c426>] ? pfn_to_mfn+0xe/0x22
[643359.899950]  [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950]  [<ffffffff810fdd75>] ? core_sys_select+0x184/0x21e
[643359.899950]  [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3
[643359.899950]  [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950]  [<ffffffff8106cbaa>] ? timekeeping_get_ns+0xe/0x2e
[643359.899950]  [<ffffffff8106d74d>] ? ktime_get_ts+0x61/0x88
[643359.899950]  [<ffffffff810fe042>] ? sys_select+0x92/0xbb
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] atop          S 0000000000000000     0 22273      1 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000282 0000000000000000 0000000200000000
[643359.899950]  ffff88002fbad8d0 0000000000000002 000000000000f9e0 ffff880002eb5fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002f167810 ffff88002f167b08
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8105c69c>] ? sys_pause+0x19/0x23
[643359.899950]  [<ffffffff81011b63>] ? sysret_check+0x17/0x5a
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] sshd          D ffff88000265b880     0 17733   1281 0x00000000
[643359.899950]  ffff88000265b880 0000000000000286 0000000000000000 0000000109951433
[643359.899950]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002bfcbfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88000256aa60 ffff88000256ad58
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.899950]  [<ffffffff810492e7>] ? finish_task_switch+0x44/0xaf
[643359.899950]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.899950]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.899950]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643359.899950]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810c17c0>] ? try_to_free_pages+0x2e8/0x38e
[643359.899950]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643359.899950]  [<ffffffff810c97cd>] ? congestion_wait+0x74/0x80
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643359.899950]  [<ffffffff81254a33>] ? sock_alloc_send_pskb+0xdd/0x2f6
[643359.899950]  [<ffffffff810e7e3a>] ? kmalloc_large_node+0x5d/0x9b
[643359.899950]  [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[643359.899950]  [<ffffffff81254a33>] ? sock_alloc_send_pskb+0xdd/0x2f6
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[643359.899950]  [<ffffffff812c408e>] ? unix_stream_sendmsg+0x13b/0x2c8
[643359.899950]  [<ffffffff812514dc>] ? sock_aio_write+0xb1/0xbc
[643359.899950]  [<ffffffff810effee>] ? do_sync_write+0xce/0x113
[643359.899950]  [<ffffffff8101079c>] ? __switch_to+0x285/0x297
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810492e7>] ? finish_task_switch+0x44/0xaf
[643359.899950]  [<ffffffff8130cd30>] ? thread_return+0x79/0xe0
[643359.899950]  [<ffffffff810f0953>] ? vfs_write+0xbc/0x102
[643359.899950]  [<ffffffff810f0a55>] ? sys_write+0x45/0x6e
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] sshd          ? 0000000000000000     0 17736  17733 0x00100000
[643359.899950]  ffff88002fdd0e20 0000000000000246 0000000000000000 ffffffff8100e635
[643359.899950]  ffffffff8100ecf2 0000000000000009 000000000000f9e0 ffff88002bba7fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88000256e9f0 ffff88000256ece8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8105287a>] ? do_exit+0x6bb/0x6c9
[643359.899950]  [<ffffffff8105d6d5>] ? __dequeue_signal+0xfb/0x124
[643359.899950]  [<ffffffff810528fe>] ? do_group_exit+0x76/0x9d
[643359.899950]  [<ffffffff8105f1eb>] ? get_signal_to_deliver+0x310/0x339
[643359.899950]  [<ffffffff81011037>] ? do_notify_resume+0x87/0x73f
[643359.899950]  [<ffffffff810f0b37>] ? vfs_read+0xb9/0xff
[643359.899950]  [<ffffffff81011e0e>] ? int_signal+0x12/0x17
[643359.899950] apache2       ? ffff88000204f810     0 17941  26160 0x00100000
[643359.899950]  ffff88000204f810 0000000000000246 0000000000000000 ffffffff8100e635
[643359.899950]  ffffffff8100ecf2 0000000000000009 000000000000f9e0 ffff88002bf87fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002f163880 ffff88002f163b78
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8105287a>] ? do_exit+0x6bb/0x6c9
[643359.899950]  [<ffffffff810528fe>] ? do_group_exit+0x76/0x9d
[643359.899950]  [<ffffffff8105f1eb>] ? get_signal_to_deliver+0x310/0x339
[643359.899950]  [<ffffffff81011037>] ? do_notify_resume+0x87/0x73f
[643359.899950]  [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643359.899950]  [<ffffffff8105cbe9>] ? do_sigaction+0x159/0x171
[643359.899950]  [<ffffffff810f0b24>] ? vfs_read+0xa6/0xff
[643359.899950]  [<ffffffff81011e0e>] ? int_signal+0x12/0x17
[643359.899950] apache2       D ffff88002b8fd6e8     0 17946  26160 0x00000004
[643359.899950]  ffff88002fe44db0 0000000000000286 ffffffff810bb9e1 0000000109951433
[643359.899950]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002b8fdfd8
[643359.899950]  0000000000015780 0000000000015780 ffff880002049c40 ffff880002049f38
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff810bb9e1>] ? __set_page_dirty_no_writeback+0x0/0xc
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.899950]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.899950]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.899950]  [<ffffffff810bb9e1>] ? __set_page_dirty_no_writeback+0x0/0xc
[643359.899950]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643359.899950]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950]  [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[643359.899950]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950]  [<ffffffff810c97cd>] ? congestion_wait+0x74/0x80
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[643359.899950]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643359.899950]  [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643359.899950]  [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[643359.899950]  [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[643359.899950]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643359.899950]  [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[643359.899950]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643359.899950]  [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[643359.899950]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643359.899950]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643359.899950]  [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[643359.899950]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643359.899950]  [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643359.899950]  [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[643359.899950]  [<ffffffff8110d00c>] ? __generic_file_splice_read+0x36e/0x3bd
[643359.899950]  [<ffffffff81251f64>] ? kernel_sendmsg+0x32/0x3f
[643359.899950]  [<ffffffff81254790>] ? sock_no_sendpage+0x87/0x94
[643359.899950]  [<ffffffff812909f1>] ? tcp_sendpage+0x4b/0x458
[643359.899950]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643359.899950]  [<ffffffff81250d0d>] ? kernel_sendpage+0x16/0x1f
[643359.899950]  [<ffffffff81250d4b>] ? sock_sendpage+0x35/0x3a
[643359.899950]  [<ffffffff8110b122>] ? pipe_to_sendpage+0x66/0x74
[643359.899950]  [<ffffffff8110b17c>] ? splice_from_pipe_feed+0x4c/0xd4
[643359.899950]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643359.899950]  [<ffffffff8110b405>] ? __splice_from_pipe+0x3e/0x69
[643359.899950]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643359.899950]  [<ffffffff8110b47d>] ? splice_from_pipe+0x4d/0x63
[643359.899950]  [<ffffffff8110c00d>] ? direct_splice_actor+0x1b/0x1e
[643359.899950]  [<ffffffff8110b9c8>] ? splice_direct_to_actor+0xdc/0x188
[643359.899950]  [<ffffffff81041432>] ? set_next_entity+0x34/0x56
[643359.899950]  [<ffffffff8110bff2>] ? direct_splice_actor+0x0/0x1e
[643359.899950]  [<ffffffff8110babe>] ? do_splice_direct+0x4a/0x64
[643359.899950]  [<ffffffff810f0312>] ? do_sendfile+0x12d/0x1a8
[643359.899950]  [<ffffffff810f03d6>] ? sys_sendfile64+0x49/0x89
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] munin-node    S 0000000000000000     0 17965   1439 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000286 0000000000000000 ffff88001cca2124
[643359.899950]  ffff88001cca2138 ffff88002de2c780 000000000000f9e0 ffff880029c61fd8
[643359.899950]  0000000000015780 0000000000015780 ffff88002f1669f0 ffff88002f166ce8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8130e129>] ? _spin_lock_bh+0x9/0x25
[643359.899950]  [<ffffffff8125481f>] ? sk_wait_data+0x82/0xc4
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8128f6df>] ? tcp_recvmsg+0x4c3/0xa9e
[643359.899950]  [<ffffffff8125361a>] ? sock_common_recvmsg+0x30/0x45
[643359.899950]  [<ffffffff81251618>] ? sock_aio_read+0xb9/0xc4
[643359.899950]  [<ffffffff810376da>] ? flush_tlb_page+0x5a/0x7b
[643359.899950]  [<ffffffff8100c2f1>] ? __raw_callee_save_xen_pte_val+0x11/0x1e
[643359.899950]  [<ffffffff810cc4d6>] ? do_wp_page+0x2fc/0x707
[643359.899950]  [<ffffffff810f0101>] ? do_sync_read+0xce/0x113
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff810f0b37>] ? vfs_read+0xb9/0xff
[643359.899950]  [<ffffffff810f0c39>] ? sys_read+0x45/0x6e
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] apache2       S ffff88000265a350     0 18163  26160 0x00000000
[643359.899950]  ffff88000265a350 0000000000000286 ffff88002e7d0188 0000000000000016
[643359.899950]  0000000000000200 ffffffff8100ecdf 000000000000f9e0 ffff88002b8dffd8
[643359.899950]  0000000000015780 0000000000015780 ffff88000256e2e0 ffff88000256e5d8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643359.899950]  [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950]  [<ffffffff81154996>] ? cap_socket_accept+0x0/0x3
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff81154996>] ? cap_socket_accept+0x0/0x3
[643359.899950]  [<ffffffff8125b47b>] ? __skb_recv_datagram+0x1e3/0x24b
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8125b504>] ? receiver_wake_function+0x0/0x14
[643359.899950]  [<ffffffff8125b4ff>] ? skb_recv_datagram+0x1c/0x21
[643359.899950]  [<ffffffff812c4426>] ? unix_accept+0x56/0xfd
[643359.899950]  [<ffffffff812530cc>] ? sys_accept4+0x106/0x1d7
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff810993bc>] ? __call_rcu+0x110/0x118
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff8106a975>] ? commit_creds+0xf5/0xfc
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] vsftpd        S 0000000000000000     0 18168   1248 0x00000000
[643359.899950]  ffff88002fdd0e20 0000000000000282 0000000000000000 0000000000000020
[643359.899950]  0000000100000001 0000000000000000 000000000000f9e0 ffff88002bf09fd8
[643359.899950]  0000000000015780 0000000000015780 ffff880002658710 ffff880002658a08
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950]  [<ffffffff812c3980>] ? unix_stream_recvmsg+0x0/0x5d3
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff812c3980>] ? unix_stream_recvmsg+0x0/0x5d3
[643359.899950]  [<ffffffff812c3c65>] ? unix_stream_recvmsg+0x2e5/0x5d3
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff81251618>] ? sock_aio_read+0xb9/0xc4
[643359.899950]  [<ffffffff810331dc>] ? no_context+0x25/0x1f8
[643359.899950]  [<ffffffff8100c2f1>] ? __raw_callee_save_xen_pte_val+0x11/0x1e
[643359.899950]  [<ffffffff810cc4d6>] ? do_wp_page+0x2fc/0x707
[643359.899950]  [<ffffffff810f0101>] ? do_sync_read+0xce/0x113
[643359.899950]  [<ffffffff81154912>] ? cap_cred_commit+0x0/0x1
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff810993bc>] ? __call_rcu+0x110/0x118
[643359.899950]  [<ffffffff810f0b37>] ? vfs_read+0xb9/0xff
[643359.899950]  [<ffffffff810f0c39>] ? sys_read+0x45/0x6e
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] vsftpd        S ffff88000265b880     0 18173  18168 0x00000000
[643359.899950]  ffff88000265b880 0000000000000286 ffff88002f690000 ffff8800024e5f00
[643359.899950]  ffff88002f69009c ffff8800020c8000 000000000000f9e0 ffff88002becdfd8
[643359.899950]  0000000000015780 0000000000015780 ffff88000265f810 ffff88000265fb08
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[643359.899950]  [<ffffffff8126122a>] ? dev_queue_xmit+0x35b/0x38d
[643359.899950]  [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8130e129>] ? _spin_lock_bh+0x9/0x25
[643359.899950]  [<ffffffff8125481f>] ? sk_wait_data+0x82/0xc4
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8128f6df>] ? tcp_recvmsg+0x4c3/0xa9e
[643359.899950]  [<ffffffff8130e129>] ? _spin_lock_bh+0x9/0x25
[643359.899950]  [<ffffffff8125361a>] ? sock_common_recvmsg+0x30/0x45
[643359.899950]  [<ffffffff81251b69>] ? sock_recvmsg+0xa6/0xbe
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
[643359.899950]  [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3
[643359.899950]  [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8125163d>] ? sockfd_lookup_light+0x1a/0x51
[643359.899950]  [<ffffffff81252dd4>] ? sys_recvfrom+0xba/0x120
[643359.899950]  [<ffffffff81052d1f>] ? do_setitimer+0x1a9/0x1d9
[643359.899950]  [<ffffffff81052e03>] ? alarm_setitimer+0x35/0x63
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] vsftpd        S 0000000000000000     0 18176   1248 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000282 0000000000000000 0000000000000000
[643359.899950]  ffff880002659c40 0000000000000000 000000000000f9e0 ffff88002ac33fd8
[643359.899950]  0000000000015780 0000000000015780 ffff880002659c40 ffff880002659f38
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950]  [<ffffffff812c3980>] ? unix_stream_recvmsg+0x0/0x5d3
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff812c3980>] ? unix_stream_recvmsg+0x0/0x5d3
[643359.899950]  [<ffffffff812c3c65>] ? unix_stream_recvmsg+0x2e5/0x5d3
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff81251618>] ? sock_aio_read+0xb9/0xc4
[643359.899950]  [<ffffffff810331dc>] ? no_context+0x25/0x1f8
[643359.899950]  [<ffffffff8100c2f1>] ? __raw_callee_save_xen_pte_val+0x11/0x1e
[643359.899950]  [<ffffffff810cc4d6>] ? do_wp_page+0x2fc/0x707
[643359.899950]  [<ffffffff810f0101>] ? do_sync_read+0xce/0x113
[643359.899950]  [<ffffffff81154912>] ? cap_cred_commit+0x0/0x1
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950]  [<ffffffff810993bc>] ? __call_rcu+0x110/0x118
[643359.899950]  [<ffffffff810f0b37>] ? vfs_read+0xb9/0xff
[643359.899950]  [<ffffffff810f0c39>] ? sys_read+0x45/0x6e
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] vsftpd        S 0000000000000000     0 18179  18176 0x00000000
[643359.899950]  ffffffff814a11f0 0000000000000282 0000000000000000 ffff880002110000
[643359.899950]  0000000000000000 000000008118ce88 000000000000f9e0 ffff88002c5fffd8
[643359.899950]  0000000000015780 0000000000015780 ffff88000265d4c0 ffff88000265d7b8
[643359.899950] Call Trace:
[643359.899950]  [<ffffffff81180f90>] ? __make_request+0x3e3/0x428
[643359.899950]  [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950]  [<ffffffff8130e129>] ? _spin_lock_bh+0x9/0x25
[643359.899950]  [<ffffffff8125481f>] ? sk_wait_data+0x82/0xc4
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8128f6df>] ? tcp_recvmsg+0x4c3/0xa9e
[643359.899950]  [<ffffffffa00337d1>] ? ext3_get_block+0x0/0xf1 [ext3]
[643359.899950]  [<ffffffff810b57e1>] ? add_to_page_cache_locked+0x98/0xc1
[643359.899950]  [<ffffffff8125361a>] ? sock_common_recvmsg+0x30/0x45
[643359.899950]  [<ffffffff81251b69>] ? sock_recvmsg+0xa6/0xbe
[643359.899950]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950]  [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950]  [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
[643359.899950]  [<ffffffff8100e585>] ? xen_set_pte_at+0xa2/0xc2
[643359.899950]  [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3
[643359.899950]  [<ffffffff8118f7c8>] ? cpumask_any_but+0x28/0x34
[643359.899950]  [<ffffffff8125163d>] ? sockfd_lookup_light+0x1a/0x51
[643359.899950]  [<ffffffff81252dd4>] ? sys_recvfrom+0xba/0x120
[643359.899950]  [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[643359.899950]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] Sched Debug Version: v0.09, 2.6.32-5-xen-amd64 #1
[643359.899950] now at 643361703.344061 msecs
[643359.899950]   .jiffies                                 : 4455732269
[643359.899950]   .sysctl_sched_latency                    : 10.000000
[643359.899950]   .sysctl_sched_min_granularity            : 2.000000
[643359.899950]   .sysctl_sched_wakeup_granularity         : 2.000000
[643359.899950]   .sysctl_sched_child_runs_first           : 0.000000
[643359.899950]   .sysctl_sched_features                   : 32611451
[643359.899950] 
[643359.899950] cpu#0, 2333.472 MHz
[643359.899950]   .nr_running                    : 0
[643359.899950]   .load                          : 0
[643359.899950]   .nr_switches                   : 174783562
[643359.899950]   .nr_load_updates               : 120151717
[643359.899950]   .nr_uninterruptible            : 2
[643359.899950]   .next_balance                  : 4455.732270
[643359.899950]   .curr->pid                     : 0
[643359.899950]   .clock                         : 643359892.013090
[643359.899950]   .cpu_load[0]                   : 0
[643359.899950]   .cpu_load[1]                   : 8
[643359.899950]   .cpu_load[2]                   : 60
[643359.899950]   .cpu_load[3]                   : 142
[643359.899950]   .cpu_load[4]                   : 225
[643359.899950] 
[643359.899950] cfs_rq[0]:/
[643359.899950]   .exec_clock                    : 0.000000
[643359.899950]   .MIN_vruntime                  : 0.000001
[643359.899950]   .min_vruntime                  : 99987640.710363
[643359.899950]   .max_vruntime                  : 0.000001
[643359.899950]   .spread                        : 0.000000
[643359.899950]   .spread0                       : 0.000000
[643359.899950]   .nr_running                    : 0
[643359.899950]   .load                          : 0
[643359.899950]   .nr_spread_over                : 0
[643359.899950]   .shares                        : 0
[643359.899950] 
[643359.899950] rt_rq[0]:
[643359.899950]   .rt_nr_running                 : 0
[643359.899950]   .rt_throttled                  : 0
[643359.899950]   .rt_time                       : 0.031685
[643359.899950]   .rt_runtime                    : 950.000000
[643359.899950] 
[643359.899950] runnable tasks:
[643359.899950]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[643359.899950] ----------------------------------------------------------------------------------------------------------
[643359.899950] 
[643359.899950] cpu#1, 2333.472 MHz
[643359.899950]   .nr_running                    : 0
[643359.899950]   .load                          : 0
[643359.899950]   .nr_switches                   : 33253033
[643359.899950]   .nr_load_updates               : 90763615
[643359.899950]   .nr_uninterruptible            : 2
[643359.899950]   .next_balance                  : 4455.732270
[643359.899950]   .curr->pid                     : 0
[643359.899950]   .clock                         : 643361701.017599
[643359.899950]   .cpu_load[0]                   : 0
[643359.899950]   .cpu_load[1]                   : 0
[643359.899950]   .cpu_load[2]                   : 0
[643359.899950]   .cpu_load[3]                   : 0
[643359.899950]   .cpu_load[4]                   : 0
[643359.899950] 
[643359.899950] cfs_rq[1]:/
[643359.899950]   .exec_clock                    : 0.000000
[643359.899950]   .MIN_vruntime                  : 0.000001
[643359.899950]   .min_vruntime                  : 123663903.078789
[643359.899950]   .max_vruntime                  : 0.000001
[643359.899950]   .spread                        : 0.000000
[643359.899950]   .spread0                       : 23676262.368426
[643359.899950]   .nr_running                    : 0
[643359.899950]   .load                          : 0
[643359.899950]   .nr_spread_over                : 0
[643359.899950]   .shares                        : 0
[643359.899950] 
[643359.899950] rt_rq[1]:
[643359.899950]   .rt_nr_running                 : 0
[643359.899950]   .rt_throttled                  : 0
[643359.899950]   .rt_time                       : 0.019934
[643359.899950]   .rt_runtime                    : 950.000000
[643359.899950] 
[643359.899950] runnable tasks:
[643359.899950]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[643359.899950] ----------------------------------------------------------------------------------------------------------
[643359.899950] 

[643362.975952] SysRq : Show Blocked State
[643362.975968]   task                        PC stack   pid father
[643362.975981] kswapd0       D 0000000000000000     0    30      2 0x00000000
[643362.975994]  ffffffff814a11f0 0000000000000246 0000000000000000 0000000109951742
[643362.976011]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002dfbdfd8
[643362.976027]  0000000000015780 0000000000015780 ffff88002fe44db0 ffff88002fe450a8
[643362.976043] Call Trace:
[643362.976057]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.976068]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.976079]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643362.976091]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643362.976101]  [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643362.976112]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643362.976122]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643362.976131]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643362.976142]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643362.976153]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.976164]  [<ffffffff810c11a8>] ? kswapd+0x5f9/0x686
[643362.976174]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643362.976183]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.976193]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.976202]  [<ffffffff810c0baf>] ? kswapd+0x0/0x686
[643362.976210]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643362.976220]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643362.976229]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643362.976239]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643362.976248]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643362.976259] sshd          D ffff880002049c40     0 17733   1281 0x00000000
[643362.976271]  ffff880002049c40 0000000000000286 ffff88002bfcb668 ffff88002bfcb664
[643362.976287]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002bfcbfd8
[643362.976302]  0000000000015780 0000000000015780 ffff88000256aa60 ffff88000256ad58
[643362.976318] Call Trace:
[643362.976326]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.976336]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.976345]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643362.976355]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643362.976364]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643362.976374]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643362.976384]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643362.976393]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.976403]  [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[643362.976413]  [<ffffffff8100c1a7>] ? xen_mc_flush+0x159/0x185
[643362.976423]  [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[643362.976433]  [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[643362.976444]  [<ffffffff810492e7>] ? finish_task_switch+0x44/0xaf
[643362.976453]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.976463]  [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[643362.976473]  [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[643362.976482]  [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[643362.976492]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643362.976502]  [<ffffffff810c97cd>] ? congestion_wait+0x74/0x80
[643362.976511]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.976521]  [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643362.976532]  [<ffffffff81254a33>] ? sock_alloc_send_pskb+0xdd/0x2f6
[643362.976543]  [<ffffffff810e7e3a>] ? kmalloc_large_node+0x5d/0x9b
[643362.976553]  [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[643362.976562]  [<ffffffff81254a33>] ? sock_alloc_send_pskb+0xdd/0x2f6
[643362.976572]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.976582]  [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[643362.976592]  [<ffffffff812c408e>] ? unix_stream_sendmsg+0x13b/0x2c8
[643362.976603]  [<ffffffff812514dc>] ? sock_aio_write+0xb1/0xbc
[643362.976614]  [<ffffffff810effee>] ? do_sync_write+0xce/0x113
[643362.976623]  [<ffffffff8101079c>] ? __switch_to+0x285/0x297
[643362.976632]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.976642]  [<ffffffff810492e7>] ? finish_task_switch+0x44/0xaf
[643362.976652]  [<ffffffff8130cd30>] ? thread_return+0x79/0xe0
[643362.976662]  [<ffffffff810f0953>] ? vfs_write+0xbc/0x102
[643362.976671]  [<ffffffff810f0a55>] ? sys_write+0x45/0x6e
[643362.976680]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643362.976688] apache2       D ffff880002659c40     0 17946  26160 0x00000004
[643362.976700]  ffff880002659c40 0000000000000286 ffff88002b8fd348 ffff88002b8fd344
[643362.976716]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002b8fdfd8
[643362.976732]  0000000000015780 0000000000015780 ffff880002049c40 ffff880002049f38
[643362.976747] Call Trace:
[643362.976755]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.976765]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.976774]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643362.976784]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643362.976793]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643362.976803]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643362.976813]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643362.976822]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.976832]  [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[643362.976842]  [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[643362.976852]  [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[643362.976862]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643362.976872]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643362.976881]  [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643362.976891]  [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[643362.976900]  [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[643362.976910]  [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[643362.976920]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643362.976930]  [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643362.976940]  [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[643362.976949]  [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[643362.976960]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.976969]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.976979]  [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[643362.976988]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.976998]  [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[643362.977008]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643362.977018]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.977028]  [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[643362.977037]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643362.977047]  [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[643362.977056]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.977066]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.977075]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643362.977084]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.977094]  [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643362.977104]  [<ffffffff81193442>] ? radix_tree_gang_lookup_slot+0x6a/0x8e
[643362.977114]  [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[643362.977123]  [<ffffffff8110d00c>] ? __generic_file_splice_read+0x36e/0x3bd
[643362.977133]  [<ffffffff81251f64>] ? kernel_sendmsg+0x32/0x3f
[643362.977142]  [<ffffffff81254790>] ? sock_no_sendpage+0x87/0x94
[643362.977152]  [<ffffffff812909f1>] ? tcp_sendpage+0x4b/0x458
[643362.977161]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643362.977170]  [<ffffffff81250d0d>] ? kernel_sendpage+0x16/0x1f
[643362.977180]  [<ffffffff81250d4b>] ? sock_sendpage+0x35/0x3a
[643362.977189]  [<ffffffff8110b122>] ? pipe_to_sendpage+0x66/0x74
[643362.977198]  [<ffffffff8110b17c>] ? splice_from_pipe_feed+0x4c/0xd4
[643362.977207]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643362.977216]  [<ffffffff8110b405>] ? __splice_from_pipe+0x3e/0x69
[643362.979942]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643362.979942]  [<ffffffff8110b47d>] ? splice_from_pipe+0x4d/0x63
[643362.979942]  [<ffffffff8110c00d>] ? direct_splice_actor+0x1b/0x1e
[643362.979942]  [<ffffffff8110b9c8>] ? splice_direct_to_actor+0xdc/0x188
[643362.979942]  [<ffffffff81041432>] ? set_next_entity+0x34/0x56
[643362.979942]  [<ffffffff8110bff2>] ? direct_splice_actor+0x0/0x1e
[643362.979942]  [<ffffffff8110babe>] ? do_splice_direct+0x4a/0x64
[643362.979942]  [<ffffffff810f0312>] ? do_sendfile+0x12d/0x1a8
[643362.979942]  [<ffffffff810f03d6>] ? sys_sendfile64+0x49/0x89
[643362.979942]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643362.979942] vsftpd        D 0000000000000000     0 18173  18168 0x00000000
[643362.979942]  ffffffff814a11f0 0000000000000286 0000000000000000 000000010995173d
[643362.979942]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002becdfd8
[643362.979942]  0000000000015780 0000000000015780 ffff88000265f810 ffff88000265fb08
[643362.979942] Call Trace:
[643362.979942]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643362.979942]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643362.979942]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643362.979942]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643362.979942]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643362.979942]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643362.979942]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.979942]  [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[643362.979942]  [<ffffffff8100ec99>] ? xen_irq_enable_direct_end+0x0/0x7
[643362.979942]  [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[643362.979942]  [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[643362.979942]  [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[643362.979942]  [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[643362.979942]  [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[643362.979942]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643362.979942]  [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643362.979942]  [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[643362.979942]  [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[643362.979942]  [<ffffffff810fa746>] ? getname+0x23/0x1a0
[643362.979942]  [<ffffffff810fa746>] ? getname+0x23/0x1a0
[643362.979942]  [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0
[643362.979942]  [<ffffffff810fa746>] ? getname+0x23/0x1a0
[643362.979942]  [<ffffffff810fb1a6>] ? user_path_at+0x20/0x79
[643362.979942]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942]  [<ffffffff810ef186>] ? sys_chdir+0x1b/0x64
[643362.979942]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643362.979942] vsftpd        D 0000000000000000     0 18176   1248 0x00000000
[643362.979942]  ffffffff814a11f0 0000000000000246 0000000000000000 ffffffff8100e635
[643362.979942]  ffffffff8100ecf2 ffff8800020c8000 000000000000f9e0 ffff88002ac33fd8
[643362.979942]  0000000000015780 0000000000015780 ffff880002659c40 ffff880002659f38
[643362.979942] Call Trace:
[643362.979942]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942]  [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643362.979942]  [<ffffffff8130e002>] ? _spin_lock_irq+0x7/0x22
[643362.979942]  [<ffffffff8130e002>] ? _spin_lock_irq+0x7/0x22
[643362.979942]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942]  [<ffffffff8130e002>] ? _spin_lock_irq+0x7/0x22
[643362.979942]  [<ffffffff8130cfa4>] ? wait_for_common+0xde/0x15b
[643362.979942]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942]  [<ffffffff8104b4bc>] ? default_wake_function+0x0/0x9
[643362.979942]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643362.979942]  [<ffffffff8106417b>] ? synchronize_sched+0x4c/0x52
[643362.979942]  [<ffffffff81064181>] ? wakeme_after_rcu+0x0/0xb
[643362.979942]  [<ffffffff8106943e>] ? switch_task_namespaces+0x32/0x43
[643362.979942]  [<ffffffff81052708>] ? do_exit+0x549/0x6c9
[643362.979942]  [<ffffffff810528fe>] ? do_group_exit+0x76/0x9d
[643362.979942]  [<ffffffff81052937>] ? sys_exit_group+0x12/0x16
[643362.979942]  [<ffffffff8130e67a>] ? error_exit+0x2a/0x60
[643362.979942]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643362.979942] vsftpd        D ffff88000256aa60     0 18197  18192 0x00000000
[643362.979942]  ffff88000256aa60 0000000000000282 0000000000000000 0000000109951742
[643362.979942]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002aaf9fd8
[643362.979942]  0000000000015780 0000000000015780 ffff88000265aa60 ffff88000265ad58
[643362.979942] Call Trace:
[643362.979942]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643362.979942]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643362.979942]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643362.979942]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643362.979942]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643362.979942]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643362.979942]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643362.979942]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.979942]  [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[643362.979942]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643362.979942]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942]  [<ffffffffa00bb1ef>] ? tcp_packet+0xb20/0xb65 [nf_conntrack]
[643362.979942]  [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[643362.979942]  [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[643362.979942]  [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[643362.979942]  [<ffffffffa01aface>] ? ipt_do_table+0x5ee/0x620 [ip_tables]
[643362.979942]  [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[643362.979942]  [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[643362.979942]  [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[643362.979942]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643362.979942]  [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643362.979942]  [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[643362.979942]  [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[643362.979942]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.979942]  [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[643362.979942]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.979942]  [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[643362.979942]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.979942]  [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[643362.979942]  [<ffffffff812514dc>] ? sock_aio_write+0xb1/0xbc
[643362.979942]  [<ffffffff8100c426>] ? pfn_to_mfn+0xe/0x22
[643362.979942]  [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
[643362.979942]  [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3
[643362.979942]  [<ffffffff810effee>] ? do_sync_write+0xce/0x113
[643362.979942]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.979942]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942]  [<ffffffff810f0953>] ? vfs_write+0xbc/0x102
[643362.979942]  [<ffffffff810f0a55>] ? sys_write+0x45/0x6e
[643362.979942]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643362.979942] exim4         D 0000000000000000     0 18198  18194 0x00000000
[643362.979942]  ffffffff814a11f0 0000000000000282 0000000000000000 00000000ae9639ab
[643362.979942]  ffffea00003599f8 ffffea00003599f8 000000000000f9e0 ffff88002b463fd8
[643362.979942]  0000000000015780 0000000000015780 ffff88000265cdb0 ffff88000265d0a8
[643362.979942] Call Trace:
[643362.979942]  [<ffffffff8106cbaa>] ? timekeeping_get_ns+0xe/0x2e
[643362.979942]  [<ffffffff810b54f9>] ? sync_page+0x0/0x46
[643362.979942]  [<ffffffff8130ce0a>] ? io_schedule+0x73/0xb7
[643362.979942]  [<ffffffff810b553a>] ? sync_page+0x41/0x46
[643362.979942]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643362.979942]  [<ffffffff8130d21a>] ? __wait_on_bit_lock+0x3f/0x84
[643362.979942]  [<ffffffff810b54c6>] ? __lock_page+0x5d/0x63
[643362.979942]  [<ffffffff810660c0>] ? wake_bit_function+0x0/0x23
[643362.979942]  [<ffffffff810b54d5>] ? lock_page+0x9/0x1f
[643362.979942]  [<ffffffff810b55eb>] ? find_lock_page+0x25/0x45
[643362.979942]  [<ffffffff810b5bfb>] ? filemap_fault+0x1a5/0x2f6
[643362.979942]  [<ffffffff810cbbb6>] ? __do_fault+0x54/0x3c3
[643362.979942]  [<ffffffff811927a9>] ? prio_tree_insert+0x185/0x20d
[643362.979942]  [<ffffffff8105337d>] ? current_fs_time+0x1e/0x24
[643362.979942]  [<ffffffff8100c3a5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[643362.979942]  [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643362.979942]  [<ffffffff810d1513>] ? get_unmapped_area+0xd7/0x139
[643362.979942]  [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[643362.979942]  [<ffffffff8130e445>] ? page_fault+0x25/0x30
[643362.979942] vsftpd        D 0000000000000000     0 18199   1248 0x00000000
[643362.979942]  ffffffff814a11f0 0000000000000282 0000000000000000 ffff88002df9d780
[643362.979942]  0000000000000000 0000000000000000 000000000000f9e0 ffff88002ac41fd8
[643362.979942]  0000000000015780 0000000000015780 ffff88000265b170 ffff88000265b468
[643362.979942] Call Trace:
[643362.979942]  [<ffffffff8106cbaa>] ? timekeeping_get_ns+0xe/0x2e
[643362.979942]  [<ffffffff8110f74e>] ? sync_buffer+0x0/0x40
[643362.979942]  [<ffffffff8130ce0a>] ? io_schedule+0x73/0xb7
[643362.979942]  [<ffffffff8110f789>] ? sync_buffer+0x3b/0x40
[643362.979942]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643362.979942]  [<ffffffff8130d317>] ? __wait_on_bit+0x41/0x70
[643362.979942]  [<ffffffff8110f74e>] ? sync_buffer+0x0/0x40
[643362.979942]  [<ffffffff8130d3b1>] ? out_of_line_wait_on_bit+0x6b/0x77
[643362.979942]  [<ffffffff810660c0>] ? wake_bit_function+0x0/0x23
[643362.979942]  [<ffffffff81111395>] ? __bread+0x4c/0x63
[643362.979942]  [<ffffffffa0032729>] ? ext3_get_branch+0x67/0xdb [ext3]
[643362.979942]  [<ffffffffa0033001>] ? ext3_get_blocks_handle+0x9e/0x86e [ext3]
[643362.979942]  [<ffffffff8118d18a>] ? cfq_dispatch_requests+0x108/0x488
[643362.979942]  [<ffffffffa000a751>] ? do_blkif_request+0x316/0x374 [xen_blkfront]
[643362.979942]  [<ffffffffa0033885>] ? ext3_get_block+0xb4/0xf1 [ext3]
[643362.979942]  [<ffffffff81115f00>] ? do_mpage_readpage+0x161/0x421
[643362.979942]  [<ffffffffa00337d1>] ? ext3_get_block+0x0/0xf1 [ext3]
[643362.979942]  [<ffffffff81193dd5>] ? radix_tree_insert+0x16a/0x1a8
[643362.979942]  [<ffffffffa00337d1>] ? ext3_get_block+0x0/0xf1 [ext3]
[643362.979942]  [<ffffffff810b57e1>] ? add_to_page_cache_locked+0x98/0xc1
[643362.979942]  [<ffffffffa00337d1>] ? ext3_get_block+0x0/0xf1 [ext3]
[643362.979942]  [<ffffffff811162fe>] ? mpage_readpages+0xcc/0x112
[643362.979942]  [<ffffffffa00337d1>] ? ext3_get_block+0x0/0xf1 [ext3]
[643362.979942]  [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643362.979942]  [<ffffffff810bce95>] ? __do_page_cache_readahead+0x11b/0x1b4
[643362.979942]  [<ffffffff810bcf4a>] ? ra_submit+0x1c/0x20
[643362.979942]  [<ffffffff810b5bd3>] ? filemap_fault+0x17d/0x2f6
[643362.979942]  [<ffffffff810cbbb6>] ? __do_fault+0x54/0x3c3
[643362.979942]  [<ffffffff8100b444>] ? xen_write_msr_safe+0x76/0xb1
[643362.979942]  [<ffffffff8100c3a5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[643362.979942]  [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643362.979942]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942]  [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[643362.979942]  [<ffffffff8130e445>] ? page_fault+0x25/0x30
[643362.979942] Sched Debug Version: v0.09, 2.6.32-5-xen-amd64 #1
[643362.979942] now at 643363376.608710 msecs
[643362.979942]   .jiffies                                 : 4455733039
[643362.979942]   .sysctl_sched_latency                    : 10.000000
[643362.979942]   .sysctl_sched_min_granularity            : 2.000000
[643362.979942]   .sysctl_sched_wakeup_granularity         : 2.000000
[643362.979942]   .sysctl_sched_child_runs_first           : 0.000000
[643362.979942]   .sysctl_sched_features                   : 32611451
[643362.979942]
[643362.979942] cpu#0, 2333.472 MHz
[643362.979942]   .nr_running                    : 0
[643362.979942]   .load                          : 0
[643362.979942]   .nr_switches                   : 174787620
[643362.979942]   .nr_load_updates               : 120151955
[643362.979942]   .nr_uninterruptible            : 8
[643362.979942]   .next_balance                  : 4455.733040
[643362.979942]   .curr->pid                     : 0
[643362.979942]   .clock                         : 643362973.080469
[643362.979942]   .cpu_load[0]                   : 0
[643362.979942]   .cpu_load[1]                   : 0
[643362.979942]   .cpu_load[2]                   : 0
[643362.979942]   .cpu_load[3]                   : 0
[643362.979942]   .cpu_load[4]                   : 5
[643362.979942]
[643362.979942] cfs_rq[0]:/
[643362.979942]   .exec_clock                    : 0.000000
[643362.979942]   .MIN_vruntime                  : 0.000001
[643362.979942]   .min_vruntime                  : 99987741.798157
[643362.979942]   .max_vruntime                  : 0.000001
[643362.979942]   .spread                        : 0.000000
[643362.979942]   .spread0                       : 0.000000
[643362.979942]   .nr_running                    : 0
[643362.979942]   .load                          : 0
[643362.979942]   .nr_spread_over                : 0
[643362.979942]   .shares                        : 0
[643362.979942]
[643362.979942] rt_rq[0]:
[643362.979942]   .rt_nr_running                 : 0
[643362.979942]   .rt_throttled                  : 0
[643362.979942]   .rt_time                       : 0.000000
[643362.979942]   .rt_runtime                    : 950.000000
[643362.979942]
[643362.979942] runnable tasks:
[643362.979942]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[643362.979942] ----------------------------------------------------------------------------------------------------------
[643362.979942]
[643362.979942] cpu#1, 2333.472 MHz
[643362.979942]   .nr_running                    : 0
[643362.979942]   .load                          : 0
[643362.979942]   .nr_switches                   : 33253903
[643362.979942]   .nr_load_updates               : 90763886
[643362.979942]   .nr_uninterruptible            : 0
[643362.979942]   .next_balance                  : 4455.733077
[643362.979942]   .curr->pid                     : 0
[643362.979942]   .clock                         : 643363373.011868
[643362.979942]   .cpu_load[0]                   : 0
[643362.979942]   .cpu_load[1]                   : 0
[643362.979942]   .cpu_load[2]                   : 0
[643362.979942]   .cpu_load[3]                   : 0
[643362.979942]   .cpu_load[4]                   : 0
[643362.979942]
[643362.979942] cfs_rq[1]:/
[643362.979942]   .exec_clock                    : 0.000000
[643362.979942]   .MIN_vruntime                  : 0.000001
[643362.979942]   .min_vruntime                  : 123663966.605916
[643362.979942]   .max_vruntime                  : 0.000001
[643362.979942]   .spread                        : 0.000000
[643362.979942]   .spread0                       : 23676224.807759
[643362.979942]   .nr_running                    : 0
[643362.979942]   .load                          : 0
[643362.979942]   .nr_spread_over                : 0
[643362.979942]   .shares                        : 0
[643362.979942]
[643362.979942] rt_rq[1]:
[643362.979942]   .rt_nr_running                 : 0
[643362.979942]   .rt_throttled                  : 0
[643362.979942]   .rt_time                       : 0.000000
[643362.979942]   .rt_runtime                    : 950.000000
[643362.979942]
[643362.979942] runnable tasks:
[643362.979942]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[643362.979942] ----------------------------------------------------------------------------------------------------------
[643362.979942]

Please help. TIA.

-- 
     2. That which causes joy or happiness.

----- End forwarded message -----
----- Forwarded message from Josip Rodin <joy@debbugs.entuzijast.net> -----

Date: Fri, 15 Feb 2013 08:56:20 +0100
From: Josip Rodin <joy@debbugs.entuzijast.net>
To: 519586@bugs.debian.org
Cc: 519586-submitter@bugs.debian.org
Subject: Re: Huge "Slab Unreclaimable" and continually growing

On Tue, Jan 22, 2013 at 10:59:17AM +0100, Josip Rodin wrote:
> I appear to be experiencing a serious problem with a 768 MB RAM Xen domU
> machine running an NFS client - every now and then (for months now), often
> in the middle of the night, it enters some kind of a broken state where a
> few semi-random processes (mainly apache2's and vsftpd's which are told to
> serve files from the NFS mount) start battling it out for the memory, and
> everything including sshd starts invoking the OOM killer, over and over
> again. Nothing seems to halt the downward spiral; manual invocation of the
> OOM killer does nothing of any use. Terminating all processes is the only
> thing that makes it go quiet, but then that's effectively the same as a
> reboot.
> 
> This is the SysRq+M output on the machine once it's been in the broken state
> for a while:
[...]
> active_anon:394 inactive_anon:3197 isolated_anon:0
>  active_file:25 inactive_file:176 isolated_file:32
>  unevictable:2659 dirty:1 writeback:0 unstable:0
>  free:21456 slab_reclaimable:16177 slab_unreclaimable:143165
>  mapped:677 shmem:76 pagetables:455 bounce:0
[...]
> The thing I noticed was the slab_unreclaimable explosion, by a factor
> of 122. That... doesn't sound like something that should be happenning.
> 
> Googling for slab_unreclaimable found me this old bug report about
> slab_unreclaimable domU problems that was mass-closed with the switch to the
> new paravirtops Xen release. Granted, our use case is not Samba like with
> the original reporter, but the pattern of a file server was close enough for
> me to be uncomfortable with it :|

I caught it earlier just now, at:

[950084.590733] active_anon:2805 inactive_anon:11835 isolated_anon:0
[950084.590735]  active_file:76 inactive_file:516 isolated_file:32
[950084.590737]  unevictable:783 dirty:1 writeback:0 unstable:0
[950084.590739]  free:26251 slab_reclaimable:15733 slab_unreclaimable:128868
[950084.590741]  mapped:938 shmem:75 pagetables:651 bounce:0

And snuck in a few slabtops (even some -o invocations were getting killed,
along with my shell and pretty much everything else):

 Active / Total Objects (% used)    : 555753 / 587128 (94.7%)
 Active / Total Slabs (% used)      : 49430 / 49430 (100.0%)
 Active / Total Caches (% used)     : 65 / 76 (85.5%)
 Active / Total Size (% used)       : 546613.78K / 553025.01K (98.8%)
 Minimum / Average / Maximum Object : 0.01K / 0.94K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
 90993  66836  73%    0.19K   4333       21     17332K dentry
 75840  73664  97%    0.12K   2370       32      9480K kmalloc-128
 68096  68092  99%    0.01K    133      512       532K kmalloc-8
 65888  65655  99%    0.25K   4118       16     16472K kmalloc-256
 65820  65778  99%    1.00K   4767       16     76272K kmalloc-1024
 65436  65414  99%    0.63K   5454       12     43632K proc_inode_cache
 65419  65419 100%    4.00K  14179        8    453728K kmalloc-4096
 65390  65390 100%    2.06K  13338       15    426816K net_namespace
  4998   4990  99%    0.08K     98       51       392K sysfs_dir_cache
  4224   2018  47%    0.06K     66       64       264K kmalloc-64
  2288   2107  92%    0.18K    104       22       416K vm_area_struct
  1792   1789  99%    0.02K      7      256        28K kmalloc-16
  1470   1203  81%    0.19K     70       21       280K kmalloc-192
  1300    402  30%    0.79K     65       20      1040K ext3_inode_cache
   896    731  81%    0.03K      7      128        28K anon_vma
   784    532  67%    0.55K     56       14       448K radix_tree_node

A bit later:

 Active / Total Objects (% used)    : 555403 / 586704 (94.7%)
 Active / Total Slabs (% used)      : 49394 / 49394 (100.0%)
 Active / Total Caches (% used)     : 65 / 76 (85.5%)
 Active / Total Size (% used)       : 546552.82K / 552827.43K (98.9%)
 Minimum / Average / Maximum Object : 0.01K / 0.94K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
 90993  66779  73%    0.19K   4333       21     17332K dentry
 75840  73654  97%    0.12K   2370       32      9480K kmalloc-128
 68096  68092  99%    0.01K    133      512       532K kmalloc-8
 65888  65601  99%    0.25K   4118       16     16472K kmalloc-256
 65852  65741  99%    1.00K   4760       16     76160K kmalloc-1024
 65436  65409  99%    0.63K   5454       12     43632K proc_inode_cache
 65428  65428 100%    4.00K  14181        8    453792K kmalloc-4096
 65391  65391 100%    2.06K  13339       15    426848K net_namespace
  4998   4986  99%    0.08K     98       51       392K sysfs_dir_cache
  4224   2017  47%    0.06K     66       64       264K kmalloc-64
  2134   2108  98%    0.18K     97       22       388K vm_area_struct
  1792   1789  99%    0.02K      7      256        28K kmalloc-16
  1449   1078  74%    0.19K     69       21       276K kmalloc-192
  1100    376  34%    0.79K     55       20       880K ext3_inode_cache
   896    639  71%    0.03K      7      128        28K anon_vma
   714    554  77%    0.55K     51       14       408K radix_tree_node

And a bit later still:

[950506.398951] active_anon:2370 inactive_anon:12180 isolated_anon:2
[950506.398953]  active_file:0 inactive_file:261 isolated_file:36
[950506.398955]  unevictable:831 dirty:1 writeback:0 unstable:0
[950506.398957]  free:26418 slab_reclaimable:15749 slab_unreclaimable:129161
[950506.398959]  mapped:743 shmem:75 pagetables:636 bounce:0

 Active / Total Objects (% used)    : 556917 / 587606 (94.8%)
 Active / Total Slabs (% used)      : 49717 / 49717 (100.0%)
 Active / Total Caches (% used)     : 65 / 76 (85.5%)
 Active / Total Size (% used)       : 548047.95K / 554096.57K (98.9%)
 Minimum / Average / Maximum Object : 0.01K / 0.94K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
 90993  66921  73%    0.19K   4333       21     17332K dentry
 75840  73825  97%    0.12K   2370       32      9480K kmalloc-128
 68096  68092  99%    0.01K    133      512       532K kmalloc-8
 65984  65766  99%    0.25K   4124       16     16496K kmalloc-256
 65964  65913  99%    1.00K   4776       16     76416K kmalloc-1024
 65592  65566  99%    0.63K   5467       12     43736K proc_inode_cache
 65584  65584 100%    4.00K  14288        8    457216K kmalloc-4096
 65552  65552 100%    2.06K  13500       15    432000K net_namespace
  4998   4986  99%    0.08K     98       51       392K sysfs_dir_cache
  4224   1976  46%    0.06K     66       64       264K kmalloc-64
  2398   2398 100%    0.18K    109       22       436K vm_area_struct
  1792   1789  99%    0.02K      7      256        28K kmalloc-16
  1428   1112  77%    0.19K     68       21       272K kmalloc-192
  1100    402  36%    0.79K     55       20       880K ext3_inode_cache
   896    648  72%    0.03K      7      128        28K anon_vma
   742    664  89%    0.55K     53       14       424K radix_tree_node

[950532.846913] SysRq : Show Blocked State
[950532.846930]   task                        PC stack   pid father
[950532.846942] kswapd0       D ffff88002f123880     0    30      2 0x00000000
[950532.846955]  ffff88002f123880 0000000000000246 ffff88001ae8cbd0 000000010e28d9ca
[950532.846971]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002dfb1fd8
[950532.846987]  0000000000015780 0000000000015780 ffff88002fe44db0 ffff88002fe450a8
[950532.847003] Call Trace:
[950532.847018]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.847028]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.847037]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.847049]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.847061]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.847071]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.847081]  [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[950532.847091]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.847100]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.847110]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.847121]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.847131]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.847142]  [<ffffffff810c11a8>] ? kswapd+0x5f9/0x686
[950532.847151]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.847160]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.847170]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.847179]  [<ffffffff810c0baf>] ? kswapd+0x0/0x686
[950532.847187]  [<ffffffff81065dc5>] ? kthread+0x79/0x81
[950532.847197]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[950532.847205]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[950532.847214]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[950532.847223]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[950532.847234] vsftpd        D 0000000000000000     0  1325      1 0x00000000
[950532.847245]  ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9ca
[950532.847261]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff880002697fd8
[950532.847276]  0000000000015780 0000000000015780 ffff88002f123880 ffff88002f123b78
[950532.847291] Call Trace:
[950532.847299]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.847308]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.847317]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.847327]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.847336]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.848079]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.848088]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.848098]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.848108]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.848117]  [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.848127]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.848137]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.848146]  [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.848156]  [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.848165]  [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.848175]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.848187]  [<ffffffff81154851>] ? cap_sb_alloc_security+0x0/0x3
[950532.848197]  [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.848207]  [<ffffffff810e8769>] ? kmem_cache_alloc+0x8c/0xf0
[950532.848216]  [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.848224]  [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.848235]  [<ffffffff8108bc1e>] ? copy_pid_ns+0x5f/0x2a5
[950532.848244]  [<ffffffff8108bc1e>] ? copy_pid_ns+0x5f/0x2a5
[950532.848253]  [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0
[950532.848262]  [<ffffffff8108bc1e>] ? copy_pid_ns+0x5f/0x2a5
[950532.848271]  [<ffffffff810692fa>] ? create_new_namespaces+0xc2/0x15e
[950532.848281]  [<ffffffff8106954e>] ? copy_namespaces+0x61/0x97
[950532.848291]  [<ffffffff8104df25>] ? copy_process+0xa7a/0x1165
[950532.848300]  [<ffffffff8104e767>] ? do_fork+0x157/0x31e
[950532.848309]  [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[950532.848318]  [<ffffffff81011e63>] ? stub_clone+0x13/0x20
[950532.848327]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.848336] vsftpd        D 0000000000000000     0 28790  28778 0x00000000
[950532.848348]  ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9d4
[950532.848363]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff8800279affd8
[950532.848379]  0000000000015780 0000000000015780 ffff88002f15c6a0 ffff88002f15c998
[950532.848395] Call Trace:
[950532.848402]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.848412]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.848421]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.848431]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.848441]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.848450]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.848460]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.848470]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.848479]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.848488]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.848498]  [<ffffffff810bfd54>] ? shrink_list+0xf8/0x767
[950532.848509]  [<ffffffff8119934b>] ? __bitmap_weight+0x3a/0x7e
[950532.848519]  [<ffffffff8118f7fe>] ? cpumask_next_and+0x2a/0x3a
[950532.848529]  [<ffffffff81045e7e>] ? find_busiest_group+0x420/0xa4e
[950532.848539]  [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.848548]  [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.848557]  [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20
[950532.848567]  [<ffffffff8130e67a>] ? error_exit+0x2a/0x60
[950532.848575]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[950532.848585]  [<ffffffff810bc5c9>] ? throttle_vm_writeout+0x30/0x8d
[950532.848595]  [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
[950532.848604]  [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.848614]  [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.848623]  [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.848633]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.848643]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.848653]  [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.848662]  [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.848671]  [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.848681]  [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.848690]  [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.848699]  [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0
[950532.848708]  [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.848717]  [<ffffffff810fb1a6>] ? user_path_at+0x20/0x79
[950532.848726]  [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[950532.848735]  [<ffffffff810ef186>] ? sys_chdir+0x1b/0x64
[950532.848744]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.848753] apache2       D 0000000000000000     0 30698  18611 0x00000000
[950532.848765]  ffff88002fdd0e20 0000000000000282 0000000000000000 000000010e28d9c8
[950532.848781]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88000222ffd8
[950532.848796]  0000000000015780 0000000000015780 ffff880002699530 ffff880002699828
[950532.848812] Call Trace:
[950532.848819]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.848829]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.848838]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.848848]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.848857]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.848867]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.848876]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.848886]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.848895]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.848905]  [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.848915]  [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.848925]  [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.848935]  [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.848944]  [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.848954]  [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.848964]  [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.848975]  [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.848985]  [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.848995]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.849004]  [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.849014]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.849024]  [<ffffffff8129835d>] ? tcp_established_options+0x2d/0xa9
[950532.849033]  [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.849041]  [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.849051]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.849061]  [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[950532.849070]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.849081]  [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[950532.849090]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.849099]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.849109]  [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[950532.849118]  [<ffffffff81041432>] ? set_next_entity+0x34/0x56
[950532.849127]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.849138]  [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[950532.849146]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.849156]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.849165]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.849174]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.849183]  [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[950532.849192]  [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[950532.849202]  [<ffffffff8110d00c>] ? __generic_file_splice_read+0x36e/0x3bd
[950532.849211]  [<ffffffff81251f64>] ? kernel_sendmsg+0x32/0x3f
[950532.849220]  [<ffffffff81254790>] ? sock_no_sendpage+0x87/0x94
[950532.849230]  [<ffffffff812909f1>] ? tcp_sendpage+0x4b/0x458
[950532.849239]  [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[950532.849248]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903]  [<ffffffff81250d0d>] ? kernel_sendpage+0x16/0x1f
[950532.850903]  [<ffffffff81250d4b>] ? sock_sendpage+0x35/0x3a
[950532.850903]  [<ffffffff8110b122>] ? pipe_to_sendpage+0x66/0x74
[950532.850903]  [<ffffffff8110b17c>] ? splice_from_pipe_feed+0x4c/0xd4
[950532.850903]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903]  [<ffffffff8110b405>] ? __splice_from_pipe+0x3e/0x69
[950532.850903]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903]  [<ffffffff8110b47d>] ? splice_from_pipe+0x4d/0x63
[950532.850903]  [<ffffffff8110c00d>] ? direct_splice_actor+0x1b/0x1e
[950532.850903]  [<ffffffff8110b9c8>] ? splice_direct_to_actor+0xdc/0x188
[950532.850903]  [<ffffffff8110bff2>] ? direct_splice_actor+0x0/0x1e
[950532.850903]  [<ffffffff8110babe>] ? do_splice_direct+0x4a/0x64
[950532.850903]  [<ffffffff810f0312>] ? do_sendfile+0x12d/0x1a8
[950532.850903]  [<ffffffff810f03d6>] ? sys_sendfile64+0x49/0x89
[950532.850903]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.850903] apache2       D 0000000000000000     0 30702  18611 0x00000000
[950532.850903]  ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9c8
[950532.850903]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff880023c6bfd8
[950532.850903]  0000000000015780 0000000000015780 ffff880002503880 ffff880002503b78
[950532.850903] Call Trace:
[950532.850903]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.850903]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.850903]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.850903]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.850903]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.850903]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903]  [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.850903]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903]  [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.850903]  [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.850903]  [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.850903]  [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.850903]  [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.850903]  [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.850903]  [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.850903]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.850903]  [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.850903]  [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
[950532.850903]  [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.850903]  [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.850903]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903]  [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[950532.850903]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903]  [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[950532.850903]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903]  [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[950532.850903]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903]  [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[950532.850903]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.850903]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903]  [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[950532.850903]  [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[950532.850903]  [<ffffffff8110d00c>] ? __generic_file_splice_read+0x36e/0x3bd
[950532.850903]  [<ffffffff81251f64>] ? kernel_sendmsg+0x32/0x3f
[950532.850903]  [<ffffffff81254790>] ? sock_no_sendpage+0x87/0x94
[950532.850903]  [<ffffffff812909f1>] ? tcp_sendpage+0x4b/0x458
[950532.850903]  [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20
[950532.850903]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903]  [<ffffffff81250d0d>] ? kernel_sendpage+0x16/0x1f
[950532.850903]  [<ffffffff81250d4b>] ? sock_sendpage+0x35/0x3a
[950532.850903]  [<ffffffff8110b122>] ? pipe_to_sendpage+0x66/0x74
[950532.850903]  [<ffffffff8110b17c>] ? splice_from_pipe_feed+0x4c/0xd4
[950532.850903]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903]  [<ffffffff8110b405>] ? __splice_from_pipe+0x3e/0x69
[950532.850903]  [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903]  [<ffffffff8110b47d>] ? splice_from_pipe+0x4d/0x63
[950532.850903]  [<ffffffff8110c00d>] ? direct_splice_actor+0x1b/0x1e
[950532.850903]  [<ffffffff8110b9c8>] ? splice_direct_to_actor+0xdc/0x188
[950532.850903]  [<ffffffff8110bff2>] ? direct_splice_actor+0x0/0x1e
[950532.850903]  [<ffffffff8110babe>] ? do_splice_direct+0x4a/0x64
[950532.850903]  [<ffffffff810f0312>] ? do_sendfile+0x12d/0x1a8
[950532.850903]  [<ffffffff810f03d6>] ? sys_sendfile64+0x49/0x89
[950532.850903]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.850903] vsftpd        D 0000000000000000     0 32054  32049 0x00000000
[950532.850903]  ffffffff814a11f0 0000000000000286 0000000000000000 000000010e28d9dd
[950532.850903]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff8800241e3fd8
[950532.850903]  0000000000015780 0000000000015780 ffff880002698710 ffff880002698a08
[950532.850903] Call Trace:
[950532.850903]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.850903]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.850903]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.850903]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.850903]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.850903]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903]  [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.850903]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.850903]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903]  [<ffffffffa00bb1ef>] ? tcp_packet+0xb20/0xb65 [nf_conntrack]
[950532.850903]  [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.850903]  [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.850903]  [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.850903]  [<ffffffffa01aface>] ? ipt_do_table+0x5ee/0x620 [ip_tables]
[950532.850903]  [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.850903]  [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.850903]  [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.850903]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.850903]  [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.850903]  [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.850903]  [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.850903]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903]  [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[950532.850903]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903]  [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[950532.850903]  [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903]  [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[950532.850903]  [<ffffffff812514dc>] ? sock_aio_write+0xb1/0xbc
[950532.850903]  [<ffffffff8100c426>] ? pfn_to_mfn+0xe/0x22
[950532.850903]  [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
[950532.850903]  [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3
[950532.850903]  [<ffffffff810effee>] ? do_sync_write+0xce/0x113
[950532.850903]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903]  [<ffffffff810f0953>] ? vfs_write+0xbc/0x102
[950532.850903]  [<ffffffff810f0a55>] ? sys_write+0x45/0x6e
[950532.850903]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.850903] update-pkg-ft D 0000000000000000     0 32082  32038 0x00000000
[950532.850903]  ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9d2
[950532.850903]  ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff880024f07fd8
[950532.850903]  0000000000015780 0000000000015780 ffff880002798000 ffff8800027982f8
[950532.850903] Call Trace:
[950532.850903]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903]  [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903]  [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.850903]  [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.850903]  [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.850903]  [<ffffffffa0038d2f>] ? bdev_try_to_free_page+0x0/0x64 [ext3]
[950532.850903]  [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.850903]  [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.850903]  [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903]  [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.850903]  [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.850903]  [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.850903]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.850903]  [<ffffffff810993bc>] ? __call_rcu+0x110/0x118
[950532.850903]  [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20
[950532.850903]  [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.850903]  [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.850903]  [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.850903]  [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.850903]  [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.850903]  [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.850903]  [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.850903]  [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.850903]  [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.850903]  [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0
[950532.850903]  [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.850903]  [<ffffffff810fb1a6>] ? user_path_at+0x20/0x79
[950532.850903]  [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[950532.850903]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903]  [<ffffffff810f361a>] ? vfs_fstatat+0x2c/0x57
[950532.850903]  [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903]  [<ffffffff810f370d>] ? sys_newstat+0x11/0x30
[950532.850903]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.850903]  [<ffffffff8100b4d3>] ? xen_mc_issue+0x21/0x23
[950532.850903]  [<ffffffff8101295b>] ? device_not_available+0x1b/0x20
[950532.850903]  [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.850903] Sched Debug Version: v0.09, 2.6.32-5-xen-amd64 #1
[950532.850903] now at 950533247.852699 msecs
[950532.850903]   .jiffies                                 : 4532525608
[950532.850903]   .sysctl_sched_latency                    : 10.000000
[950532.850903]   .sysctl_sched_min_granularity            : 2.000000
[950532.850903]   .sysctl_sched_wakeup_granularity         : 2.000000
[950532.850903]   .sysctl_sched_child_runs_first           : 0.000000
[950532.850903]   .sysctl_sched_features                   : 32611451
[950532.850903]
[950532.850903] cpu#0, 2333.472 MHz
[950532.850903]   .nr_running                    : 0
[950532.850903]   .load                          : 0
[950532.850903]   .nr_switches                   : 238671752
[950532.850903]   .nr_load_updates               : 165114440
[950532.850903]   .nr_uninterruptible            : 6
[950532.850903]   .next_balance                  : 4532.525542
[950532.850903]   .curr->pid                     : 0
[950532.850903]   .clock                         : 950532844.214897
[950532.850903]   .cpu_load[0]                   : 0
[950532.850903]   .cpu_load[1]                   : 192
[950532.850903]   .cpu_load[2]                   : 254
[950532.850903]   .cpu_load[3]                   : 197
[950532.850903]   .cpu_load[4]                   : 130
[950532.850903]
[950532.850903] cfs_rq[0]:/
[950532.850903]   .exec_clock                    : 0.000000
[950532.850903]   .MIN_vruntime                  : 0.000001
[950532.850903]   .min_vruntime                  : 136491473.007815
[950532.850903]   .max_vruntime                  : 0.000001
[950532.850903]   .spread                        : 0.000000
[950532.850903]   .spread0                       : 0.000000
[950532.850903]   .nr_running                    : 0
[950532.850903]   .load                          : 0
[950532.850903]   .nr_spread_over                : 0
[950532.850903]   .shares                        : 0
[950532.850903]
[950532.850903] rt_rq[0]:
[950532.850903]   .rt_nr_running                 : 0
[950532.850903]   .rt_throttled                  : 0
[950532.850903]   .rt_time                       : 0.000000
[950532.850903]   .rt_runtime                    : 950.000000
[950532.850903]
[950532.850903] runnable tasks:
[950532.850903]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[950532.850903] ----------------------------------------------------------------------------------------------------------
[950532.850903]
[950532.850903] cpu#1, 2333.472 MHz
[950532.850903]   .nr_running                    : 3
[950532.850903]   .load                          : 3072
[950532.850903]   .nr_switches                   : 45263996
[950532.850903]   .nr_load_updates               : 121842283
[950532.850903]   .nr_uninterruptible            : 0
[950532.850903]   .next_balance                  : 4532.525625
[950532.850903]   .curr->pid                     : 32083
[950532.850903]   .clock                         : 950533245.009907
[950532.850903]   .cpu_load[0]                   : 3072
[950532.850903]   .cpu_load[1]                   : 3072
[950532.850903]   .cpu_load[2]                   : 3072
[950532.850903]   .cpu_load[3]                   : 3072
[950532.850903]   .cpu_load[4]                   : 3048
[950532.850903]
[950532.850903] cfs_rq[1]:/
[950532.850903]   .exec_clock                    : 0.000000
[950532.850903]   .MIN_vruntime                  : 161709543.323397
[950532.850903]   .min_vruntime                  : 161709548.323397
[950532.850903]   .max_vruntime                  : 161709543.323397
[950532.850903]   .spread                        : 0.000000
[950532.850903]   .spread0                       : 25218075.315582
[950532.850903]   .nr_running                    : 3
[950532.850903]   .load                          : 3072
[950532.850903]   .nr_spread_over                : 0
[950532.850903]   .shares                        : 0
[950532.850903]
[950532.850903] rt_rq[1]:
[950532.850903]   .rt_nr_running                 : 0
[950532.850903]   .rt_throttled                  : 0
[950532.850903]   .rt_time                       : 0.000000
[950532.850903]   .rt_runtime                    : 950.000000
[950532.850903]
[950532.850903] runnable tasks:
[950532.850903]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[950532.850903] ----------------------------------------------------------------------------------------------------------
[950532.850903]         events/1    10 161709543.323397   1041646   120               0               0               0.000000               0.000000               0.000000 /
[950532.850903]          apache2 30698 161709543.323397      3096   120               0               0               0.000000               0.000000               0.000000 /
[950532.850903] R     network.sh 32083 161709832.320247         2   120               0               0               0.000000               0.000000               0.000000 /
[950532.850903]

-- 
     2. That which causes joy or happiness.

----- End forwarded message -----
----- Forwarded message from Josip Rodin <joy@debbugs.entuzijast.net> -----

Date: Sat, 16 Feb 2013 22:37:05 +0100
From: Josip Rodin <joy@debbugs.entuzijast.net>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: 519586@bugs.debian.org
Subject: Re: Bug#519586: Huge "Slab Unreclaimable" and continually growing

On Sat, Feb 16, 2013 at 03:13:06AM +0000, Ben Hutchings wrote:
> On Fri, 2013-02-15 at 08:56 +0100, Josip Rodin wrote:
> > > I appear to be experiencing a serious problem with a 768 MB RAM Xen domU
> > > machine running an NFS client - every now and then (for months now), often
> > > in the middle of the night, it enters some kind of a broken state where a
> > > few semi-random processes (mainly apache2's and vsftpd's which are told to
> > > serve files from the NFS mount)
> [...]
> > I caught it earlier just now, at:
> > 
> > [950084.590733] active_anon:2805 inactive_anon:11835 isolated_anon:0
> > [950084.590735]  active_file:76 inactive_file:516 isolated_file:32
> > [950084.590737]  unevictable:783 dirty:1 writeback:0 unstable:0
> > [950084.590739]  free:26251 slab_reclaimable:15733 slab_unreclaimable:128868
> > [950084.590741]  mapped:938 shmem:75 pagetables:651 bounce:0
> > 
> > And snuck in a few slabtops (even some -o invocations were getting killed,
> > along with my shell and pretty much everything else):
> [...]
> >  65390  65390 100%    2.06K  13338       15    426816K net_namespace
> [...]
> 
> Looks like CVE-2011-2189, for which there was a fix/workaround in:
> 
> vsftpd (2.3.2-3+squeeze2) stable-security; urgency=high
> 
>    * Non-maintainer upload by the Security Team.
>    * Disable network isolation due to a problem with cleaning up network
>      namespaces fast enough in kernels < 2.6.35 (CVE-2011-2189).
>      Thanks Ben Hutchings for the patch!
>    * Fix possible DoS via globa expressions in STAT commands by
>      limiting the matching loop (CVE-2011-0762; Closes: #622741).
> 
>  -- Nico Golde <nion@debian.org>  Wed, 07 Sep 2011 20:39:59 +0000
> 
> Do you have an old version of vsftpd, or perhaps an upstream version
> which doesn't include the workaround?

No, 2.3.2-3+squeeze2 is there, has been since 2012-03-22.

> Anyway, I'm closing the bug report; please don't hijack closed bugs.

Eh? It was not closed for being fixed, it was closed en masse on a
procedural reason that could easily be wrong, and I don't believe I was
hijacking it; you just confirmed that this is a kernel problem above,
so how could this possibly be improper?!

-- 
     2. That which causes joy or happiness.

----- End forwarded message -----

--- End Message ---

--- Begin Message ---

To: 700755-done@bugs.debian.org
Subject: Closing bugs assigned to linux-2.6 package
From: Ben Hutchings <benh@debian.org>
Date: Sun, 29 May 2016 21:24:52 +0100
Message-id: 6b4b485b-8381-4de4-bee3-82a89519ae57@decadent.org.uk

Version: 3.4.1-1~experimental.1+rm

Debian 6.0 Long Term Support has ended, and the 'linux-2.6' source
package will no longer be updated.  This bug was reassigned to the
'linux' source package earlier today, but I am now closing it on the
assumption that it does not affect the kernel versions in newer Debian
releases.

If you can still reproduce this bug in a newer release, please reopen
the bug report and reassign it to 'src:linux' and the affected version
of the package.  You can find the package version for the running
kernel by running:

    uname -v

or the versions of all installed kernel packages by running:

    dpkg -l 'linux-image-[34]*' | grep ^.i

and looking at the third column.

I apologise that we weren't able to provide a specific resolution for
this bug.

Ben.

-- 
Ben Hutchings - Debian developer, member of Linux kernel and LTS teams

--- End Message ---

Reply to:

Prev by Date: Bug#702876: marked as done (linux-image-2.6.32-5-openvz-amd64: OpenVZ 'vzctl start VEID --wait' hangs.)
Next by Date: Bug#760361: marked as done (linux: chown on NFS4 mounts impossible)
Previous by thread: Bug#702876: marked as done (linux-image-2.6.32-5-openvz-amd64: OpenVZ 'vzctl start VEID --wait' hangs.)
Next by thread: Bug#760361: marked as done (linux: chown on NFS4 mounts impossible)
Index(es):
- Date
- Thread