--- Begin Message ---
Package: linux-image-2.6.32-5-xen-amd64
Version: 2.6.32-46
Moved out of #519586 because duplicating e-mails is easy :p
----- Forwarded message from Josip Rodin <joy@debbugs.entuzijast.net> -----
Date: Tue, 22 Jan 2013 10:59:17 +0100
From: Josip Rodin <joy@debbugs.entuzijast.net>
To: 519586@bugs.debian.org
Cc: control@bugs.debian.org, 519586-submitter@bugs.debian.org
Subject: Re: Huge "Slab Unreclaimable" and continually growing
unarchive 519586
reopen 519586
reassign 519586 linux-image-2.6.32-5-xen-amd64
found 519586 2.6.32-46
retitle 519586 huge slab_unreclaimable in Xen domU
thanks
Hi,
I appear to be experiencing a serious problem with a 768 MB RAM Xen domU
machine running an NFS client - every now and then (for months now), often
in the middle of the night, it enters some kind of a broken state where a
few semi-random processes (mainly apache2's and vsftpd's which are told to
serve files from the NFS mount) start battling it out for the memory, and
everything including sshd starts invoking the OOM killer, over and over
again. Nothing seems to halt the downward spiral; manual invocation of the
OOM killer does nothing of any use. Terminating all processes is the only
thing that makes it go quiet, but then that's effectively the same as a
reboot.
This is the SysRq+M output on the machine once it's been in the broken state
for a while:
SysRq : Show Memory
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 35
CPU 1: hi: 186, btch: 31 usd: 0
active_anon:394 inactive_anon:3197 isolated_anon:0
active_file:25 inactive_file:176 isolated_file:32
unevictable:2659 dirty:1 writeback:0 unstable:0
free:21456 slab_reclaimable:16177 slab_unreclaimable:143165
mapped:677 shmem:76 pagetables:455 bounce:0
Node 0 DMA free:3432kB min:1212kB low:1512kB high:1816kB active_anon:0kB inactive_anon:0kB active_file:0kB inac
lowmem_reserve[]: 0 741 741 741
Node 0 DMA32 free:82392kB min:64320kB low:80400kB high:96480kB active_anon:1576kB inactive_anon:12788kB active_
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 42*4kB 42*8kB 45*16kB 15*32kB 5*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 3432kB
Node 0 DMA32: 9560*4kB 4201*8kB 365*16kB 41*32kB 15*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 8
906 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
196608 pages RAM
5319 pages reserved
3301 pages shared
168474 pages non-shared
While this is when it's working normally:
SysRq : Show Memory
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 123
CPU 1: hi: 186, btch: 31 usd: 60
active_anon:13098 inactive_anon:72 isolated_anon:0
active_file:10401 inactive_file:58662 isolated_file:0
unevictable:1902 dirty:8 writeback:0 unstable:0
free:92863 slab_reclaimable:9038 slab_unreclaimable:1166
mapped:2560 shmem:75 pagetables:597 bounce:0
Node 0 DMA free:14524kB min:1212kB low:1512kB high:1816kB active_anon:0kB inactive_anon:0kB active_file:0kB inac
lowmem_reserve[]: 0 741 741 741
Node 0 DMA32 free:356928kB min:64320kB low:80400kB high:96480kB active_anon:52392kB inactive_anon:288kB active_f
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 2*64kB 2*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 3*4096kB = 14524kB
Node 0 DMA32: 32*4kB 145*8kB 84*16kB 40*32kB 22*64kB 15*128kB 12*256kB 21*512kB 10*1024kB 3*2048kB 78*4096kB = 3
69760 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
196608 pages RAM
5319 pages reserved
17310 pages shared
85757 pages non-shared
The thing I noticed was the slab_unreclaimable explosion, by a factor
of 122. That... doesn't sound like something that should be happenning.
Googling for slab_unreclaimable found me this old bug report about
slab_unreclaimable domU problems that was mass-closed with the switch to the
new paravirtops Xen release. Granted, our use case is not Samba like with
the original reporter, but the pattern of a file server was close enough for
me to be uncomfortable with it :|
I'm going to try to run slabtop the next time I catch it in this state,
in order to try to glean some more information.
Here's some more SysRq output from the broken state, hope it's informative:
[643359.895958] SysRq : Show State
[643359.895973] task PC stack pid father
[643359.895981] init D ffff88000256aa60 0 1 0 0x00000000
[643359.895994] ffff88000256aa60 0000000000000282 ffffffff810bb9e1 000000010995142e
[643359.896013] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002fdcbfd8
[643359.896029] 0000000000015780 0000000000015780 ffff88002fdd0000 ffff88002fdd02f8
[643359.896045] Call Trace:
[643359.896061] [<ffffffff810bb9e1>] ? __set_page_dirty_no_writeback+0x0/0xc
[643359.896073] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.896083] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.896092] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.896104] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.896116] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.896126] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.896135] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.896145] [<ffffffff810bb9e1>] ? __set_page_dirty_no_writeback+0x0/0xc
[643359.896155] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643359.896166] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643359.896176] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.896187] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[643359.896196] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.896205] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[643359.896215] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[643359.896225] [<ffffffff81045e7e>] ? find_busiest_group+0x420/0xa4e
[643359.896235] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.896244] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[643359.896254] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[643359.896263] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[643359.896273] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643359.896283] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643359.896293] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[643359.896302] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[643359.896311] [<ffffffff810660c0>] ? wake_bit_function+0x0/0x23
[643359.896321] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[643359.896330] [<ffffffff810b54d5>] ? lock_page+0x9/0x1f
[643359.896339] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[643359.896347] [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0
[643359.897424] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[643359.897433] [<ffffffff810fb1a6>] ? user_path_at+0x20/0x79
[643359.897442] [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643359.897452] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.897461] [<ffffffff810f361a>] ? vfs_fstatat+0x2c/0x57
[643359.897470] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.897480] [<ffffffff810f370d>] ? sys_newstat+0x11/0x30
[643359.897489] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.897498] [<ffffffff8100b4d3>] ? xen_mc_issue+0x21/0x23
[643359.897508] [<ffffffff8101295b>] ? device_not_available+0x1b/0x20
[643359.897518] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.897526] kthreadd S 0000000000000000 0 2 0 0x00000000
[643359.897538] ffffffff814a11f0 0000000000000246 0000000000000000 0000000000000001
[643359.897554] 00000020304045a0 ffffffff81012b42 000000000000f9e0 ffff88002fdd9fd8
[643359.897570] 0000000000015780 0000000000015780 ffff88002fdd0710 ffff88002fdd0a08
[643359.897586] Call Trace:
[643359.897593] [<ffffffff81012b42>] ? kernel_thread+0x82/0xe0
[643359.897603] [<ffffffff81065cd1>] ? kthreadd+0x71/0xec
[643359.897612] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.897621] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.897630] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.897641] [<ffffffff810e178d>] ? do_set_mempolicy+0x128/0x13a
[643359.897650] [<ffffffff810e178d>] ? do_set_mempolicy+0x128/0x13a
[643359.897659] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.897666] migration/0 S 0000000000000000 0 3 2 0x00000000
[643359.897679] ffffffff814a11f0 0000000000000246 0000000000000000 ffff8800031947e8
[643359.897694] ffff88000265b8b8 ffff88000265b8b8 000000000000f9e0 ffff88002fdddfd8
[643359.897710] 0000000000015780 0000000000015780 ffff88002fdd1530 ffff88002fdd1828
[643359.897725] Call Trace:
[643359.897733] [<ffffffff8100f0bc>] ? xen_smp_send_reschedule+0x0/0x7
[643359.897744] [<ffffffff8104c304>] ? migration_thread+0x161/0x236
[643359.897753] [<ffffffff8104c1a3>] ? migration_thread+0x0/0x236
[643359.897762] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.897771] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.897779] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.897788] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.897798] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.897806] ksoftirqd/0 S ffff88000256aa60 0 4 2 0x00000000
[643359.897817] ffff88000256aa60 0000000000000246 0000000000000000 ffff88002fddfe7c
[643359.897834] 000000000000e030 0000000000000246 000000000000f9e0 ffff88002fddffd8
[643359.897849] 0000000000015780 0000000000015780 ffff88002fdd1c40 ffff88002fdd1f38
[643359.897865] Call Trace:
[643359.897873] [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[643359.897884] [<ffffffff810549cd>] ? ksoftirqd+0x46/0xd3
[643359.897893] [<ffffffff81054987>] ? ksoftirqd+0x0/0xd3
[643359.897902] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.897910] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.897919] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.897928] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.897938] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.897945] watchdog/0 S 0000000000000001 0 5 2 0x00000000
[643359.897957] ffff88002fdd2a60 0000000000000246 ffffffff8104a1b9 ffff880003194780
[643359.897973] ffff880003194780 ffffffff8104a1b9 000000000000f9e0 ffff88002fde9fd8
[643359.897989] 0000000000015780 0000000000015780 ffff88002fdd2350 ffff88002fdd2648
[643359.898005] Call Trace:
[643359.898013] [<ffffffff8104a1b9>] ? switched_to_rt+0x0/0x58
[643359.898022] [<ffffffff8104a1b9>] ? switched_to_rt+0x0/0x58
[643359.898032] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.898043] [<ffffffff8106a53b>] ? cpu_clock+0x28/0x30
[643359.898053] [<ffffffff810954c7>] ? watchdog+0x0/0x75
[643359.898061] [<ffffffff8109550b>] ? watchdog+0x44/0x75
[643359.898070] [<ffffffff810954c7>] ? watchdog+0x0/0x75
[643359.898078] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.898087] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.898096] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.898105] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.898114] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.898121] migration/1 S 0000000000000000 0 6 2 0x00000000
[643359.898133] ffff88002fdd0e20 0000000000000246 0000000000000000 ffff8800031b27e8
[643359.898149] ffff88000265f138 ffff88000265f138 000000000000f9e0 ffff88002fdebfd8
[643359.898165] 0000000000015780 0000000000015780 ffff88002fdd2a60 ffff88002fdd2d58
[643359.898181] Call Trace:
[643359.898189] [<ffffffff8100f0bc>] ? xen_smp_send_reschedule+0x0/0x7
[643359.898198] [<ffffffff8104c1a3>] ? migration_thread+0x0/0x236
[643359.898207] [<ffffffff8104c304>] ? migration_thread+0x161/0x236
[643359.898217] [<ffffffff8104c1a3>] ? migration_thread+0x0/0x236
[643359.898226] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.898235] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.898243] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.898252] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.898262] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.898269] ksoftirqd/1 S 0000000000000000 0 7 2 0x00000000
[643359.898281] ffff88002fdd0e20 0000000000000246 0000000000000000 ffff88002fdf1e7c
[643359.898297] 000000000000e030 0000000000000246 000000000000f9e0 ffff88002fdf1fd8
[643359.898312] 0000000000015780 0000000000015780 ffff88002fdd3170 ffff88002fdd3468
[643359.898328] Call Trace:
[643359.898336] [<ffffffff810549cd>] ? ksoftirqd+0x46/0xd3
[643359.898345] [<ffffffff81054987>] ? ksoftirqd+0x0/0xd3
[643359.898353] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.898362] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.898371] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.898380] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.898389] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.898396] watchdog/1 S 0000000000000000 0 8 2 0x00000000
[643359.898408] ffff88002fdd0e20 0000000000000246 ffffffff8104a1b9 ffff8800031b2780
[643359.898424] ffff8800031b2780 ffffffff8104a1b9 000000000000f9e0 ffff88002fdf3fd8
[643359.898439] 0000000000015780 0000000000015780 ffff88002fdd3880 ffff88002fdd3b78
[643359.898455] Call Trace:
[643359.898462] [<ffffffff8104a1b9>] ? switched_to_rt+0x0/0x58
[643359.898472] [<ffffffff8104a1b9>] ? switched_to_rt+0x0/0x58
[643359.898481] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.898491] [<ffffffff8106a53b>] ? cpu_clock+0x28/0x30
[643359.898500] [<ffffffff810954c7>] ? watchdog+0x0/0x75
[643359.898508] [<ffffffff8109550b>] ? watchdog+0x44/0x75
[643359.898517] [<ffffffff810954c7>] ? watchdog+0x0/0x75
[643359.898525] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.898534] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.898542] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.898551] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.898560] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.898567] events/0 S ffff88000256aa60 0 9 2 0x00000000
[643359.898579] ffff88000256aa60 0000000000000246 00000000000000fa 00000000000000f9
[643359.898595] 0000000109950214 0000000109950214 000000000000f9e0 ffff88002fdfdfd8
[643359.898610] 0000000000015780 0000000000015780 ffff88002fdd3f90 ffff88002fdd4288
[643359.898626] Call Trace:
[643359.898633] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.898643] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.898652] [<ffffffff810c93a8>] ? vmstat_update+0x0/0x39
[643359.898661] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.898671] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.898680] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.898688] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] events/1 S 0000000000000000 0 10 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000000000000000 00000000000000f9
[643359.899950] 0000000109950211 0000000109950211 000000000000f9e0 ffff88002fdfffd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fdd46a0 ffff88002fdd4998
[643359.899950] Call Trace:
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff810c93a8>] ? vmstat_update+0x0/0x39
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] cpuset S ffff8800031acc40 0 11 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 ffff88002fe01e00 ffff88002fe01dfc
[643359.899950] 0000000000000000 0000000000000000 000000000000f9e0 ffff88002fe01fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fdd4db0 ffff88002fdd50a8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] khelper S 0000000000000000 0 12 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000000000000000 ffff88002fdd54c0
[643359.899950] ffff88002e7ce248 ffff88002e7ce240 000000000000f9e0 ffff88002fe03fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fdd54c0 ffff88002fdd57b8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff81061251>] ? __call_usermodehelper+0x0/0x69
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] netns S 0000000000000000 0 13 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000000000000000 0000000000000000
[643359.899950] 0000000000000000 0000000000000000 000000000000f9e0 ffff88002fe29fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fdd5bd0 ffff88002fdd5ec8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] async/mgr S ffff8800031acc40 0 14 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 ffff88002fe2be50 ffff88002fe2be4c
[643359.899950] 0000000000000001 ffff8800031b2780 000000000000f9e0 ffff88002fe2bfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fdd62e0 ffff88002fdd65d8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8106afbe>] ? async_manager_thread+0xae/0xd3
[643359.899950] [<ffffffff8104b4bc>] ? default_wake_function+0x0/0x9
[643359.899950] [<ffffffff8106af10>] ? async_manager_thread+0x0/0xd3
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] pm S 0000000000000001 0 15 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000000000000000 0000000000000000
[643359.899950] 0000000000000000 0000000000000000 000000000000f9e0 ffff88002fe2dfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fdd69f0 ffff88002fdd6ce8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] xenwatch S ffff88002fdd7100 0 16 2 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000246 ffffffff8100ecf2 0000000000000001
[643359.899950] 0000000000000010 ffffffff814964f0 000000000000f9e0 ffff88002fe2ffd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fdd7100 ffff88002fdd73f8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffffa0000a18>] ? netback_changed+0x0/0x8e3 [xen_netfront]
[643359.899950] [<ffffffff811f62b4>] ? xenwatch_thread+0x87/0x14a
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff811f622d>] ? xenwatch_thread+0x0/0x14a
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] xenbus S 0000000000000001 0 17 2 0x00000000
[643359.899950] ffff88002fdd7100 0000000000000246 ffff88002fdd7138 ffff88002fdd7848
[643359.899950] 0000000000000000 0000000000000000 000000000000f9e0 ffff88002fe35fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fdd7810 ffff88002fdd7b08
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[643359.899950] [<ffffffff811f5faf>] ? xenbus_thread+0x0/0x27e
[643359.899950] [<ffffffff811f5b1a>] ? xb_wait_for_data_to_read+0x98/0xb4
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff811f5fb8>] ? xenbus_thread+0x9/0x27e
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] sync_supers S 0000000000000000 0 18 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000000000000000 00000020304045a0
[643359.899950] 0000000000000001 ffffffff810c971a 000000000000f9e0 ffff88002fe49fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe40000 ffff88002fe402f8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff810c971a>] ? bdi_sync_supers+0x0/0x3f
[643359.899950] [<ffffffff810f2046>] ? sync_supers+0x82/0x99
[643359.899950] [<ffffffff810c971a>] ? bdi_sync_supers+0x0/0x3f
[643359.899950] [<ffffffff810c9745>] ? bdi_sync_supers+0x2b/0x3f
[643359.899950] [<ffffffff810c971a>] ? bdi_sync_supers+0x0/0x3f
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] bdi-default S 0000000000000000 0 19 2 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000246 0000000000000000 0000000000000170
[643359.899950] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002fe4bfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe40710 ffff88002fe40a08
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.899950] [<ffffffff8105b9ec>] ? try_to_del_timer_sync+0x63/0x6c
[643359.899950] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.899950] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.899950] [<ffffffff810ca1be>] ? bdi_forker_task+0x19a/0x2d8
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8100f0bc>] ? xen_smp_send_reschedule+0x0/0x7
[643359.899950] [<ffffffff810ca024>] ? bdi_forker_task+0x0/0x2d8
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kintegrityd/0 S 0000000000000001 0 20 2 0x00000000
[643359.899950] ffff88002fdd0000 0000000000000246 0000001000000000 ffff88002fdd0000
[643359.899950] ffff880003194780 ffff8800031b27e8 000000000000f9e0 ffff88002fe4dfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe40e20 ffff88002fe41118
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kintegrityd/1 S 0000000000000001 0 21 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000001000000000 ffff88002fdd0000
[643359.899950] ffff880003194780 ffff8800031b27e8 000000000000f9e0 ffff88002fe4ffd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe41530 ffff88002fe41828
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kblockd/0 S 0000000000000000 0 22 2 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000246 0000000000000000 ffffffffa000a751
[643359.899950] 000000000000e030 ffff880002110000 000000000000f9e0 ffff88002fe5dfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe41c40 ffff88002fe41f38
[643359.899950] Call Trace:
[643359.899950] [<ffffffffa000a751>] ? do_blkif_request+0x316/0x374 [xen_blkfront]
[643359.899950] [<ffffffff8118a7bc>] ? cfq_queue_empty+0x0/0x13
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff8118b9dc>] ? cfq_kick_queue+0x0/0x41
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kblockd/1 S 0000000000000000 0 23 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000000000000000 ffffffffa000a7a0
[643359.899950] ffff88000255b9e0 ffff880002110000 000000000000f9e0 ffff88002fe5ffd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe42350 ffff88002fe42648
[643359.899950] Call Trace:
[643359.899950] [<ffffffffa000a7a0>] ? do_blkif_request+0x365/0x374 [xen_blkfront]
[643359.899950] [<ffffffff8118a7bc>] ? cfq_queue_empty+0x0/0x13
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff8118b9dc>] ? cfq_kick_queue+0x0/0x41
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kseriod S 0000000000000000 0 24 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000000000000000 ffff8800031b2780
[643359.899950] 0000000000000001 ffffffff81041497 000000000000f9e0 ffff88002fe61fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe42a60 ffff88002fe42d58
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81041497>] ? dequeue_entity+0x18/0x135
[643359.899950] [<ffffffff8123b986>] ? serio_thread+0x304/0x373
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8123b682>] ? serio_thread+0x0/0x373
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kondemand/0 S 0000000000000000 0 27 2 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000246 0000000000000000 ffffffff81040c8e
[643359.899950] ffff8800031b2780 ffff8800031b27e8 000000000000f9e0 ffff88002fe99fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe43f90 ffff88002fe44288
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81040c8e>] ? update_curr+0xa6/0x147
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kondemand/1 S 0000000000000001 0 28 2 0x00000000
[643359.899950] ffff88002fdd0000 0000000000000246 ffff88002fe446d8 ffffffff81040c8e
[643359.899950] ffff8800031b2780 ffff8800031b27e8 000000000000f9e0 ffff88002fe9bfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe446a0 ffff88002fe44998
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81040c8e>] ? update_curr+0xa6/0x147
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] khungtaskd S 0000000000000000 0 29 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000000000000000 0000000000000150
[643359.899950] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002dfbbfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe43170 ffff88002fe43468
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.899950] [<ffffffff8105b9ec>] ? try_to_del_timer_sync+0x63/0x6c
[643359.899950] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.899950] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.899950] [<ffffffff81095795>] ? watchdog+0x40/0x1ba
[643359.899950] [<ffffffff81095755>] ? watchdog+0x0/0x1ba
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kswapd0 D 0000000000000000 0 30 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000000000000000 0000000109951433
[643359.899950] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002dfbdfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe44db0 ffff88002fe450a8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.899950] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.899950] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.899950] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643359.899950] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810c11a8>] ? kswapd+0x5f9/0x686
[643359.899950] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff810c0baf>] ? kswapd+0x0/0x686
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] ksmd S 0000000000000001 0 31 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 ffff88002fe454c0 ffff88002fe457c0
[643359.899950] 0000000100000001 ffff88002dfbe000 000000000000f9e0 ffff88002dfbffd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe454c0 ffff88002fe457b8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff810e4673>] ? ksm_scan_thread+0x0/0xa1a
[643359.899950] [<ffffffff810e502a>] ? ksm_scan_thread+0x9b7/0xa1a
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810e4673>] ? ksm_scan_thread+0x0/0xa1a
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] aio/0 S 0000000000000000 0 32 2 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000246 0000000000000000 ffffffff81040c8e
[643359.899950] ffff8800031b2780 ffff8800031b27e8 000000000000f9e0 ffff88002dfe1fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe45bd0 ffff88002fe45ec8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81040c8e>] ? update_curr+0xa6/0x147
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] aio/1 S 0000000000000001 0 33 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 ffff88002fe46318 ffffffff81040c8e
[643359.899950] ffff8800031b2780 ffff8800031b27e8 000000000000f9e0 ffff88002dfe3fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe462e0 ffff88002fe465d8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81040c8e>] ? update_curr+0xa6/0x147
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] crypto/0 S 0000000000000000 0 34 2 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000246 0000000000000000 ffffffff81040c8e
[643359.899950] ffff8800031b2780 ffff8800031b27e8 000000000000f9e0 ffff88002dfe7fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe469f0 ffff88002fe46ce8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81040c8e>] ? update_curr+0xa6/0x147
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] crypto/1 S 0000000000000001 0 35 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 ffff88002fe47138 ffffffff81040c8e
[643359.899950] ffff8800031b2780 ffff8800031b27e8 000000000000f9e0 ffff88002dff5fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe47100 ffff88002fe473f8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81040c8e>] ? update_curr+0xa6/0x147
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] khvcd S 0000000000000000 0 38 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000000000000000 0000000000000000
[643359.899950] 0000000000000200 0000000000000200 000000000000f9e0 ffff88000250bfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002fe43880 ffff88002fe43b78
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8121db33>] ? khvcd+0xba/0x107
[643359.899950] [<ffffffff8121da79>] ? khvcd+0x0/0x107
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kjournald S ffff88000256aa60 0 114 2 0x00000000
[643359.899950] ffff88000256aa60 0000000000000246 ffff880002122424 ffff88001c4e9060
[643359.899950] ffff880002180000 0000000300200200 000000000000f9e0 ffff880002125fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88000256f100 ffff88000256f3f8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[643359.899950] [<ffffffffa00201ef>] ? kjournald+0x1a6/0x226 [jbd]
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffffa0020049>] ? kjournald+0x0/0x226 [jbd]
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] udevd S ffff88000265e9f0 0 165 1 0x00000000
[643359.899950] ffff88000265e9f0 0000000000000286 ffff8800027b5a58 ffff8800027b5a54
[643359.899950] dead000000200200 ffff8800027b5a98 000000000000f9e0 ffff8800027b5fd8
[643359.899950] 0000000000015780 0000000000015780 ffff880002568710 ffff880002568a08
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff81068d06>] ? hrtimer_try_to_cancel+0x3a/0x43
[643359.899950] [<ffffffff8130d6d4>] ? schedule_hrtimeout_range+0x4a/0x112
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff812c35be>] ? unix_dgram_poll+0x0/0x139
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950] [<ffffffff810fd3a0>] ? do_sys_poll+0x2f8/0x391
[643359.899950] [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950] [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950] [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950] [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950] [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950] [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff810993bc>] ? __call_rcu+0x110/0x118
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff810993bc>] ? __call_rcu+0x110/0x118
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff81051399>] ? release_task+0x3e9/0x3fa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff81154942>] ? cap_task_wait+0x0/0x3
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff81051f61>] ? do_wait+0x1dd/0x21b
[643359.899950] [<ffffffff8105203f>] ? sys_wait4+0xa0/0xb0
[643359.899950] [<ffffffff810fd5c6>] ? sys_poll+0x47/0xb2
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] udevd S ffff88002f163f90 0 205 165 0x00000000
[643359.899950] ffff88002f163f90 0000000000000282 0000000000000000 0000000000000000
[643359.899950] 0000000000000000 0000000000000000 000000000000f9e0 ffff880002703fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002f164db0 ffff88002f1650a8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8130d6d4>] ? schedule_hrtimeout_range+0x4a/0x112
[643359.899950] [<ffffffff811549b7>] ? cap_socket_getpeersec_dgram+0x0/0x6
[643359.899950] [<ffffffff8127bec0>] ? netlink_sendmsg+0x162/0x255
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8125a6e8>] ? datagram_poll+0x0/0xc2
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8125a6e8>] ? datagram_poll+0x0/0xc2
[643359.899950] [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950] [<ffffffff810fd3a0>] ? do_sys_poll+0x2f8/0x391
[643359.899950] [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950] [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff81154984>] ? cap_unix_may_send+0x0/0x3
[643359.899950] [<ffffffff8125462f>] ? sock_def_readable+0x10/0x62
[643359.899950] [<ffffffff812c352e>] ? unix_dgram_sendmsg+0x3fd/0x444
[643359.899950] [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff810ba77e>] ? free_hot_cold_page+0x1a2/0x1af
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950] [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
[643359.899950] [<ffffffff8100e585>] ? xen_set_pte_at+0xa2/0xc2
[643359.899950] [<ffffffff8125163d>] ? sockfd_lookup_light+0x1a/0x51
[643359.899950] [<ffffffff8125285e>] ? sys_sendto+0xf5/0x138
[643359.899950] [<ffffffff8130e002>] ? _spin_lock_irq+0x7/0x22
[643359.899950] [<ffffffff810fd4eb>] ? sys_ppoll+0xb2/0x146
[643359.899950] [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] udevd S ffff88002f165bd0 0 206 165 0x00000000
[643359.899950] ffff88002f165bd0 0000000000000286 0000000000000000 0000000000000000
[643359.899950] 0000000000000000 0000000000000000 000000000000f9e0 ffff8800025d3fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002f1654c0 ffff88002f1657b8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8130d6d4>] ? schedule_hrtimeout_range+0x4a/0x112
[643359.899950] [<ffffffff811549b7>] ? cap_socket_getpeersec_dgram+0x0/0x6
[643359.899950] [<ffffffff81259304>] ? __alloc_skb+0x3e/0x15a
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8125a6e8>] ? datagram_poll+0x0/0xc2
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8125a6e8>] ? datagram_poll+0x0/0xc2
[643359.899950] [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950] [<ffffffff810fd3a0>] ? do_sys_poll+0x2f8/0x391
[643359.899950] [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950] [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff81154984>] ? cap_unix_may_send+0x0/0x3
[643359.899950] [<ffffffff8125462f>] ? sock_def_readable+0x10/0x62
[643359.899950] [<ffffffff812c352e>] ? unix_dgram_sendmsg+0x3fd/0x444
[643359.899950] [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff810ba77e>] ? free_hot_cold_page+0x1a2/0x1af
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950] [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
[643359.899950] [<ffffffff8100e585>] ? xen_set_pte_at+0xa2/0xc2
[643359.899950] [<ffffffff8125163d>] ? sockfd_lookup_light+0x1a/0x51
[643359.899950] [<ffffffff8125285e>] ? sys_sendto+0xf5/0x138
[643359.899950] [<ffffffff8130e002>] ? _spin_lock_irq+0x7/0x22
[643359.899950] [<ffffffff810fd4eb>] ? sys_ppoll+0xb2/0x146
[643359.899950] [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] flush-202:0 S 0000000000000000 0 418 2 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000246 0000000000000000 0000000000000180
[643359.899950] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002f675fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002ea08710 ffff88002ea08a08
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.899950] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.899950] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.899950] [<ffffffff8110a6ea>] ? bdi_writeback_task+0x7f/0xaa
[643359.899950] [<ffffffff810ca2fc>] ? bdi_start_fn+0x0/0xd0
[643359.899950] [<ffffffff810ca36c>] ? bdi_start_fn+0x70/0xd0
[643359.899950] [<ffffffff810ca2fc>] ? bdi_start_fn+0x0/0xd0
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] rpciod/0 S ffff88002ea0dbd0 0 512 2 0x00000000
[643359.899950] ffff88002ea0f100 0000000000000246 ffff88000319cf20 0000000000000000
[643359.899950] 0000000000000000 ffffffff8106609b 000000000000f9e0 ffff88002ea33fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002ea0dbd0 ffff88002ea0dec8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8106609b>] ? autoremove_wake_function+0x9/0x2e
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffffa00dc59c>] ? rpc_async_schedule+0x0/0xc [sunrpc]
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] rpciod/1 S 0000000000000000 0 513 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000000000000000 ffffffff81054915
[643359.899950] ffff8800026a0180 ffffffffa00dc203 000000000000f9e0 ffff88002ea35fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002ea0b170 ffff88002ea0b468
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[643359.899950] [<ffffffffa00dc203>] ? rpc_wake_up_next+0x181/0x188 [sunrpc]
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffffa00dc59c>] ? rpc_async_schedule+0x0/0xc [sunrpc]
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kslowd000 S 0000000000000000 0 514 2 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000246 0000000000000000 ffff8800031947e8
[643359.899950] 0000000000000001 ffff88002ea36000 000000000000f9e0 ffff88002ea37fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002ea0b880 ffff88002ea0bb78
[643359.899950] Call Trace:
[643359.899950] [<ffffffff810af145>] ? slow_work_thread+0x140/0x2c6
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810af005>] ? slow_work_thread+0x0/0x2c6
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] kslowd001 S 0000000000000000 0 515 2 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000000000000000 ffff8800031b27e8
[643359.899950] 0000000000000001 ffff88002e7e0000 000000000000f9e0 ffff88002e7e1fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002ea0f810 ffff88002ea0fb08
[643359.899950] Call Trace:
[643359.899950] [<ffffffff810af145>] ? slow_work_thread+0x140/0x2c6
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810af005>] ? slow_work_thread+0x0/0x2c6
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] nfsiod S 0000000000000000 0 516 2 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000246 0000000000000000 ffffffff8100e635
[643359.899950] ffffffff8100ecf2 ffff88002b89d300 000000000000f9e0 ffff88002df7ffd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002ea0f100 ffff88002ea0f3f8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff810b6e17>] ? mempool_free_slab+0x0/0xe
[643359.899950] [<ffffffff810629a3>] ? worker_thread+0xcc/0x21d
[643359.899950] [<ffffffffa00db998>] ? rpc_async_release+0x0/0xc [sunrpc]
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810628d7>] ? worker_thread+0x0/0x21d
[643359.899950] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643359.899950] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643359.899950] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643359.899950] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643359.899950] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643359.899950] syslogd S 0000000000000000 0 776 1 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000286 0000000000000000 ffff8800031acd10
[643359.899950] ffff880003194780 ffff88002ea2c000 000000000000f9e0 ffff88002ea2dfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002ea0bf90 ffff88002ea0c288
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8109b022>] ? delayacct_end+0x74/0x7f
[643359.899950] [<ffffffff81065fb3>] ? bit_waitqueue+0x10/0xa0
[643359.899950] [<ffffffff8130d6d4>] ? schedule_hrtimeout_range+0x4a/0x112
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff812c35be>] ? unix_dgram_poll+0x0/0x139
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950] [<ffffffff810fdb9f>] ? do_select+0x528/0x57a
[643359.899950] [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950] [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950] [<ffffffff810b620e>] ? generic_file_buffered_write+0x1f5/0x278
[643359.899950] [<ffffffff810b6642>] ? __generic_file_aio_write+0x25f/0x293
[643359.899950] [<ffffffff8130d666>] ? mutex_lock+0xd/0x31
[643359.899950] [<ffffffff810f6f6a>] ? pipe_write+0x48f/0x4da
[643359.899950] [<ffffffff810b66f9>] ? generic_file_aio_write+0x83/0x9f
[643359.899950] [<ffffffff810f6adb>] ? pipe_write+0x0/0x4da
[643359.899950] [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950] [<ffffffff810fdd75>] ? core_sys_select+0x184/0x21e
[643359.899950] [<ffffffff810efd65>] ? rw_copy_check_uvector+0x6d/0xe4
[643359.899950] [<ffffffff810f054f>] ? do_readv_writev+0xb2/0x115
[643359.899950] [<ffffffff810fe042>] ? sys_select+0x92/0xbb
[643359.899950] [<ffffffff8105c978>] ? sys_rt_sigprocmask+0x4c/0xc4
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] bacula-fd S 0000000000000000 0 859 1 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000286 ffffffff8100e635 ffffffff8100ecf2
[643359.899950] ffffffff8118c035 ffff880002101000 000000000000f9e0 ffff8800021a1fd8
[643359.899950] 0000000000015780 0000000000015780 ffff880002658000 ffff8800026582f8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8118c035>] ? cfq_set_request+0x0/0x44d
[643359.899950] [<ffffffff8117864c>] ? elv_rb_del+0x2f/0x47
[643359.899950] [<ffffffffa000a43b>] ? do_blkif_request+0x0/0x374 [xen_blkfront]
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffffa000a43b>] ? do_blkif_request+0x0/0x374 [xen_blkfront]
[643359.899950] [<ffffffff8130d6d4>] ? schedule_hrtimeout_range+0x4a/0x112
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8128d2cf>] ? tcp_poll+0x0/0x14a
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8128d2cf>] ? tcp_poll+0x0/0x14a
[643359.899950] [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950] [<ffffffff810fdb9f>] ? do_select+0x528/0x57a
[643359.899950] [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950] [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950] [<ffffffff81041497>] ? dequeue_entity+0x18/0x135
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff810baed0>] ? get_page_from_freelist+0x4bc/0x760
[643359.899950] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8119934b>] ? __bitmap_weight+0x3a/0x7e
[643359.899950] [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950] [<ffffffff810fdd75>] ? core_sys_select+0x184/0x21e
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643359.899950] [<ffffffff8104e864>] ? do_fork+0x254/0x31e
[643359.899950] [<ffffffff810fe042>] ? sys_select+0x92/0xbb
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] bacula-fd S 0000000000000000 0 861 1 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000286 0000000000000000 00007fa60c740e70
[643359.899950] 0000000085145b7e 0000000000000001 000000000000f9e0 ffff8800025f7fd8
[643359.899950] 0000000000015780 0000000000015780 ffff880002658e20 ffff880002659118
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81068f88>] ? __hrtimer_start_range_ns+0x22f/0x242
[643359.899950] [<ffffffff81072a43>] ? futex_wait_queue_me+0xa4/0xbd
[643359.899950] [<ffffffff81072ba1>] ? futex_wait+0x145/0x291
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff81068587>] ? hrtimer_wakeup+0x0/0x22
[643359.899950] [<ffffffff81072a19>] ? futex_wait_queue_me+0x7a/0xbd
[643359.899950] [<ffffffff81074799>] ? do_futex+0x9d/0x995
[643359.899950] [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3
[643359.899950] [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643359.899950] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950] [<ffffffff810751a4>] ? sys_futex+0x113/0x131
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] cron S 0000000000000000 0 886 1 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000282 0000000000000000 0000000000000008
[643359.899950] 00000000000007dd 0000000000000001 000000000000f9e0 ffff88002dfd5fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002ea09530 ffff88002ea09828
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81068f88>] ? __hrtimer_start_range_ns+0x22f/0x242
[643359.899950] [<ffffffff8130d80a>] ? do_nanosleep+0x65/0xa3
[643359.899950] [<ffffffff81069040>] ? hrtimer_nanosleep+0x8b/0x10e
[643359.899950] [<ffffffff81068587>] ? hrtimer_wakeup+0x0/0x22
[643359.899950] [<ffffffff8130d7ef>] ? do_nanosleep+0x4a/0xa3
[643359.899950] [<ffffffff8106910f>] ? sys_nanosleep+0x4c/0x62
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] klogd S ffff88002ea0bf90 0 1223 1 0x00000000
[643359.899950] ffff88002ea0bf90 0000000000000286 ffff88000203dd80 0000000000000000
[643359.899950] ffff880000000000 ffff88000203ded8 000000000000f9e0 ffff88000203dfd8
[643359.899950] 0000000000015780 0000000000015780 ffff880002569c40 ffff880002569f38
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff81152d27>] ? cap_capable+0x0/0x1c
[643359.899950] [<ffffffff8104ffa1>] ? do_syslog+0x102/0x394
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8113aa1d>] ? kmsg_read+0x3a/0x45
[643359.899950] [<ffffffff8113251d>] ? proc_reg_read+0x6f/0x8a
[643359.899950] [<ffffffff810f0b24>] ? vfs_read+0xa6/0xff
[643359.899950] [<ffffffff810f0c39>] ? sys_read+0x45/0x6e
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] vsftpd S 0000000000000000 0 1248 1 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000286 0000000000000000 ffff88002f11bd78
[643359.899950] 0000000000000040 00000000000200da 000000000000f9e0 ffff88002f11bfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002ea09c40 ffff88002ea09f38
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8130e129>] ? _spin_lock_bh+0x9/0x25
[643359.899950] [<ffffffff8128cc86>] ? inet_csk_accept+0xce/0x1e9
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff81118afc>] ? inotify_d_instantiate+0x12/0x39
[643359.899950] [<ffffffff812ab101>] ? inet_accept+0x25/0xb1
[643359.899950] [<ffffffff812530cc>] ? sys_accept4+0x106/0x1d7
[643359.899950] [<ffffffff8104e864>] ? do_fork+0x254/0x31e
[643359.899950] [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] sshd S ffff88000256aa60 0 1281 1 0x00000000
[643359.899950] ffff88000256aa60 0000000000000286 ffffffff8100e635 ffffffff8100ecf2
[643359.899950] ffffffff8118c035 0000160000000000 000000000000f9e0 ffff88000209bfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88000256b170 ffff88000256b468
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8118c035>] ? cfq_set_request+0x0/0x44d
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8118c418>] ? cfq_set_request+0x3e3/0x44d
[643359.899950] [<ffffffff8130d6d4>] ? schedule_hrtimeout_range+0x4a/0x112
[643359.899950] [<ffffffff8128d2cf>] ? tcp_poll+0x0/0x14a
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8128d2cf>] ? tcp_poll+0x0/0x14a
[643359.899950] [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950] [<ffffffff810fdb9f>] ? do_select+0x528/0x57a
[643359.899950] [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950] [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950] [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950] [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff8109b022>] ? delayacct_end+0x74/0x7f
[643359.899950] [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643359.899950] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[643359.899950] [<ffffffff810b54c6>] ? __lock_page+0x5d/0x63
[643359.899950] [<ffffffff810660c0>] ? wake_bit_function+0x0/0x23
[643359.899950] [<ffffffff810b54d5>] ? lock_page+0x9/0x1f
[643359.899950] [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950] [<ffffffff810fdd75>] ? core_sys_select+0x184/0x21e
[643359.899950] [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff810fe042>] ? sys_select+0x92/0xbb
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] munin-node S 0000000000000000 0 1439 1 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000282 0000000000000000 0000000000000000
[643359.899950] ffff88002f9b6c20 0000000000000001 000000000000f9e0 ffff880002575fd8
[643359.899950] 0000000000015780 0000000000015780 ffff880002568000 ffff8800025682f8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81068f88>] ? __hrtimer_start_range_ns+0x22f/0x242
[643359.899950] [<ffffffff81068d06>] ? hrtimer_try_to_cancel+0x3a/0x43
[643359.899950] [<ffffffff8130d757>] ? schedule_hrtimeout_range+0xcd/0x112
[643359.899950] [<ffffffff81068587>] ? hrtimer_wakeup+0x0/0x22
[643359.899950] [<ffffffff8130d739>] ? schedule_hrtimeout_range+0xaf/0x112
[643359.899950] [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950] [<ffffffff810fdb9f>] ? do_select+0x528/0x57a
[643359.899950] [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950] [<ffffffff810fe141>] ? pollwake+0x0/0x5b
[643359.899950] [<ffffffff810492e7>] ? finish_task_switch+0x44/0xaf
[643359.899950] [<ffffffff8130cd30>] ? thread_return+0x79/0xe0
[643359.899950] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff8109b022>] ? delayacct_end+0x74/0x7f
[643359.899950] [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643359.899950] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[643359.899950] [<ffffffff810b54c6>] ? __lock_page+0x5d/0x63
[643359.899950] [<ffffffff810660c0>] ? wake_bit_function+0x0/0x23
[643359.899950] [<ffffffff810b54d5>] ? lock_page+0x9/0x1f
[643359.899950] [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950] [<ffffffff810fdd75>] ? core_sys_select+0x184/0x21e
[643359.899950] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950] [<ffffffff8106cbaa>] ? timekeeping_get_ns+0xe/0x2e
[643359.899950] [<ffffffff8106d74d>] ? ktime_get_ts+0x61/0x88
[643359.899950] [<ffffffff810fe042>] ? sys_select+0x92/0xbb
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] getty S 0000000000000000 0 1486 1 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000286 0000000000000000 ffffffff8120e775
[643359.899950] 000000000000001c ffffffff8130e0ca 000000000000f9e0 ffff8800027e3fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88000256c6a0 ffff88000256c998
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8120e775>] ? set_termios+0x3c9/0x3fb
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff811548e9>] ? cap_file_permission+0x0/0x3
[643359.899950] [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950] [<ffffffff811548e9>] ? cap_file_permission+0x0/0x3
[643359.899950] [<ffffffff8130e002>] ? _spin_lock_irq+0x7/0x22
[643359.899950] [<ffffffff8106337f>] ? flush_work+0x66/0x87
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff8105b9ec>] ? try_to_del_timer_sync+0x63/0x6c
[643359.899950] [<ffffffff8120cbb5>] ? n_tty_read+0x498/0x774
[643359.899950] [<ffffffff811548e9>] ? cap_file_permission+0x0/0x3
[643359.899950] [<ffffffff8104b4bc>] ? default_wake_function+0x0/0x9
[643359.899950] [<ffffffff81208906>] ? tty_read+0x76/0xb0
[643359.899950] [<ffffffff810f0b24>] ? vfs_read+0xa6/0xff
[643359.899950] [<ffffffff810f0c39>] ? sys_read+0x45/0x6e
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] apache2 S 0000000000000000 0 26160 1 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000286 0000000000000000 0000000000000000
[643359.899950] 0000000000000000 0000000000000001 000000000000f9e0 ffff8800022d3fd8
[643359.899950] 0000000000015780 0000000000015780 ffff880002048e20 ffff880002049118
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81068f88>] ? __hrtimer_start_range_ns+0x22f/0x242
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8130d757>] ? schedule_hrtimeout_range+0xcd/0x112
[643359.899950] [<ffffffff81068587>] ? hrtimer_wakeup+0x0/0x22
[643359.899950] [<ffffffff8130d739>] ? schedule_hrtimeout_range+0xaf/0x112
[643359.899950] [<ffffffff810fd000>] ? poll_schedule_timeout+0x31/0x4f
[643359.899950] [<ffffffff810fdb9f>] ? do_select+0x528/0x57a
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8100c1a7>] ? xen_mc_flush+0x159/0x185
[643359.899950] [<ffffffff810fe06b>] ? __pollwait+0x0/0xd6
[643359.899950] [<ffffffff810492e7>] ? finish_task_switch+0x44/0xaf
[643359.899950] [<ffffffff8130cd30>] ? thread_return+0x79/0xe0
[643359.899950] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff8109b022>] ? delayacct_end+0x74/0x7f
[643359.899950] [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643359.899950] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[643359.899950] [<ffffffff810b54c6>] ? __lock_page+0x5d/0x63
[643359.899950] [<ffffffff810660c0>] ? wake_bit_function+0x0/0x23
[643359.899950] [<ffffffff810b54d5>] ? lock_page+0x9/0x1f
[643359.899950] [<ffffffff8100c426>] ? pfn_to_mfn+0xe/0x22
[643359.899950] [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950] [<ffffffff810fdd75>] ? core_sys_select+0x184/0x21e
[643359.899950] [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3
[643359.899950] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950] [<ffffffff8106cbaa>] ? timekeeping_get_ns+0xe/0x2e
[643359.899950] [<ffffffff8106d74d>] ? ktime_get_ts+0x61/0x88
[643359.899950] [<ffffffff810fe042>] ? sys_select+0x92/0xbb
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] atop S 0000000000000000 0 22273 1 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000282 0000000000000000 0000000200000000
[643359.899950] ffff88002fbad8d0 0000000000000002 000000000000f9e0 ffff880002eb5fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002f167810 ffff88002f167b08
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8105c69c>] ? sys_pause+0x19/0x23
[643359.899950] [<ffffffff81011b63>] ? sysret_check+0x17/0x5a
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] sshd D ffff88000265b880 0 17733 1281 0x00000000
[643359.899950] ffff88000265b880 0000000000000286 0000000000000000 0000000109951433
[643359.899950] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002bfcbfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88000256aa60 ffff88000256ad58
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.899950] [<ffffffff810492e7>] ? finish_task_switch+0x44/0xaf
[643359.899950] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.899950] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.899950] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643359.899950] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810c17c0>] ? try_to_free_pages+0x2e8/0x38e
[643359.899950] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643359.899950] [<ffffffff810c97cd>] ? congestion_wait+0x74/0x80
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643359.899950] [<ffffffff81254a33>] ? sock_alloc_send_pskb+0xdd/0x2f6
[643359.899950] [<ffffffff810e7e3a>] ? kmalloc_large_node+0x5d/0x9b
[643359.899950] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[643359.899950] [<ffffffff81254a33>] ? sock_alloc_send_pskb+0xdd/0x2f6
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[643359.899950] [<ffffffff812c408e>] ? unix_stream_sendmsg+0x13b/0x2c8
[643359.899950] [<ffffffff812514dc>] ? sock_aio_write+0xb1/0xbc
[643359.899950] [<ffffffff810effee>] ? do_sync_write+0xce/0x113
[643359.899950] [<ffffffff8101079c>] ? __switch_to+0x285/0x297
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810492e7>] ? finish_task_switch+0x44/0xaf
[643359.899950] [<ffffffff8130cd30>] ? thread_return+0x79/0xe0
[643359.899950] [<ffffffff810f0953>] ? vfs_write+0xbc/0x102
[643359.899950] [<ffffffff810f0a55>] ? sys_write+0x45/0x6e
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] sshd ? 0000000000000000 0 17736 17733 0x00100000
[643359.899950] ffff88002fdd0e20 0000000000000246 0000000000000000 ffffffff8100e635
[643359.899950] ffffffff8100ecf2 0000000000000009 000000000000f9e0 ffff88002bba7fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88000256e9f0 ffff88000256ece8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8105287a>] ? do_exit+0x6bb/0x6c9
[643359.899950] [<ffffffff8105d6d5>] ? __dequeue_signal+0xfb/0x124
[643359.899950] [<ffffffff810528fe>] ? do_group_exit+0x76/0x9d
[643359.899950] [<ffffffff8105f1eb>] ? get_signal_to_deliver+0x310/0x339
[643359.899950] [<ffffffff81011037>] ? do_notify_resume+0x87/0x73f
[643359.899950] [<ffffffff810f0b37>] ? vfs_read+0xb9/0xff
[643359.899950] [<ffffffff81011e0e>] ? int_signal+0x12/0x17
[643359.899950] apache2 ? ffff88000204f810 0 17941 26160 0x00100000
[643359.899950] ffff88000204f810 0000000000000246 0000000000000000 ffffffff8100e635
[643359.899950] ffffffff8100ecf2 0000000000000009 000000000000f9e0 ffff88002bf87fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002f163880 ffff88002f163b78
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8105287a>] ? do_exit+0x6bb/0x6c9
[643359.899950] [<ffffffff810528fe>] ? do_group_exit+0x76/0x9d
[643359.899950] [<ffffffff8105f1eb>] ? get_signal_to_deliver+0x310/0x339
[643359.899950] [<ffffffff81011037>] ? do_notify_resume+0x87/0x73f
[643359.899950] [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643359.899950] [<ffffffff8105cbe9>] ? do_sigaction+0x159/0x171
[643359.899950] [<ffffffff810f0b24>] ? vfs_read+0xa6/0xff
[643359.899950] [<ffffffff81011e0e>] ? int_signal+0x12/0x17
[643359.899950] apache2 D ffff88002b8fd6e8 0 17946 26160 0x00000004
[643359.899950] ffff88002fe44db0 0000000000000286 ffffffff810bb9e1 0000000109951433
[643359.899950] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002b8fdfd8
[643359.899950] 0000000000015780 0000000000015780 ffff880002049c40 ffff880002049f38
[643359.899950] Call Trace:
[643359.899950] [<ffffffff810bb9e1>] ? __set_page_dirty_no_writeback+0x0/0xc
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643359.899950] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643359.899950] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643359.899950] [<ffffffff810bb9e1>] ? __set_page_dirty_no_writeback+0x0/0xc
[643359.899950] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643359.899950] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[643359.899950] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643359.899950] [<ffffffff810c97cd>] ? congestion_wait+0x74/0x80
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[643359.899950] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643359.899950] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643359.899950] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[643359.899950] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[643359.899950] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643359.899950] [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[643359.899950] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643359.899950] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[643359.899950] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643359.899950] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643359.899950] [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[643359.899950] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643359.899950] [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643359.899950] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[643359.899950] [<ffffffff8110d00c>] ? __generic_file_splice_read+0x36e/0x3bd
[643359.899950] [<ffffffff81251f64>] ? kernel_sendmsg+0x32/0x3f
[643359.899950] [<ffffffff81254790>] ? sock_no_sendpage+0x87/0x94
[643359.899950] [<ffffffff812909f1>] ? tcp_sendpage+0x4b/0x458
[643359.899950] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643359.899950] [<ffffffff81250d0d>] ? kernel_sendpage+0x16/0x1f
[643359.899950] [<ffffffff81250d4b>] ? sock_sendpage+0x35/0x3a
[643359.899950] [<ffffffff8110b122>] ? pipe_to_sendpage+0x66/0x74
[643359.899950] [<ffffffff8110b17c>] ? splice_from_pipe_feed+0x4c/0xd4
[643359.899950] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643359.899950] [<ffffffff8110b405>] ? __splice_from_pipe+0x3e/0x69
[643359.899950] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643359.899950] [<ffffffff8110b47d>] ? splice_from_pipe+0x4d/0x63
[643359.899950] [<ffffffff8110c00d>] ? direct_splice_actor+0x1b/0x1e
[643359.899950] [<ffffffff8110b9c8>] ? splice_direct_to_actor+0xdc/0x188
[643359.899950] [<ffffffff81041432>] ? set_next_entity+0x34/0x56
[643359.899950] [<ffffffff8110bff2>] ? direct_splice_actor+0x0/0x1e
[643359.899950] [<ffffffff8110babe>] ? do_splice_direct+0x4a/0x64
[643359.899950] [<ffffffff810f0312>] ? do_sendfile+0x12d/0x1a8
[643359.899950] [<ffffffff810f03d6>] ? sys_sendfile64+0x49/0x89
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] munin-node S 0000000000000000 0 17965 1439 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000286 0000000000000000 ffff88001cca2124
[643359.899950] ffff88001cca2138 ffff88002de2c780 000000000000f9e0 ffff880029c61fd8
[643359.899950] 0000000000015780 0000000000015780 ffff88002f1669f0 ffff88002f166ce8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8130e129>] ? _spin_lock_bh+0x9/0x25
[643359.899950] [<ffffffff8125481f>] ? sk_wait_data+0x82/0xc4
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8128f6df>] ? tcp_recvmsg+0x4c3/0xa9e
[643359.899950] [<ffffffff8125361a>] ? sock_common_recvmsg+0x30/0x45
[643359.899950] [<ffffffff81251618>] ? sock_aio_read+0xb9/0xc4
[643359.899950] [<ffffffff810376da>] ? flush_tlb_page+0x5a/0x7b
[643359.899950] [<ffffffff8100c2f1>] ? __raw_callee_save_xen_pte_val+0x11/0x1e
[643359.899950] [<ffffffff810cc4d6>] ? do_wp_page+0x2fc/0x707
[643359.899950] [<ffffffff810f0101>] ? do_sync_read+0xce/0x113
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff810f0b37>] ? vfs_read+0xb9/0xff
[643359.899950] [<ffffffff810f0c39>] ? sys_read+0x45/0x6e
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] apache2 S ffff88000265a350 0 18163 26160 0x00000000
[643359.899950] ffff88000265a350 0000000000000286 ffff88002e7d0188 0000000000000016
[643359.899950] 0000000000000200 ffffffff8100ecdf 000000000000f9e0 ffff88002b8dffd8
[643359.899950] 0000000000015780 0000000000015780 ffff88000256e2e0 ffff88000256e5d8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643359.899950] [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950] [<ffffffff81154996>] ? cap_socket_accept+0x0/0x3
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff81154996>] ? cap_socket_accept+0x0/0x3
[643359.899950] [<ffffffff8125b47b>] ? __skb_recv_datagram+0x1e3/0x24b
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8125b504>] ? receiver_wake_function+0x0/0x14
[643359.899950] [<ffffffff8125b4ff>] ? skb_recv_datagram+0x1c/0x21
[643359.899950] [<ffffffff812c4426>] ? unix_accept+0x56/0xfd
[643359.899950] [<ffffffff812530cc>] ? sys_accept4+0x106/0x1d7
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff810993bc>] ? __call_rcu+0x110/0x118
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff8106a975>] ? commit_creds+0xf5/0xfc
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] vsftpd S 0000000000000000 0 18168 1248 0x00000000
[643359.899950] ffff88002fdd0e20 0000000000000282 0000000000000000 0000000000000020
[643359.899950] 0000000100000001 0000000000000000 000000000000f9e0 ffff88002bf09fd8
[643359.899950] 0000000000015780 0000000000015780 ffff880002658710 ffff880002658a08
[643359.899950] Call Trace:
[643359.899950] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950] [<ffffffff812c3980>] ? unix_stream_recvmsg+0x0/0x5d3
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff812c3980>] ? unix_stream_recvmsg+0x0/0x5d3
[643359.899950] [<ffffffff812c3c65>] ? unix_stream_recvmsg+0x2e5/0x5d3
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff81251618>] ? sock_aio_read+0xb9/0xc4
[643359.899950] [<ffffffff810331dc>] ? no_context+0x25/0x1f8
[643359.899950] [<ffffffff8100c2f1>] ? __raw_callee_save_xen_pte_val+0x11/0x1e
[643359.899950] [<ffffffff810cc4d6>] ? do_wp_page+0x2fc/0x707
[643359.899950] [<ffffffff810f0101>] ? do_sync_read+0xce/0x113
[643359.899950] [<ffffffff81154912>] ? cap_cred_commit+0x0/0x1
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff810993bc>] ? __call_rcu+0x110/0x118
[643359.899950] [<ffffffff810f0b37>] ? vfs_read+0xb9/0xff
[643359.899950] [<ffffffff810f0c39>] ? sys_read+0x45/0x6e
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] vsftpd S ffff88000265b880 0 18173 18168 0x00000000
[643359.899950] ffff88000265b880 0000000000000286 ffff88002f690000 ffff8800024e5f00
[643359.899950] ffff88002f69009c ffff8800020c8000 000000000000f9e0 ffff88002becdfd8
[643359.899950] 0000000000015780 0000000000015780 ffff88000265f810 ffff88000265fb08
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[643359.899950] [<ffffffff8126122a>] ? dev_queue_xmit+0x35b/0x38d
[643359.899950] [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8130e129>] ? _spin_lock_bh+0x9/0x25
[643359.899950] [<ffffffff8125481f>] ? sk_wait_data+0x82/0xc4
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8128f6df>] ? tcp_recvmsg+0x4c3/0xa9e
[643359.899950] [<ffffffff8130e129>] ? _spin_lock_bh+0x9/0x25
[643359.899950] [<ffffffff8125361a>] ? sock_common_recvmsg+0x30/0x45
[643359.899950] [<ffffffff81251b69>] ? sock_recvmsg+0xa6/0xbe
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
[643359.899950] [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3
[643359.899950] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8125163d>] ? sockfd_lookup_light+0x1a/0x51
[643359.899950] [<ffffffff81252dd4>] ? sys_recvfrom+0xba/0x120
[643359.899950] [<ffffffff81052d1f>] ? do_setitimer+0x1a9/0x1d9
[643359.899950] [<ffffffff81052e03>] ? alarm_setitimer+0x35/0x63
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] vsftpd S 0000000000000000 0 18176 1248 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000282 0000000000000000 0000000000000000
[643359.899950] ffff880002659c40 0000000000000000 000000000000f9e0 ffff88002ac33fd8
[643359.899950] 0000000000015780 0000000000015780 ffff880002659c40 ffff880002659f38
[643359.899950] Call Trace:
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950] [<ffffffff812c3980>] ? unix_stream_recvmsg+0x0/0x5d3
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff812c3980>] ? unix_stream_recvmsg+0x0/0x5d3
[643359.899950] [<ffffffff812c3c65>] ? unix_stream_recvmsg+0x2e5/0x5d3
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff81251618>] ? sock_aio_read+0xb9/0xc4
[643359.899950] [<ffffffff810331dc>] ? no_context+0x25/0x1f8
[643359.899950] [<ffffffff8100c2f1>] ? __raw_callee_save_xen_pte_val+0x11/0x1e
[643359.899950] [<ffffffff810cc4d6>] ? do_wp_page+0x2fc/0x707
[643359.899950] [<ffffffff810f0101>] ? do_sync_read+0xce/0x113
[643359.899950] [<ffffffff81154912>] ? cap_cred_commit+0x0/0x1
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643359.899950] [<ffffffff810993bc>] ? __call_rcu+0x110/0x118
[643359.899950] [<ffffffff810f0b37>] ? vfs_read+0xb9/0xff
[643359.899950] [<ffffffff810f0c39>] ? sys_read+0x45/0x6e
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] vsftpd S 0000000000000000 0 18179 18176 0x00000000
[643359.899950] ffffffff814a11f0 0000000000000282 0000000000000000 ffff880002110000
[643359.899950] 0000000000000000 000000008118ce88 000000000000f9e0 ffff88002c5fffd8
[643359.899950] 0000000000015780 0000000000015780 ffff88000265d4c0 ffff88000265d7b8
[643359.899950] Call Trace:
[643359.899950] [<ffffffff81180f90>] ? __make_request+0x3e3/0x428
[643359.899950] [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643359.899950] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643359.899950] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643359.899950] [<ffffffff8130e129>] ? _spin_lock_bh+0x9/0x25
[643359.899950] [<ffffffff8125481f>] ? sk_wait_data+0x82/0xc4
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8128f6df>] ? tcp_recvmsg+0x4c3/0xa9e
[643359.899950] [<ffffffffa00337d1>] ? ext3_get_block+0x0/0xf1 [ext3]
[643359.899950] [<ffffffff810b57e1>] ? add_to_page_cache_locked+0x98/0xc1
[643359.899950] [<ffffffff8125361a>] ? sock_common_recvmsg+0x30/0x45
[643359.899950] [<ffffffff81251b69>] ? sock_recvmsg+0xa6/0xbe
[643359.899950] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643359.899950] [<ffffffff8100c4e8>] ? pte_pfn_to_mfn+0x21/0x30
[643359.899950] [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
[643359.899950] [<ffffffff8100e585>] ? xen_set_pte_at+0xa2/0xc2
[643359.899950] [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3
[643359.899950] [<ffffffff8118f7c8>] ? cpumask_any_but+0x28/0x34
[643359.899950] [<ffffffff8125163d>] ? sockfd_lookup_light+0x1a/0x51
[643359.899950] [<ffffffff81252dd4>] ? sys_recvfrom+0xba/0x120
[643359.899950] [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[643359.899950] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643359.899950] Sched Debug Version: v0.09, 2.6.32-5-xen-amd64 #1
[643359.899950] now at 643361703.344061 msecs
[643359.899950] .jiffies : 4455732269
[643359.899950] .sysctl_sched_latency : 10.000000
[643359.899950] .sysctl_sched_min_granularity : 2.000000
[643359.899950] .sysctl_sched_wakeup_granularity : 2.000000
[643359.899950] .sysctl_sched_child_runs_first : 0.000000
[643359.899950] .sysctl_sched_features : 32611451
[643359.899950]
[643359.899950] cpu#0, 2333.472 MHz
[643359.899950] .nr_running : 0
[643359.899950] .load : 0
[643359.899950] .nr_switches : 174783562
[643359.899950] .nr_load_updates : 120151717
[643359.899950] .nr_uninterruptible : 2
[643359.899950] .next_balance : 4455.732270
[643359.899950] .curr->pid : 0
[643359.899950] .clock : 643359892.013090
[643359.899950] .cpu_load[0] : 0
[643359.899950] .cpu_load[1] : 8
[643359.899950] .cpu_load[2] : 60
[643359.899950] .cpu_load[3] : 142
[643359.899950] .cpu_load[4] : 225
[643359.899950]
[643359.899950] cfs_rq[0]:/
[643359.899950] .exec_clock : 0.000000
[643359.899950] .MIN_vruntime : 0.000001
[643359.899950] .min_vruntime : 99987640.710363
[643359.899950] .max_vruntime : 0.000001
[643359.899950] .spread : 0.000000
[643359.899950] .spread0 : 0.000000
[643359.899950] .nr_running : 0
[643359.899950] .load : 0
[643359.899950] .nr_spread_over : 0
[643359.899950] .shares : 0
[643359.899950]
[643359.899950] rt_rq[0]:
[643359.899950] .rt_nr_running : 0
[643359.899950] .rt_throttled : 0
[643359.899950] .rt_time : 0.031685
[643359.899950] .rt_runtime : 950.000000
[643359.899950]
[643359.899950] runnable tasks:
[643359.899950] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[643359.899950] ----------------------------------------------------------------------------------------------------------
[643359.899950]
[643359.899950] cpu#1, 2333.472 MHz
[643359.899950] .nr_running : 0
[643359.899950] .load : 0
[643359.899950] .nr_switches : 33253033
[643359.899950] .nr_load_updates : 90763615
[643359.899950] .nr_uninterruptible : 2
[643359.899950] .next_balance : 4455.732270
[643359.899950] .curr->pid : 0
[643359.899950] .clock : 643361701.017599
[643359.899950] .cpu_load[0] : 0
[643359.899950] .cpu_load[1] : 0
[643359.899950] .cpu_load[2] : 0
[643359.899950] .cpu_load[3] : 0
[643359.899950] .cpu_load[4] : 0
[643359.899950]
[643359.899950] cfs_rq[1]:/
[643359.899950] .exec_clock : 0.000000
[643359.899950] .MIN_vruntime : 0.000001
[643359.899950] .min_vruntime : 123663903.078789
[643359.899950] .max_vruntime : 0.000001
[643359.899950] .spread : 0.000000
[643359.899950] .spread0 : 23676262.368426
[643359.899950] .nr_running : 0
[643359.899950] .load : 0
[643359.899950] .nr_spread_over : 0
[643359.899950] .shares : 0
[643359.899950]
[643359.899950] rt_rq[1]:
[643359.899950] .rt_nr_running : 0
[643359.899950] .rt_throttled : 0
[643359.899950] .rt_time : 0.019934
[643359.899950] .rt_runtime : 950.000000
[643359.899950]
[643359.899950] runnable tasks:
[643359.899950] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[643359.899950] ----------------------------------------------------------------------------------------------------------
[643359.899950]
[643362.975952] SysRq : Show Blocked State
[643362.975968] task PC stack pid father
[643362.975981] kswapd0 D 0000000000000000 0 30 2 0x00000000
[643362.975994] ffffffff814a11f0 0000000000000246 0000000000000000 0000000109951742
[643362.976011] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002dfbdfd8
[643362.976027] 0000000000015780 0000000000015780 ffff88002fe44db0 ffff88002fe450a8
[643362.976043] Call Trace:
[643362.976057] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.976068] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.976079] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643362.976091] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643362.976101] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643362.976112] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643362.976122] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643362.976131] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643362.976142] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643362.976153] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.976164] [<ffffffff810c11a8>] ? kswapd+0x5f9/0x686
[643362.976174] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643362.976183] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.976193] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.976202] [<ffffffff810c0baf>] ? kswapd+0x0/0x686
[643362.976210] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[643362.976220] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[643362.976229] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[643362.976239] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[643362.976248] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[643362.976259] sshd D ffff880002049c40 0 17733 1281 0x00000000
[643362.976271] ffff880002049c40 0000000000000286 ffff88002bfcb668 ffff88002bfcb664
[643362.976287] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002bfcbfd8
[643362.976302] 0000000000015780 0000000000015780 ffff88000256aa60 ffff88000256ad58
[643362.976318] Call Trace:
[643362.976326] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.976336] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.976345] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643362.976355] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643362.976364] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643362.976374] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643362.976384] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643362.976393] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.976403] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[643362.976413] [<ffffffff8100c1a7>] ? xen_mc_flush+0x159/0x185
[643362.976423] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[643362.976433] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[643362.976444] [<ffffffff810492e7>] ? finish_task_switch+0x44/0xaf
[643362.976453] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.976463] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[643362.976473] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[643362.976482] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[643362.976492] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643362.976502] [<ffffffff810c97cd>] ? congestion_wait+0x74/0x80
[643362.976511] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.976521] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643362.976532] [<ffffffff81254a33>] ? sock_alloc_send_pskb+0xdd/0x2f6
[643362.976543] [<ffffffff810e7e3a>] ? kmalloc_large_node+0x5d/0x9b
[643362.976553] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[643362.976562] [<ffffffff81254a33>] ? sock_alloc_send_pskb+0xdd/0x2f6
[643362.976572] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.976582] [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[643362.976592] [<ffffffff812c408e>] ? unix_stream_sendmsg+0x13b/0x2c8
[643362.976603] [<ffffffff812514dc>] ? sock_aio_write+0xb1/0xbc
[643362.976614] [<ffffffff810effee>] ? do_sync_write+0xce/0x113
[643362.976623] [<ffffffff8101079c>] ? __switch_to+0x285/0x297
[643362.976632] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.976642] [<ffffffff810492e7>] ? finish_task_switch+0x44/0xaf
[643362.976652] [<ffffffff8130cd30>] ? thread_return+0x79/0xe0
[643362.976662] [<ffffffff810f0953>] ? vfs_write+0xbc/0x102
[643362.976671] [<ffffffff810f0a55>] ? sys_write+0x45/0x6e
[643362.976680] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643362.976688] apache2 D ffff880002659c40 0 17946 26160 0x00000004
[643362.976700] ffff880002659c40 0000000000000286 ffff88002b8fd348 ffff88002b8fd344
[643362.976716] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002b8fdfd8
[643362.976732] 0000000000015780 0000000000015780 ffff880002049c40 ffff880002049f38
[643362.976747] Call Trace:
[643362.976755] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.976765] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.976774] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643362.976784] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643362.976793] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643362.976803] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643362.976813] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643362.976822] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.976832] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[643362.976842] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[643362.976852] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[643362.976862] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643362.976872] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643362.976881] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[643362.976891] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[643362.976900] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[643362.976910] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[643362.976920] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643362.976930] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643362.976940] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[643362.976949] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[643362.976960] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.976969] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.976979] [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[643362.976988] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.976998] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[643362.977008] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643362.977018] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.977028] [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[643362.977037] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643362.977047] [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[643362.977056] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.977066] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.977075] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643362.977084] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.977094] [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643362.977104] [<ffffffff81193442>] ? radix_tree_gang_lookup_slot+0x6a/0x8e
[643362.977114] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[643362.977123] [<ffffffff8110d00c>] ? __generic_file_splice_read+0x36e/0x3bd
[643362.977133] [<ffffffff81251f64>] ? kernel_sendmsg+0x32/0x3f
[643362.977142] [<ffffffff81254790>] ? sock_no_sendpage+0x87/0x94
[643362.977152] [<ffffffff812909f1>] ? tcp_sendpage+0x4b/0x458
[643362.977161] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643362.977170] [<ffffffff81250d0d>] ? kernel_sendpage+0x16/0x1f
[643362.977180] [<ffffffff81250d4b>] ? sock_sendpage+0x35/0x3a
[643362.977189] [<ffffffff8110b122>] ? pipe_to_sendpage+0x66/0x74
[643362.977198] [<ffffffff8110b17c>] ? splice_from_pipe_feed+0x4c/0xd4
[643362.977207] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643362.977216] [<ffffffff8110b405>] ? __splice_from_pipe+0x3e/0x69
[643362.979942] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[643362.979942] [<ffffffff8110b47d>] ? splice_from_pipe+0x4d/0x63
[643362.979942] [<ffffffff8110c00d>] ? direct_splice_actor+0x1b/0x1e
[643362.979942] [<ffffffff8110b9c8>] ? splice_direct_to_actor+0xdc/0x188
[643362.979942] [<ffffffff81041432>] ? set_next_entity+0x34/0x56
[643362.979942] [<ffffffff8110bff2>] ? direct_splice_actor+0x0/0x1e
[643362.979942] [<ffffffff8110babe>] ? do_splice_direct+0x4a/0x64
[643362.979942] [<ffffffff810f0312>] ? do_sendfile+0x12d/0x1a8
[643362.979942] [<ffffffff810f03d6>] ? sys_sendfile64+0x49/0x89
[643362.979942] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643362.979942] vsftpd D 0000000000000000 0 18173 18168 0x00000000
[643362.979942] ffffffff814a11f0 0000000000000286 0000000000000000 000000010995173d
[643362.979942] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002becdfd8
[643362.979942] 0000000000015780 0000000000015780 ffff88000265f810 ffff88000265fb08
[643362.979942] Call Trace:
[643362.979942] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643362.979942] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643362.979942] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643362.979942] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643362.979942] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643362.979942] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643362.979942] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.979942] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[643362.979942] [<ffffffff8100ec99>] ? xen_irq_enable_direct_end+0x0/0x7
[643362.979942] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[643362.979942] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[643362.979942] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[643362.979942] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[643362.979942] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[643362.979942] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643362.979942] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643362.979942] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[643362.979942] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[643362.979942] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[643362.979942] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[643362.979942] [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0
[643362.979942] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[643362.979942] [<ffffffff810fb1a6>] ? user_path_at+0x20/0x79
[643362.979942] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942] [<ffffffff810ef186>] ? sys_chdir+0x1b/0x64
[643362.979942] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643362.979942] vsftpd D 0000000000000000 0 18176 1248 0x00000000
[643362.979942] ffffffff814a11f0 0000000000000246 0000000000000000 ffffffff8100e635
[643362.979942] ffffffff8100ecf2 ffff8800020c8000 000000000000f9e0 ffff88002ac33fd8
[643362.979942] 0000000000015780 0000000000015780 ffff880002659c40 ffff880002659f38
[643362.979942] Call Trace:
[643362.979942] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942] [<ffffffff8130d0ed>] ? schedule_timeout+0x2e/0xdd
[643362.979942] [<ffffffff8130e002>] ? _spin_lock_irq+0x7/0x22
[643362.979942] [<ffffffff8130e002>] ? _spin_lock_irq+0x7/0x22
[643362.979942] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942] [<ffffffff8130e002>] ? _spin_lock_irq+0x7/0x22
[643362.979942] [<ffffffff8130cfa4>] ? wait_for_common+0xde/0x15b
[643362.979942] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942] [<ffffffff8104b4bc>] ? default_wake_function+0x0/0x9
[643362.979942] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643362.979942] [<ffffffff8106417b>] ? synchronize_sched+0x4c/0x52
[643362.979942] [<ffffffff81064181>] ? wakeme_after_rcu+0x0/0xb
[643362.979942] [<ffffffff8106943e>] ? switch_task_namespaces+0x32/0x43
[643362.979942] [<ffffffff81052708>] ? do_exit+0x549/0x6c9
[643362.979942] [<ffffffff810528fe>] ? do_group_exit+0x76/0x9d
[643362.979942] [<ffffffff81052937>] ? sys_exit_group+0x12/0x16
[643362.979942] [<ffffffff8130e67a>] ? error_exit+0x2a/0x60
[643362.979942] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643362.979942] vsftpd D ffff88000256aa60 0 18197 18192 0x00000000
[643362.979942] ffff88000256aa60 0000000000000282 0000000000000000 0000000109951742
[643362.979942] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002aaf9fd8
[643362.979942] 0000000000015780 0000000000015780 ffff88000265aa60 ffff88000265ad58
[643362.979942] Call Trace:
[643362.979942] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643362.979942] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643362.979942] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[643362.979942] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[643362.979942] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[643362.979942] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[643362.979942] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[643362.979942] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.979942] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[643362.979942] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[643362.979942] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942] [<ffffffffa00bb1ef>] ? tcp_packet+0xb20/0xb65 [nf_conntrack]
[643362.979942] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[643362.979942] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[643362.979942] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[643362.979942] [<ffffffffa01aface>] ? ipt_do_table+0x5ee/0x620 [ip_tables]
[643362.979942] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[643362.979942] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[643362.979942] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[643362.979942] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[643362.979942] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[643362.979942] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[643362.979942] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[643362.979942] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.979942] [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[643362.979942] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.979942] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[643362.979942] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[643362.979942] [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[643362.979942] [<ffffffff812514dc>] ? sock_aio_write+0xb1/0xbc
[643362.979942] [<ffffffff8100c426>] ? pfn_to_mfn+0xe/0x22
[643362.979942] [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
[643362.979942] [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3
[643362.979942] [<ffffffff810effee>] ? do_sync_write+0xce/0x113
[643362.979942] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[643362.979942] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942] [<ffffffff810f0953>] ? vfs_write+0xbc/0x102
[643362.979942] [<ffffffff810f0a55>] ? sys_write+0x45/0x6e
[643362.979942] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[643362.979942] exim4 D 0000000000000000 0 18198 18194 0x00000000
[643362.979942] ffffffff814a11f0 0000000000000282 0000000000000000 00000000ae9639ab
[643362.979942] ffffea00003599f8 ffffea00003599f8 000000000000f9e0 ffff88002b463fd8
[643362.979942] 0000000000015780 0000000000015780 ffff88000265cdb0 ffff88000265d0a8
[643362.979942] Call Trace:
[643362.979942] [<ffffffff8106cbaa>] ? timekeeping_get_ns+0xe/0x2e
[643362.979942] [<ffffffff810b54f9>] ? sync_page+0x0/0x46
[643362.979942] [<ffffffff8130ce0a>] ? io_schedule+0x73/0xb7
[643362.979942] [<ffffffff810b553a>] ? sync_page+0x41/0x46
[643362.979942] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643362.979942] [<ffffffff8130d21a>] ? __wait_on_bit_lock+0x3f/0x84
[643362.979942] [<ffffffff810b54c6>] ? __lock_page+0x5d/0x63
[643362.979942] [<ffffffff810660c0>] ? wake_bit_function+0x0/0x23
[643362.979942] [<ffffffff810b54d5>] ? lock_page+0x9/0x1f
[643362.979942] [<ffffffff810b55eb>] ? find_lock_page+0x25/0x45
[643362.979942] [<ffffffff810b5bfb>] ? filemap_fault+0x1a5/0x2f6
[643362.979942] [<ffffffff810cbbb6>] ? __do_fault+0x54/0x3c3
[643362.979942] [<ffffffff811927a9>] ? prio_tree_insert+0x185/0x20d
[643362.979942] [<ffffffff8105337d>] ? current_fs_time+0x1e/0x24
[643362.979942] [<ffffffff8100c3a5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[643362.979942] [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643362.979942] [<ffffffff810d1513>] ? get_unmapped_area+0xd7/0x139
[643362.979942] [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[643362.979942] [<ffffffff8130e445>] ? page_fault+0x25/0x30
[643362.979942] vsftpd D 0000000000000000 0 18199 1248 0x00000000
[643362.979942] ffffffff814a11f0 0000000000000282 0000000000000000 ffff88002df9d780
[643362.979942] 0000000000000000 0000000000000000 000000000000f9e0 ffff88002ac41fd8
[643362.979942] 0000000000015780 0000000000015780 ffff88000265b170 ffff88000265b468
[643362.979942] Call Trace:
[643362.979942] [<ffffffff8106cbaa>] ? timekeeping_get_ns+0xe/0x2e
[643362.979942] [<ffffffff8110f74e>] ? sync_buffer+0x0/0x40
[643362.979942] [<ffffffff8130ce0a>] ? io_schedule+0x73/0xb7
[643362.979942] [<ffffffff8110f789>] ? sync_buffer+0x3b/0x40
[643362.979942] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[643362.979942] [<ffffffff8130d317>] ? __wait_on_bit+0x41/0x70
[643362.979942] [<ffffffff8110f74e>] ? sync_buffer+0x0/0x40
[643362.979942] [<ffffffff8130d3b1>] ? out_of_line_wait_on_bit+0x6b/0x77
[643362.979942] [<ffffffff810660c0>] ? wake_bit_function+0x0/0x23
[643362.979942] [<ffffffff81111395>] ? __bread+0x4c/0x63
[643362.979942] [<ffffffffa0032729>] ? ext3_get_branch+0x67/0xdb [ext3]
[643362.979942] [<ffffffffa0033001>] ? ext3_get_blocks_handle+0x9e/0x86e [ext3]
[643362.979942] [<ffffffff8118d18a>] ? cfq_dispatch_requests+0x108/0x488
[643362.979942] [<ffffffffa000a751>] ? do_blkif_request+0x316/0x374 [xen_blkfront]
[643362.979942] [<ffffffffa0033885>] ? ext3_get_block+0xb4/0xf1 [ext3]
[643362.979942] [<ffffffff81115f00>] ? do_mpage_readpage+0x161/0x421
[643362.979942] [<ffffffffa00337d1>] ? ext3_get_block+0x0/0xf1 [ext3]
[643362.979942] [<ffffffff81193dd5>] ? radix_tree_insert+0x16a/0x1a8
[643362.979942] [<ffffffffa00337d1>] ? ext3_get_block+0x0/0xf1 [ext3]
[643362.979942] [<ffffffff810b57e1>] ? add_to_page_cache_locked+0x98/0xc1
[643362.979942] [<ffffffffa00337d1>] ? ext3_get_block+0x0/0xf1 [ext3]
[643362.979942] [<ffffffff811162fe>] ? mpage_readpages+0xcc/0x112
[643362.979942] [<ffffffffa00337d1>] ? ext3_get_block+0x0/0xf1 [ext3]
[643362.979942] [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[643362.979942] [<ffffffff810bce95>] ? __do_page_cache_readahead+0x11b/0x1b4
[643362.979942] [<ffffffff810bcf4a>] ? ra_submit+0x1c/0x20
[643362.979942] [<ffffffff810b5bd3>] ? filemap_fault+0x17d/0x2f6
[643362.979942] [<ffffffff810cbbb6>] ? __do_fault+0x54/0x3c3
[643362.979942] [<ffffffff8100b444>] ? xen_write_msr_safe+0x76/0xb1
[643362.979942] [<ffffffff8100c3a5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[643362.979942] [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[643362.979942] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[643362.979942] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[643362.979942] [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[643362.979942] [<ffffffff8130e445>] ? page_fault+0x25/0x30
[643362.979942] Sched Debug Version: v0.09, 2.6.32-5-xen-amd64 #1
[643362.979942] now at 643363376.608710 msecs
[643362.979942] .jiffies : 4455733039
[643362.979942] .sysctl_sched_latency : 10.000000
[643362.979942] .sysctl_sched_min_granularity : 2.000000
[643362.979942] .sysctl_sched_wakeup_granularity : 2.000000
[643362.979942] .sysctl_sched_child_runs_first : 0.000000
[643362.979942] .sysctl_sched_features : 32611451
[643362.979942]
[643362.979942] cpu#0, 2333.472 MHz
[643362.979942] .nr_running : 0
[643362.979942] .load : 0
[643362.979942] .nr_switches : 174787620
[643362.979942] .nr_load_updates : 120151955
[643362.979942] .nr_uninterruptible : 8
[643362.979942] .next_balance : 4455.733040
[643362.979942] .curr->pid : 0
[643362.979942] .clock : 643362973.080469
[643362.979942] .cpu_load[0] : 0
[643362.979942] .cpu_load[1] : 0
[643362.979942] .cpu_load[2] : 0
[643362.979942] .cpu_load[3] : 0
[643362.979942] .cpu_load[4] : 5
[643362.979942]
[643362.979942] cfs_rq[0]:/
[643362.979942] .exec_clock : 0.000000
[643362.979942] .MIN_vruntime : 0.000001
[643362.979942] .min_vruntime : 99987741.798157
[643362.979942] .max_vruntime : 0.000001
[643362.979942] .spread : 0.000000
[643362.979942] .spread0 : 0.000000
[643362.979942] .nr_running : 0
[643362.979942] .load : 0
[643362.979942] .nr_spread_over : 0
[643362.979942] .shares : 0
[643362.979942]
[643362.979942] rt_rq[0]:
[643362.979942] .rt_nr_running : 0
[643362.979942] .rt_throttled : 0
[643362.979942] .rt_time : 0.000000
[643362.979942] .rt_runtime : 950.000000
[643362.979942]
[643362.979942] runnable tasks:
[643362.979942] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[643362.979942] ----------------------------------------------------------------------------------------------------------
[643362.979942]
[643362.979942] cpu#1, 2333.472 MHz
[643362.979942] .nr_running : 0
[643362.979942] .load : 0
[643362.979942] .nr_switches : 33253903
[643362.979942] .nr_load_updates : 90763886
[643362.979942] .nr_uninterruptible : 0
[643362.979942] .next_balance : 4455.733077
[643362.979942] .curr->pid : 0
[643362.979942] .clock : 643363373.011868
[643362.979942] .cpu_load[0] : 0
[643362.979942] .cpu_load[1] : 0
[643362.979942] .cpu_load[2] : 0
[643362.979942] .cpu_load[3] : 0
[643362.979942] .cpu_load[4] : 0
[643362.979942]
[643362.979942] cfs_rq[1]:/
[643362.979942] .exec_clock : 0.000000
[643362.979942] .MIN_vruntime : 0.000001
[643362.979942] .min_vruntime : 123663966.605916
[643362.979942] .max_vruntime : 0.000001
[643362.979942] .spread : 0.000000
[643362.979942] .spread0 : 23676224.807759
[643362.979942] .nr_running : 0
[643362.979942] .load : 0
[643362.979942] .nr_spread_over : 0
[643362.979942] .shares : 0
[643362.979942]
[643362.979942] rt_rq[1]:
[643362.979942] .rt_nr_running : 0
[643362.979942] .rt_throttled : 0
[643362.979942] .rt_time : 0.000000
[643362.979942] .rt_runtime : 950.000000
[643362.979942]
[643362.979942] runnable tasks:
[643362.979942] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[643362.979942] ----------------------------------------------------------------------------------------------------------
[643362.979942]
Please help. TIA.
--
2. That which causes joy or happiness.
----- End forwarded message -----
----- Forwarded message from Josip Rodin <joy@debbugs.entuzijast.net> -----
Date: Fri, 15 Feb 2013 08:56:20 +0100
From: Josip Rodin <joy@debbugs.entuzijast.net>
To: 519586@bugs.debian.org
Cc: 519586-submitter@bugs.debian.org
Subject: Re: Huge "Slab Unreclaimable" and continually growing
On Tue, Jan 22, 2013 at 10:59:17AM +0100, Josip Rodin wrote:
> I appear to be experiencing a serious problem with a 768 MB RAM Xen domU
> machine running an NFS client - every now and then (for months now), often
> in the middle of the night, it enters some kind of a broken state where a
> few semi-random processes (mainly apache2's and vsftpd's which are told to
> serve files from the NFS mount) start battling it out for the memory, and
> everything including sshd starts invoking the OOM killer, over and over
> again. Nothing seems to halt the downward spiral; manual invocation of the
> OOM killer does nothing of any use. Terminating all processes is the only
> thing that makes it go quiet, but then that's effectively the same as a
> reboot.
>
> This is the SysRq+M output on the machine once it's been in the broken state
> for a while:
[...]
> active_anon:394 inactive_anon:3197 isolated_anon:0
> active_file:25 inactive_file:176 isolated_file:32
> unevictable:2659 dirty:1 writeback:0 unstable:0
> free:21456 slab_reclaimable:16177 slab_unreclaimable:143165
> mapped:677 shmem:76 pagetables:455 bounce:0
[...]
> The thing I noticed was the slab_unreclaimable explosion, by a factor
> of 122. That... doesn't sound like something that should be happenning.
>
> Googling for slab_unreclaimable found me this old bug report about
> slab_unreclaimable domU problems that was mass-closed with the switch to the
> new paravirtops Xen release. Granted, our use case is not Samba like with
> the original reporter, but the pattern of a file server was close enough for
> me to be uncomfortable with it :|
I caught it earlier just now, at:
[950084.590733] active_anon:2805 inactive_anon:11835 isolated_anon:0
[950084.590735] active_file:76 inactive_file:516 isolated_file:32
[950084.590737] unevictable:783 dirty:1 writeback:0 unstable:0
[950084.590739] free:26251 slab_reclaimable:15733 slab_unreclaimable:128868
[950084.590741] mapped:938 shmem:75 pagetables:651 bounce:0
And snuck in a few slabtops (even some -o invocations were getting killed,
along with my shell and pretty much everything else):
Active / Total Objects (% used) : 555753 / 587128 (94.7%)
Active / Total Slabs (% used) : 49430 / 49430 (100.0%)
Active / Total Caches (% used) : 65 / 76 (85.5%)
Active / Total Size (% used) : 546613.78K / 553025.01K (98.8%)
Minimum / Average / Maximum Object : 0.01K / 0.94K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
90993 66836 73% 0.19K 4333 21 17332K dentry
75840 73664 97% 0.12K 2370 32 9480K kmalloc-128
68096 68092 99% 0.01K 133 512 532K kmalloc-8
65888 65655 99% 0.25K 4118 16 16472K kmalloc-256
65820 65778 99% 1.00K 4767 16 76272K kmalloc-1024
65436 65414 99% 0.63K 5454 12 43632K proc_inode_cache
65419 65419 100% 4.00K 14179 8 453728K kmalloc-4096
65390 65390 100% 2.06K 13338 15 426816K net_namespace
4998 4990 99% 0.08K 98 51 392K sysfs_dir_cache
4224 2018 47% 0.06K 66 64 264K kmalloc-64
2288 2107 92% 0.18K 104 22 416K vm_area_struct
1792 1789 99% 0.02K 7 256 28K kmalloc-16
1470 1203 81% 0.19K 70 21 280K kmalloc-192
1300 402 30% 0.79K 65 20 1040K ext3_inode_cache
896 731 81% 0.03K 7 128 28K anon_vma
784 532 67% 0.55K 56 14 448K radix_tree_node
A bit later:
Active / Total Objects (% used) : 555403 / 586704 (94.7%)
Active / Total Slabs (% used) : 49394 / 49394 (100.0%)
Active / Total Caches (% used) : 65 / 76 (85.5%)
Active / Total Size (% used) : 546552.82K / 552827.43K (98.9%)
Minimum / Average / Maximum Object : 0.01K / 0.94K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
90993 66779 73% 0.19K 4333 21 17332K dentry
75840 73654 97% 0.12K 2370 32 9480K kmalloc-128
68096 68092 99% 0.01K 133 512 532K kmalloc-8
65888 65601 99% 0.25K 4118 16 16472K kmalloc-256
65852 65741 99% 1.00K 4760 16 76160K kmalloc-1024
65436 65409 99% 0.63K 5454 12 43632K proc_inode_cache
65428 65428 100% 4.00K 14181 8 453792K kmalloc-4096
65391 65391 100% 2.06K 13339 15 426848K net_namespace
4998 4986 99% 0.08K 98 51 392K sysfs_dir_cache
4224 2017 47% 0.06K 66 64 264K kmalloc-64
2134 2108 98% 0.18K 97 22 388K vm_area_struct
1792 1789 99% 0.02K 7 256 28K kmalloc-16
1449 1078 74% 0.19K 69 21 276K kmalloc-192
1100 376 34% 0.79K 55 20 880K ext3_inode_cache
896 639 71% 0.03K 7 128 28K anon_vma
714 554 77% 0.55K 51 14 408K radix_tree_node
And a bit later still:
[950506.398951] active_anon:2370 inactive_anon:12180 isolated_anon:2
[950506.398953] active_file:0 inactive_file:261 isolated_file:36
[950506.398955] unevictable:831 dirty:1 writeback:0 unstable:0
[950506.398957] free:26418 slab_reclaimable:15749 slab_unreclaimable:129161
[950506.398959] mapped:743 shmem:75 pagetables:636 bounce:0
Active / Total Objects (% used) : 556917 / 587606 (94.8%)
Active / Total Slabs (% used) : 49717 / 49717 (100.0%)
Active / Total Caches (% used) : 65 / 76 (85.5%)
Active / Total Size (% used) : 548047.95K / 554096.57K (98.9%)
Minimum / Average / Maximum Object : 0.01K / 0.94K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
90993 66921 73% 0.19K 4333 21 17332K dentry
75840 73825 97% 0.12K 2370 32 9480K kmalloc-128
68096 68092 99% 0.01K 133 512 532K kmalloc-8
65984 65766 99% 0.25K 4124 16 16496K kmalloc-256
65964 65913 99% 1.00K 4776 16 76416K kmalloc-1024
65592 65566 99% 0.63K 5467 12 43736K proc_inode_cache
65584 65584 100% 4.00K 14288 8 457216K kmalloc-4096
65552 65552 100% 2.06K 13500 15 432000K net_namespace
4998 4986 99% 0.08K 98 51 392K sysfs_dir_cache
4224 1976 46% 0.06K 66 64 264K kmalloc-64
2398 2398 100% 0.18K 109 22 436K vm_area_struct
1792 1789 99% 0.02K 7 256 28K kmalloc-16
1428 1112 77% 0.19K 68 21 272K kmalloc-192
1100 402 36% 0.79K 55 20 880K ext3_inode_cache
896 648 72% 0.03K 7 128 28K anon_vma
742 664 89% 0.55K 53 14 424K radix_tree_node
[950532.846913] SysRq : Show Blocked State
[950532.846930] task PC stack pid father
[950532.846942] kswapd0 D ffff88002f123880 0 30 2 0x00000000
[950532.846955] ffff88002f123880 0000000000000246 ffff88001ae8cbd0 000000010e28d9ca
[950532.846971] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002dfb1fd8
[950532.846987] 0000000000015780 0000000000015780 ffff88002fe44db0 ffff88002fe450a8
[950532.847003] Call Trace:
[950532.847018] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.847028] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.847037] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.847049] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.847061] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.847071] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.847081] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[950532.847091] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.847100] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.847110] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.847121] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.847131] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.847142] [<ffffffff810c11a8>] ? kswapd+0x5f9/0x686
[950532.847151] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.847160] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.847170] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.847179] [<ffffffff810c0baf>] ? kswapd+0x0/0x686
[950532.847187] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[950532.847197] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[950532.847205] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[950532.847214] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[950532.847223] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[950532.847234] vsftpd D 0000000000000000 0 1325 1 0x00000000
[950532.847245] ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9ca
[950532.847261] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff880002697fd8
[950532.847276] 0000000000015780 0000000000015780 ffff88002f123880 ffff88002f123b78
[950532.847291] Call Trace:
[950532.847299] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.847308] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.847317] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.847327] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.847336] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.848079] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.848088] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.848098] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.848108] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.848117] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.848127] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.848137] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.848146] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.848156] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.848165] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.848175] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.848187] [<ffffffff81154851>] ? cap_sb_alloc_security+0x0/0x3
[950532.848197] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.848207] [<ffffffff810e8769>] ? kmem_cache_alloc+0x8c/0xf0
[950532.848216] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.848224] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.848235] [<ffffffff8108bc1e>] ? copy_pid_ns+0x5f/0x2a5
[950532.848244] [<ffffffff8108bc1e>] ? copy_pid_ns+0x5f/0x2a5
[950532.848253] [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0
[950532.848262] [<ffffffff8108bc1e>] ? copy_pid_ns+0x5f/0x2a5
[950532.848271] [<ffffffff810692fa>] ? create_new_namespaces+0xc2/0x15e
[950532.848281] [<ffffffff8106954e>] ? copy_namespaces+0x61/0x97
[950532.848291] [<ffffffff8104df25>] ? copy_process+0xa7a/0x1165
[950532.848300] [<ffffffff8104e767>] ? do_fork+0x157/0x31e
[950532.848309] [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[950532.848318] [<ffffffff81011e63>] ? stub_clone+0x13/0x20
[950532.848327] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.848336] vsftpd D 0000000000000000 0 28790 28778 0x00000000
[950532.848348] ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9d4
[950532.848363] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff8800279affd8
[950532.848379] 0000000000015780 0000000000015780 ffff88002f15c6a0 ffff88002f15c998
[950532.848395] Call Trace:
[950532.848402] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.848412] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.848421] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.848431] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.848441] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.848450] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.848460] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.848470] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.848479] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.848488] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.848498] [<ffffffff810bfd54>] ? shrink_list+0xf8/0x767
[950532.848509] [<ffffffff8119934b>] ? __bitmap_weight+0x3a/0x7e
[950532.848519] [<ffffffff8118f7fe>] ? cpumask_next_and+0x2a/0x3a
[950532.848529] [<ffffffff81045e7e>] ? find_busiest_group+0x420/0xa4e
[950532.848539] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.848548] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.848557] [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20
[950532.848567] [<ffffffff8130e67a>] ? error_exit+0x2a/0x60
[950532.848575] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[950532.848585] [<ffffffff810bc5c9>] ? throttle_vm_writeout+0x30/0x8d
[950532.848595] [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
[950532.848604] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.848614] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.848623] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.848633] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.848643] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.848653] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.848662] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.848671] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.848681] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.848690] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.848699] [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0
[950532.848708] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.848717] [<ffffffff810fb1a6>] ? user_path_at+0x20/0x79
[950532.848726] [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[950532.848735] [<ffffffff810ef186>] ? sys_chdir+0x1b/0x64
[950532.848744] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.848753] apache2 D 0000000000000000 0 30698 18611 0x00000000
[950532.848765] ffff88002fdd0e20 0000000000000282 0000000000000000 000000010e28d9c8
[950532.848781] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88000222ffd8
[950532.848796] 0000000000015780 0000000000015780 ffff880002699530 ffff880002699828
[950532.848812] Call Trace:
[950532.848819] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.848829] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.848838] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.848848] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.848857] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.848867] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.848876] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.848886] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.848895] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.848905] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.848915] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.848925] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.848935] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.848944] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.848954] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.848964] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.848975] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.848985] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.848995] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.849004] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.849014] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.849024] [<ffffffff8129835d>] ? tcp_established_options+0x2d/0xa9
[950532.849033] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.849041] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.849051] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.849061] [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[950532.849070] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.849081] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[950532.849090] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.849099] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.849109] [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[950532.849118] [<ffffffff81041432>] ? set_next_entity+0x34/0x56
[950532.849127] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.849138] [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[950532.849146] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.849156] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.849165] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.849174] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.849183] [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[950532.849192] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[950532.849202] [<ffffffff8110d00c>] ? __generic_file_splice_read+0x36e/0x3bd
[950532.849211] [<ffffffff81251f64>] ? kernel_sendmsg+0x32/0x3f
[950532.849220] [<ffffffff81254790>] ? sock_no_sendpage+0x87/0x94
[950532.849230] [<ffffffff812909f1>] ? tcp_sendpage+0x4b/0x458
[950532.849239] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[950532.849248] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff81250d0d>] ? kernel_sendpage+0x16/0x1f
[950532.850903] [<ffffffff81250d4b>] ? sock_sendpage+0x35/0x3a
[950532.850903] [<ffffffff8110b122>] ? pipe_to_sendpage+0x66/0x74
[950532.850903] [<ffffffff8110b17c>] ? splice_from_pipe_feed+0x4c/0xd4
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff8110b405>] ? __splice_from_pipe+0x3e/0x69
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff8110b47d>] ? splice_from_pipe+0x4d/0x63
[950532.850903] [<ffffffff8110c00d>] ? direct_splice_actor+0x1b/0x1e
[950532.850903] [<ffffffff8110b9c8>] ? splice_direct_to_actor+0xdc/0x188
[950532.850903] [<ffffffff8110bff2>] ? direct_splice_actor+0x0/0x1e
[950532.850903] [<ffffffff8110babe>] ? do_splice_direct+0x4a/0x64
[950532.850903] [<ffffffff810f0312>] ? do_sendfile+0x12d/0x1a8
[950532.850903] [<ffffffff810f03d6>] ? sys_sendfile64+0x49/0x89
[950532.850903] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.850903] apache2 D 0000000000000000 0 30702 18611 0x00000000
[950532.850903] ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9c8
[950532.850903] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff880023c6bfd8
[950532.850903] 0000000000015780 0000000000015780 ffff880002503880 ffff880002503b78
[950532.850903] Call Trace:
[950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.850903] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.850903] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.850903] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.850903] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.850903] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.850903] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.850903] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.850903] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.850903] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.850903] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.850903] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.850903] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.850903] [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
[950532.850903] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.850903] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903] [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903] [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903] [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[950532.850903] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[950532.850903] [<ffffffff8110d00c>] ? __generic_file_splice_read+0x36e/0x3bd
[950532.850903] [<ffffffff81251f64>] ? kernel_sendmsg+0x32/0x3f
[950532.850903] [<ffffffff81254790>] ? sock_no_sendpage+0x87/0x94
[950532.850903] [<ffffffff812909f1>] ? tcp_sendpage+0x4b/0x458
[950532.850903] [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff81250d0d>] ? kernel_sendpage+0x16/0x1f
[950532.850903] [<ffffffff81250d4b>] ? sock_sendpage+0x35/0x3a
[950532.850903] [<ffffffff8110b122>] ? pipe_to_sendpage+0x66/0x74
[950532.850903] [<ffffffff8110b17c>] ? splice_from_pipe_feed+0x4c/0xd4
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff8110b405>] ? __splice_from_pipe+0x3e/0x69
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff8110b47d>] ? splice_from_pipe+0x4d/0x63
[950532.850903] [<ffffffff8110c00d>] ? direct_splice_actor+0x1b/0x1e
[950532.850903] [<ffffffff8110b9c8>] ? splice_direct_to_actor+0xdc/0x188
[950532.850903] [<ffffffff8110bff2>] ? direct_splice_actor+0x0/0x1e
[950532.850903] [<ffffffff8110babe>] ? do_splice_direct+0x4a/0x64
[950532.850903] [<ffffffff810f0312>] ? do_sendfile+0x12d/0x1a8
[950532.850903] [<ffffffff810f03d6>] ? sys_sendfile64+0x49/0x89
[950532.850903] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.850903] vsftpd D 0000000000000000 0 32054 32049 0x00000000
[950532.850903] ffffffff814a11f0 0000000000000286 0000000000000000 000000010e28d9dd
[950532.850903] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff8800241e3fd8
[950532.850903] 0000000000015780 0000000000015780 ffff880002698710 ffff880002698a08
[950532.850903] Call Trace:
[950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.850903] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.850903] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.850903] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.850903] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.850903] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903] [<ffffffffa00bb1ef>] ? tcp_packet+0xb20/0xb65 [nf_conntrack]
[950532.850903] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.850903] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.850903] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.850903] [<ffffffffa01aface>] ? ipt_do_table+0x5ee/0x620 [ip_tables]
[950532.850903] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.850903] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.850903] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.850903] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.850903] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.850903] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.850903] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903] [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903] [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[950532.850903] [<ffffffff812514dc>] ? sock_aio_write+0xb1/0xbc
[950532.850903] [<ffffffff8100c426>] ? pfn_to_mfn+0xe/0x22
[950532.850903] [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
[950532.850903] [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3
[950532.850903] [<ffffffff810effee>] ? do_sync_write+0xce/0x113
[950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903] [<ffffffff810f0953>] ? vfs_write+0xbc/0x102
[950532.850903] [<ffffffff810f0a55>] ? sys_write+0x45/0x6e
[950532.850903] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.850903] update-pkg-ft D 0000000000000000 0 32082 32038 0x00000000
[950532.850903] ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9d2
[950532.850903] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff880024f07fd8
[950532.850903] 0000000000015780 0000000000015780 ffff880002798000 ffff8800027982f8
[950532.850903] Call Trace:
[950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.850903] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.850903] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.850903] [<ffffffffa0038d2f>] ? bdev_try_to_free_page+0x0/0x64 [ext3]
[950532.850903] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.850903] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.850903] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.850903] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.850903] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.850903] [<ffffffff810993bc>] ? __call_rcu+0x110/0x118
[950532.850903] [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20
[950532.850903] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.850903] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.850903] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.850903] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.850903] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.850903] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.850903] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.850903] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.850903] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.850903] [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0
[950532.850903] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.850903] [<ffffffff810fb1a6>] ? user_path_at+0x20/0x79
[950532.850903] [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903] [<ffffffff810f361a>] ? vfs_fstatat+0x2c/0x57
[950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903] [<ffffffff810f370d>] ? sys_newstat+0x11/0x30
[950532.850903] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.850903] [<ffffffff8100b4d3>] ? xen_mc_issue+0x21/0x23
[950532.850903] [<ffffffff8101295b>] ? device_not_available+0x1b/0x20
[950532.850903] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.850903] Sched Debug Version: v0.09, 2.6.32-5-xen-amd64 #1
[950532.850903] now at 950533247.852699 msecs
[950532.850903] .jiffies : 4532525608
[950532.850903] .sysctl_sched_latency : 10.000000
[950532.850903] .sysctl_sched_min_granularity : 2.000000
[950532.850903] .sysctl_sched_wakeup_granularity : 2.000000
[950532.850903] .sysctl_sched_child_runs_first : 0.000000
[950532.850903] .sysctl_sched_features : 32611451
[950532.850903]
[950532.850903] cpu#0, 2333.472 MHz
[950532.850903] .nr_running : 0
[950532.850903] .load : 0
[950532.850903] .nr_switches : 238671752
[950532.850903] .nr_load_updates : 165114440
[950532.850903] .nr_uninterruptible : 6
[950532.850903] .next_balance : 4532.525542
[950532.850903] .curr->pid : 0
[950532.850903] .clock : 950532844.214897
[950532.850903] .cpu_load[0] : 0
[950532.850903] .cpu_load[1] : 192
[950532.850903] .cpu_load[2] : 254
[950532.850903] .cpu_load[3] : 197
[950532.850903] .cpu_load[4] : 130
[950532.850903]
[950532.850903] cfs_rq[0]:/
[950532.850903] .exec_clock : 0.000000
[950532.850903] .MIN_vruntime : 0.000001
[950532.850903] .min_vruntime : 136491473.007815
[950532.850903] .max_vruntime : 0.000001
[950532.850903] .spread : 0.000000
[950532.850903] .spread0 : 0.000000
[950532.850903] .nr_running : 0
[950532.850903] .load : 0
[950532.850903] .nr_spread_over : 0
[950532.850903] .shares : 0
[950532.850903]
[950532.850903] rt_rq[0]:
[950532.850903] .rt_nr_running : 0
[950532.850903] .rt_throttled : 0
[950532.850903] .rt_time : 0.000000
[950532.850903] .rt_runtime : 950.000000
[950532.850903]
[950532.850903] runnable tasks:
[950532.850903] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[950532.850903] ----------------------------------------------------------------------------------------------------------
[950532.850903]
[950532.850903] cpu#1, 2333.472 MHz
[950532.850903] .nr_running : 3
[950532.850903] .load : 3072
[950532.850903] .nr_switches : 45263996
[950532.850903] .nr_load_updates : 121842283
[950532.850903] .nr_uninterruptible : 0
[950532.850903] .next_balance : 4532.525625
[950532.850903] .curr->pid : 32083
[950532.850903] .clock : 950533245.009907
[950532.850903] .cpu_load[0] : 3072
[950532.850903] .cpu_load[1] : 3072
[950532.850903] .cpu_load[2] : 3072
[950532.850903] .cpu_load[3] : 3072
[950532.850903] .cpu_load[4] : 3048
[950532.850903]
[950532.850903] cfs_rq[1]:/
[950532.850903] .exec_clock : 0.000000
[950532.850903] .MIN_vruntime : 161709543.323397
[950532.850903] .min_vruntime : 161709548.323397
[950532.850903] .max_vruntime : 161709543.323397
[950532.850903] .spread : 0.000000
[950532.850903] .spread0 : 25218075.315582
[950532.850903] .nr_running : 3
[950532.850903] .load : 3072
[950532.850903] .nr_spread_over : 0
[950532.850903] .shares : 0
[950532.850903]
[950532.850903] rt_rq[1]:
[950532.850903] .rt_nr_running : 0
[950532.850903] .rt_throttled : 0
[950532.850903] .rt_time : 0.000000
[950532.850903] .rt_runtime : 950.000000
[950532.850903]
[950532.850903] runnable tasks:
[950532.850903] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[950532.850903] ----------------------------------------------------------------------------------------------------------
[950532.850903] events/1 10 161709543.323397 1041646 120 0 0 0.000000 0.000000 0.000000 /
[950532.850903] apache2 30698 161709543.323397 3096 120 0 0 0.000000 0.000000 0.000000 /
[950532.850903] R network.sh 32083 161709832.320247 2 120 0 0 0.000000 0.000000 0.000000 /
[950532.850903]
--
2. That which causes joy or happiness.
----- End forwarded message -----
----- Forwarded message from Josip Rodin <joy@debbugs.entuzijast.net> -----
Date: Sat, 16 Feb 2013 22:37:05 +0100
From: Josip Rodin <joy@debbugs.entuzijast.net>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: 519586@bugs.debian.org
Subject: Re: Bug#519586: Huge "Slab Unreclaimable" and continually growing
On Sat, Feb 16, 2013 at 03:13:06AM +0000, Ben Hutchings wrote:
> On Fri, 2013-02-15 at 08:56 +0100, Josip Rodin wrote:
> > > I appear to be experiencing a serious problem with a 768 MB RAM Xen domU
> > > machine running an NFS client - every now and then (for months now), often
> > > in the middle of the night, it enters some kind of a broken state where a
> > > few semi-random processes (mainly apache2's and vsftpd's which are told to
> > > serve files from the NFS mount)
> [...]
> > I caught it earlier just now, at:
> >
> > [950084.590733] active_anon:2805 inactive_anon:11835 isolated_anon:0
> > [950084.590735] active_file:76 inactive_file:516 isolated_file:32
> > [950084.590737] unevictable:783 dirty:1 writeback:0 unstable:0
> > [950084.590739] free:26251 slab_reclaimable:15733 slab_unreclaimable:128868
> > [950084.590741] mapped:938 shmem:75 pagetables:651 bounce:0
> >
> > And snuck in a few slabtops (even some -o invocations were getting killed,
> > along with my shell and pretty much everything else):
> [...]
> > 65390 65390 100% 2.06K 13338 15 426816K net_namespace
> [...]
>
> Looks like CVE-2011-2189, for which there was a fix/workaround in:
>
> vsftpd (2.3.2-3+squeeze2) stable-security; urgency=high
>
> * Non-maintainer upload by the Security Team.
> * Disable network isolation due to a problem with cleaning up network
> namespaces fast enough in kernels < 2.6.35 (CVE-2011-2189).
> Thanks Ben Hutchings for the patch!
> * Fix possible DoS via globa expressions in STAT commands by
> limiting the matching loop (CVE-2011-0762; Closes: #622741).
>
> -- Nico Golde <nion@debian.org> Wed, 07 Sep 2011 20:39:59 +0000
>
> Do you have an old version of vsftpd, or perhaps an upstream version
> which doesn't include the workaround?
No, 2.3.2-3+squeeze2 is there, has been since 2012-03-22.
> Anyway, I'm closing the bug report; please don't hijack closed bugs.
Eh? It was not closed for being fixed, it was closed en masse on a
procedural reason that could easily be wrong, and I don't believe I was
hijacking it; you just confirmed that this is a kernel problem above,
so how could this possibly be improper?!
--
2. That which causes joy or happiness.
----- End forwarded message -----
--- End Message ---