Bug#519586: Huge "Slab Unreclaimable" and continually growing
On Tue, Jan 22, 2013 at 10:59:17AM +0100, Josip Rodin wrote:
> I appear to be experiencing a serious problem with a 768 MB RAM Xen domU
> machine running an NFS client - every now and then (for months now), often
> in the middle of the night, it enters some kind of a broken state where a
> few semi-random processes (mainly apache2's and vsftpd's which are told to
> serve files from the NFS mount) start battling it out for the memory, and
> everything including sshd starts invoking the OOM killer, over and over
> again. Nothing seems to halt the downward spiral; manual invocation of the
> OOM killer does nothing of any use. Terminating all processes is the only
> thing that makes it go quiet, but then that's effectively the same as a
> reboot.
>
> This is the SysRq+M output on the machine once it's been in the broken state
> for a while:
[...]
> active_anon:394 inactive_anon:3197 isolated_anon:0
> active_file:25 inactive_file:176 isolated_file:32
> unevictable:2659 dirty:1 writeback:0 unstable:0
> free:21456 slab_reclaimable:16177 slab_unreclaimable:143165
> mapped:677 shmem:76 pagetables:455 bounce:0
[...]
> The thing I noticed was the slab_unreclaimable explosion, by a factor
> of 122. That... doesn't sound like something that should be happenning.
>
> Googling for slab_unreclaimable found me this old bug report about
> slab_unreclaimable domU problems that was mass-closed with the switch to the
> new paravirtops Xen release. Granted, our use case is not Samba like with
> the original reporter, but the pattern of a file server was close enough for
> me to be uncomfortable with it :|
I caught it earlier just now, at:
[950084.590733] active_anon:2805 inactive_anon:11835 isolated_anon:0
[950084.590735] active_file:76 inactive_file:516 isolated_file:32
[950084.590737] unevictable:783 dirty:1 writeback:0 unstable:0
[950084.590739] free:26251 slab_reclaimable:15733 slab_unreclaimable:128868
[950084.590741] mapped:938 shmem:75 pagetables:651 bounce:0
And snuck in a few slabtops (even some -o invocations were getting killed,
along with my shell and pretty much everything else):
Active / Total Objects (% used) : 555753 / 587128 (94.7%)
Active / Total Slabs (% used) : 49430 / 49430 (100.0%)
Active / Total Caches (% used) : 65 / 76 (85.5%)
Active / Total Size (% used) : 546613.78K / 553025.01K (98.8%)
Minimum / Average / Maximum Object : 0.01K / 0.94K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
90993 66836 73% 0.19K 4333 21 17332K dentry
75840 73664 97% 0.12K 2370 32 9480K kmalloc-128
68096 68092 99% 0.01K 133 512 532K kmalloc-8
65888 65655 99% 0.25K 4118 16 16472K kmalloc-256
65820 65778 99% 1.00K 4767 16 76272K kmalloc-1024
65436 65414 99% 0.63K 5454 12 43632K proc_inode_cache
65419 65419 100% 4.00K 14179 8 453728K kmalloc-4096
65390 65390 100% 2.06K 13338 15 426816K net_namespace
4998 4990 99% 0.08K 98 51 392K sysfs_dir_cache
4224 2018 47% 0.06K 66 64 264K kmalloc-64
2288 2107 92% 0.18K 104 22 416K vm_area_struct
1792 1789 99% 0.02K 7 256 28K kmalloc-16
1470 1203 81% 0.19K 70 21 280K kmalloc-192
1300 402 30% 0.79K 65 20 1040K ext3_inode_cache
896 731 81% 0.03K 7 128 28K anon_vma
784 532 67% 0.55K 56 14 448K radix_tree_node
A bit later:
Active / Total Objects (% used) : 555403 / 586704 (94.7%)
Active / Total Slabs (% used) : 49394 / 49394 (100.0%)
Active / Total Caches (% used) : 65 / 76 (85.5%)
Active / Total Size (% used) : 546552.82K / 552827.43K (98.9%)
Minimum / Average / Maximum Object : 0.01K / 0.94K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
90993 66779 73% 0.19K 4333 21 17332K dentry
75840 73654 97% 0.12K 2370 32 9480K kmalloc-128
68096 68092 99% 0.01K 133 512 532K kmalloc-8
65888 65601 99% 0.25K 4118 16 16472K kmalloc-256
65852 65741 99% 1.00K 4760 16 76160K kmalloc-1024
65436 65409 99% 0.63K 5454 12 43632K proc_inode_cache
65428 65428 100% 4.00K 14181 8 453792K kmalloc-4096
65391 65391 100% 2.06K 13339 15 426848K net_namespace
4998 4986 99% 0.08K 98 51 392K sysfs_dir_cache
4224 2017 47% 0.06K 66 64 264K kmalloc-64
2134 2108 98% 0.18K 97 22 388K vm_area_struct
1792 1789 99% 0.02K 7 256 28K kmalloc-16
1449 1078 74% 0.19K 69 21 276K kmalloc-192
1100 376 34% 0.79K 55 20 880K ext3_inode_cache
896 639 71% 0.03K 7 128 28K anon_vma
714 554 77% 0.55K 51 14 408K radix_tree_node
And a bit later still:
[950506.398951] active_anon:2370 inactive_anon:12180 isolated_anon:2
[950506.398953] active_file:0 inactive_file:261 isolated_file:36
[950506.398955] unevictable:831 dirty:1 writeback:0 unstable:0
[950506.398957] free:26418 slab_reclaimable:15749 slab_unreclaimable:129161
[950506.398959] mapped:743 shmem:75 pagetables:636 bounce:0
Active / Total Objects (% used) : 556917 / 587606 (94.8%)
Active / Total Slabs (% used) : 49717 / 49717 (100.0%)
Active / Total Caches (% used) : 65 / 76 (85.5%)
Active / Total Size (% used) : 548047.95K / 554096.57K (98.9%)
Minimum / Average / Maximum Object : 0.01K / 0.94K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
90993 66921 73% 0.19K 4333 21 17332K dentry
75840 73825 97% 0.12K 2370 32 9480K kmalloc-128
68096 68092 99% 0.01K 133 512 532K kmalloc-8
65984 65766 99% 0.25K 4124 16 16496K kmalloc-256
65964 65913 99% 1.00K 4776 16 76416K kmalloc-1024
65592 65566 99% 0.63K 5467 12 43736K proc_inode_cache
65584 65584 100% 4.00K 14288 8 457216K kmalloc-4096
65552 65552 100% 2.06K 13500 15 432000K net_namespace
4998 4986 99% 0.08K 98 51 392K sysfs_dir_cache
4224 1976 46% 0.06K 66 64 264K kmalloc-64
2398 2398 100% 0.18K 109 22 436K vm_area_struct
1792 1789 99% 0.02K 7 256 28K kmalloc-16
1428 1112 77% 0.19K 68 21 272K kmalloc-192
1100 402 36% 0.79K 55 20 880K ext3_inode_cache
896 648 72% 0.03K 7 128 28K anon_vma
742 664 89% 0.55K 53 14 424K radix_tree_node
[950532.846913] SysRq : Show Blocked State
[950532.846930] task PC stack pid father
[950532.846942] kswapd0 D ffff88002f123880 0 30 2 0x00000000
[950532.846955] ffff88002f123880 0000000000000246 ffff88001ae8cbd0 000000010e28d9ca
[950532.846971] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002dfb1fd8
[950532.846987] 0000000000015780 0000000000015780 ffff88002fe44db0 ffff88002fe450a8
[950532.847003] Call Trace:
[950532.847018] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.847028] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.847037] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.847049] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.847061] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.847071] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.847081] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b
[950532.847091] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.847100] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.847110] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.847121] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.847131] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.847142] [<ffffffff810c11a8>] ? kswapd+0x5f9/0x686
[950532.847151] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.847160] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.847170] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.847179] [<ffffffff810c0baf>] ? kswapd+0x0/0x686
[950532.847187] [<ffffffff81065dc5>] ? kthread+0x79/0x81
[950532.847197] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[950532.847205] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[950532.847214] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[950532.847223] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[950532.847234] vsftpd D 0000000000000000 0 1325 1 0x00000000
[950532.847245] ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9ca
[950532.847261] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff880002697fd8
[950532.847276] 0000000000015780 0000000000015780 ffff88002f123880 ffff88002f123b78
[950532.847291] Call Trace:
[950532.847299] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.847308] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.847317] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.847327] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.847336] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.848079] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.848088] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.848098] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.848108] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.848117] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.848127] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.848137] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.848146] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.848156] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.848165] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.848175] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.848187] [<ffffffff81154851>] ? cap_sb_alloc_security+0x0/0x3
[950532.848197] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.848207] [<ffffffff810e8769>] ? kmem_cache_alloc+0x8c/0xf0
[950532.848216] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.848224] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.848235] [<ffffffff8108bc1e>] ? copy_pid_ns+0x5f/0x2a5
[950532.848244] [<ffffffff8108bc1e>] ? copy_pid_ns+0x5f/0x2a5
[950532.848253] [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0
[950532.848262] [<ffffffff8108bc1e>] ? copy_pid_ns+0x5f/0x2a5
[950532.848271] [<ffffffff810692fa>] ? create_new_namespaces+0xc2/0x15e
[950532.848281] [<ffffffff8106954e>] ? copy_namespaces+0x61/0x97
[950532.848291] [<ffffffff8104df25>] ? copy_process+0xa7a/0x1165
[950532.848300] [<ffffffff8104e767>] ? do_fork+0x157/0x31e
[950532.848309] [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[950532.848318] [<ffffffff81011e63>] ? stub_clone+0x13/0x20
[950532.848327] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.848336] vsftpd D 0000000000000000 0 28790 28778 0x00000000
[950532.848348] ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9d4
[950532.848363] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff8800279affd8
[950532.848379] 0000000000015780 0000000000015780 ffff88002f15c6a0 ffff88002f15c998
[950532.848395] Call Trace:
[950532.848402] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.848412] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.848421] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.848431] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.848441] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.848450] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.848460] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.848470] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.848479] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.848488] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.848498] [<ffffffff810bfd54>] ? shrink_list+0xf8/0x767
[950532.848509] [<ffffffff8119934b>] ? __bitmap_weight+0x3a/0x7e
[950532.848519] [<ffffffff8118f7fe>] ? cpumask_next_and+0x2a/0x3a
[950532.848529] [<ffffffff81045e7e>] ? find_busiest_group+0x420/0xa4e
[950532.848539] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.848548] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.848557] [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20
[950532.848567] [<ffffffff8130e67a>] ? error_exit+0x2a/0x60
[950532.848575] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[950532.848585] [<ffffffff810bc5c9>] ? throttle_vm_writeout+0x30/0x8d
[950532.848595] [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
[950532.848604] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.848614] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.848623] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.848633] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.848643] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.848653] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.848662] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.848671] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.848681] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.848690] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.848699] [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0
[950532.848708] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.848717] [<ffffffff810fb1a6>] ? user_path_at+0x20/0x79
[950532.848726] [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc
[950532.848735] [<ffffffff810ef186>] ? sys_chdir+0x1b/0x64
[950532.848744] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.848753] apache2 D 0000000000000000 0 30698 18611 0x00000000
[950532.848765] ffff88002fdd0e20 0000000000000282 0000000000000000 000000010e28d9c8
[950532.848781] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88000222ffd8
[950532.848796] 0000000000015780 0000000000015780 ffff880002699530 ffff880002699828
[950532.848812] Call Trace:
[950532.848819] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.848829] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.848838] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.848848] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.848857] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.848867] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.848876] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.848886] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.848895] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.848905] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.848915] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.848925] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.848935] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.848944] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.848954] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.848964] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.848975] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.848985] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.848995] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.849004] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.849014] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.849024] [<ffffffff8129835d>] ? tcp_established_options+0x2d/0xa9
[950532.849033] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.849041] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.849051] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.849061] [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[950532.849070] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.849081] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[950532.849090] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.849099] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.849109] [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[950532.849118] [<ffffffff81041432>] ? set_next_entity+0x34/0x56
[950532.849127] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.849138] [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[950532.849146] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.849156] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.849165] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.849174] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.849183] [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[950532.849192] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[950532.849202] [<ffffffff8110d00c>] ? __generic_file_splice_read+0x36e/0x3bd
[950532.849211] [<ffffffff81251f64>] ? kernel_sendmsg+0x32/0x3f
[950532.849220] [<ffffffff81254790>] ? sock_no_sendpage+0x87/0x94
[950532.849230] [<ffffffff812909f1>] ? tcp_sendpage+0x4b/0x458
[950532.849239] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[950532.849248] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff81250d0d>] ? kernel_sendpage+0x16/0x1f
[950532.850903] [<ffffffff81250d4b>] ? sock_sendpage+0x35/0x3a
[950532.850903] [<ffffffff8110b122>] ? pipe_to_sendpage+0x66/0x74
[950532.850903] [<ffffffff8110b17c>] ? splice_from_pipe_feed+0x4c/0xd4
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff8110b405>] ? __splice_from_pipe+0x3e/0x69
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff8110b47d>] ? splice_from_pipe+0x4d/0x63
[950532.850903] [<ffffffff8110c00d>] ? direct_splice_actor+0x1b/0x1e
[950532.850903] [<ffffffff8110b9c8>] ? splice_direct_to_actor+0xdc/0x188
[950532.850903] [<ffffffff8110bff2>] ? direct_splice_actor+0x0/0x1e
[950532.850903] [<ffffffff8110babe>] ? do_splice_direct+0x4a/0x64
[950532.850903] [<ffffffff810f0312>] ? do_sendfile+0x12d/0x1a8
[950532.850903] [<ffffffff810f03d6>] ? sys_sendfile64+0x49/0x89
[950532.850903] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.850903] apache2 D 0000000000000000 0 30702 18611 0x00000000
[950532.850903] ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9c8
[950532.850903] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff880023c6bfd8
[950532.850903] 0000000000015780 0000000000015780 ffff880002503880 ffff880002503b78
[950532.850903] Call Trace:
[950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.850903] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.850903] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.850903] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.850903] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.850903] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.850903] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.850903] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.850903] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.850903] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.850903] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.850903] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.850903] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.850903] [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
[950532.850903] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.850903] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903] [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903] [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb
[950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903] [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7
[950532.850903] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84
[950532.850903] [<ffffffff8110d00c>] ? __generic_file_splice_read+0x36e/0x3bd
[950532.850903] [<ffffffff81251f64>] ? kernel_sendmsg+0x32/0x3f
[950532.850903] [<ffffffff81254790>] ? sock_no_sendpage+0x87/0x94
[950532.850903] [<ffffffff812909f1>] ? tcp_sendpage+0x4b/0x458
[950532.850903] [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff81250d0d>] ? kernel_sendpage+0x16/0x1f
[950532.850903] [<ffffffff81250d4b>] ? sock_sendpage+0x35/0x3a
[950532.850903] [<ffffffff8110b122>] ? pipe_to_sendpage+0x66/0x74
[950532.850903] [<ffffffff8110b17c>] ? splice_from_pipe_feed+0x4c/0xd4
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff8110b405>] ? __splice_from_pipe+0x3e/0x69
[950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74
[950532.850903] [<ffffffff8110b47d>] ? splice_from_pipe+0x4d/0x63
[950532.850903] [<ffffffff8110c00d>] ? direct_splice_actor+0x1b/0x1e
[950532.850903] [<ffffffff8110b9c8>] ? splice_direct_to_actor+0xdc/0x188
[950532.850903] [<ffffffff8110bff2>] ? direct_splice_actor+0x0/0x1e
[950532.850903] [<ffffffff8110babe>] ? do_splice_direct+0x4a/0x64
[950532.850903] [<ffffffff810f0312>] ? do_sendfile+0x12d/0x1a8
[950532.850903] [<ffffffff810f03d6>] ? sys_sendfile64+0x49/0x89
[950532.850903] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.850903] vsftpd D 0000000000000000 0 32054 32049 0x00000000
[950532.850903] ffffffff814a11f0 0000000000000286 0000000000000000 000000010e28d9dd
[950532.850903] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff8800241e3fd8
[950532.850903] 0000000000015780 0000000000015780 ffff880002698710 ffff880002698a08
[950532.850903] Call Trace:
[950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.850903] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.850903] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.850903] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.850903] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.850903] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903] [<ffffffffa00bb1ef>] ? tcp_packet+0xb20/0xb65 [nf_conntrack]
[950532.850903] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.850903] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.850903] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f
[950532.850903] [<ffffffffa01aface>] ? ipt_do_table+0x5ee/0x620 [ip_tables]
[950532.850903] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.850903] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.850903] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.850903] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.850903] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.850903] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.850903] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903] [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b
[950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a
[950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5
[950532.850903] [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f
[950532.850903] [<ffffffff812514dc>] ? sock_aio_write+0xb1/0xbc
[950532.850903] [<ffffffff8100c426>] ? pfn_to_mfn+0xe/0x22
[950532.850903] [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83
[950532.850903] [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3
[950532.850903] [<ffffffff810effee>] ? do_sync_write+0xce/0x113
[950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903] [<ffffffff810f0953>] ? vfs_write+0xbc/0x102
[950532.850903] [<ffffffff810f0a55>] ? sys_write+0x45/0x6e
[950532.850903] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.850903] update-pkg-ft D 0000000000000000 0 32082 32038 0x00000000
[950532.850903] ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9d2
[950532.850903] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff880024f07fd8
[950532.850903] 0000000000015780 0000000000015780 ffff880002798000 ffff8800027982f8
[950532.850903] Call Trace:
[950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe
[950532.850903] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153
[950532.850903] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd
[950532.850903] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5
[950532.850903] [<ffffffffa0038d2f>] ? bdev_try_to_free_page+0x0/0x64 [ext3]
[950532.850903] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca
[950532.850903] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80
[950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e
[950532.850903] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767
[950532.850903] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d
[950532.850903] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259
[950532.850903] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.850903] [<ffffffff810993bc>] ? __call_rcu+0x110/0x118
[950532.850903] [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20
[950532.850903] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342
[950532.850903] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153
[950532.850903] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e
[950532.850903] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f
[950532.850903] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd
[950532.850903] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca
[950532.850903] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b
[950532.850903] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.850903] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.850903] [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0
[950532.850903] [<ffffffff810fa746>] ? getname+0x23/0x1a0
[950532.850903] [<ffffffff810fb1a6>] ? user_path_at+0x20/0x79
[950532.850903] [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f
[950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903] [<ffffffff810f361a>] ? vfs_fstatat+0x2c/0x57
[950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa
[950532.850903] [<ffffffff810f370d>] ? sys_newstat+0x11/0x30
[950532.850903] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[950532.850903] [<ffffffff8100b4d3>] ? xen_mc_issue+0x21/0x23
[950532.850903] [<ffffffff8101295b>] ? device_not_available+0x1b/0x20
[950532.850903] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b
[950532.850903] Sched Debug Version: v0.09, 2.6.32-5-xen-amd64 #1
[950532.850903] now at 950533247.852699 msecs
[950532.850903] .jiffies : 4532525608
[950532.850903] .sysctl_sched_latency : 10.000000
[950532.850903] .sysctl_sched_min_granularity : 2.000000
[950532.850903] .sysctl_sched_wakeup_granularity : 2.000000
[950532.850903] .sysctl_sched_child_runs_first : 0.000000
[950532.850903] .sysctl_sched_features : 32611451
[950532.850903]
[950532.850903] cpu#0, 2333.472 MHz
[950532.850903] .nr_running : 0
[950532.850903] .load : 0
[950532.850903] .nr_switches : 238671752
[950532.850903] .nr_load_updates : 165114440
[950532.850903] .nr_uninterruptible : 6
[950532.850903] .next_balance : 4532.525542
[950532.850903] .curr->pid : 0
[950532.850903] .clock : 950532844.214897
[950532.850903] .cpu_load[0] : 0
[950532.850903] .cpu_load[1] : 192
[950532.850903] .cpu_load[2] : 254
[950532.850903] .cpu_load[3] : 197
[950532.850903] .cpu_load[4] : 130
[950532.850903]
[950532.850903] cfs_rq[0]:/
[950532.850903] .exec_clock : 0.000000
[950532.850903] .MIN_vruntime : 0.000001
[950532.850903] .min_vruntime : 136491473.007815
[950532.850903] .max_vruntime : 0.000001
[950532.850903] .spread : 0.000000
[950532.850903] .spread0 : 0.000000
[950532.850903] .nr_running : 0
[950532.850903] .load : 0
[950532.850903] .nr_spread_over : 0
[950532.850903] .shares : 0
[950532.850903]
[950532.850903] rt_rq[0]:
[950532.850903] .rt_nr_running : 0
[950532.850903] .rt_throttled : 0
[950532.850903] .rt_time : 0.000000
[950532.850903] .rt_runtime : 950.000000
[950532.850903]
[950532.850903] runnable tasks:
[950532.850903] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[950532.850903] ----------------------------------------------------------------------------------------------------------
[950532.850903]
[950532.850903] cpu#1, 2333.472 MHz
[950532.850903] .nr_running : 3
[950532.850903] .load : 3072
[950532.850903] .nr_switches : 45263996
[950532.850903] .nr_load_updates : 121842283
[950532.850903] .nr_uninterruptible : 0
[950532.850903] .next_balance : 4532.525625
[950532.850903] .curr->pid : 32083
[950532.850903] .clock : 950533245.009907
[950532.850903] .cpu_load[0] : 3072
[950532.850903] .cpu_load[1] : 3072
[950532.850903] .cpu_load[2] : 3072
[950532.850903] .cpu_load[3] : 3072
[950532.850903] .cpu_load[4] : 3048
[950532.850903]
[950532.850903] cfs_rq[1]:/
[950532.850903] .exec_clock : 0.000000
[950532.850903] .MIN_vruntime : 161709543.323397
[950532.850903] .min_vruntime : 161709548.323397
[950532.850903] .max_vruntime : 161709543.323397
[950532.850903] .spread : 0.000000
[950532.850903] .spread0 : 25218075.315582
[950532.850903] .nr_running : 3
[950532.850903] .load : 3072
[950532.850903] .nr_spread_over : 0
[950532.850903] .shares : 0
[950532.850903]
[950532.850903] rt_rq[1]:
[950532.850903] .rt_nr_running : 0
[950532.850903] .rt_throttled : 0
[950532.850903] .rt_time : 0.000000
[950532.850903] .rt_runtime : 950.000000
[950532.850903]
[950532.850903] runnable tasks:
[950532.850903] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[950532.850903] ----------------------------------------------------------------------------------------------------------
[950532.850903] events/1 10 161709543.323397 1041646 120 0 0 0.000000 0.000000 0.000000 /
[950532.850903] apache2 30698 161709543.323397 3096 120 0 0 0.000000 0.000000 0.000000 /
[950532.850903] R network.sh 32083 161709832.320247 2 120 0 0 0.000000 0.000000 0.000000 /
[950532.850903]
--
2. That which causes joy or happiness.
Reply to: