Bug#638316: linux-image-2.6.32-5-amd64: We have same problem with Squeeze DomU (Lenny works fine) under CentOS Xen 5.6 Dom0
Package: linux-2.6
Version: 2.6.32-41
Severity: normal
The system does hand under heavy load, like cp -ra bigfolder or tar czvpf bigfolder.
After the IO intensive task completes or is killed, system (and by system i mean web/sql/.. servers) continue to work. Changing priority with
ionice or nice, doesn't help.
If i boot using kernel from Lenny, no such problem occurs. But that it's not an option anymore, because i use aufs kernel module and
userspace tools from Squeeze are not complatible with Lenny's aufs kernel module. If more info is needed, i can provide it. Just ask.
-- Package-specific info:
** Version:
Linux version 2.6.32-5-amd64 (Debian 2.6.32-41) (ben@decadent.org.uk) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Mon Jan 16 16:22:28 UTC 2012
** Command line:
root=/dev/sda1 ro console=hvc0 root=UUID=a27f095d-d5da-4bb1-bdc7-e7c999285dc8 ro
** Tainted: C (1024)
* Module from drivers/staging has been loaded.
** Kernel log:
[526680.120060] [<ffffffffa00283f8>] ? ext3_sync_file+0x94/0xc8 [ext3]
[526680.120060] [<ffffffff8110c02c>] ? vfs_fsync_range+0x73/0x9e
[526680.120060] [<ffffffff8110c0d6>] ? do_fsync+0x28/0x39
[526680.120060] [<ffffffff8110c105>] ? sys_fsync+0xb/0x10
[526680.120060] [<ffffffff81010b63>] ? sysret_check+0x17/0x5a
[526680.120060] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[526680.120060] INFO: task cleanup:24697 blocked for more than 120 seconds.
[526680.120060] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[526680.120060] cleanup D 0000000000000000 0 24697 1747 0x00000000
[526680.120060] ffffffff814891f0 0000000000000286 0000000000000000 000000010000000e
[526680.120060] ffff8800d94bb478 ffff88002dc11e18 000000000000f9e0 ffff88002dc11fd8
[526680.120060] 0000000000015780 0000000000015780 ffff880097998e20 ffff880097999118
[526680.120060] Call Trace:
[526680.120060] [<ffffffffa0017e20>] ? log_wait_commit+0xbf/0x112 [jbd]
[526680.120060] [<ffffffff8106502a>] ? autoremove_wake_function+0x0/0x2e
[526680.120060] [<ffffffff8100e23f>] ? xen_restore_fl_direct_end+0x0/0x1
[526680.120060] [<ffffffff812fcc8a>] ? _spin_unlock_irqrestore+0xd/0xe
[526680.120060] [<ffffffffa00283f8>] ? ext3_sync_file+0x94/0xc8 [ext3]
[526680.120060] [<ffffffff8110c02c>] ? vfs_fsync_range+0x73/0x9e
[526680.120060] [<ffffffff8110c0d6>] ? do_fsync+0x28/0x39
[526680.153044] [<ffffffff8110c105>] ? sys_fsync+0xb/0x10
[526680.154077] [<ffffffff81010b63>] ? sysret_check+0x17/0x5a
[526680.155037] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[526800.157473] INFO: task qmgr:29078 blocked for more than 120 seconds.
[526800.158641] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[526800.159585] qmgr D ffff88000387ac40 0 29078 1747 0x00000000
[526800.160512] ffff8800db258e20 0000000000000282 ffff8800abc47e58 ffff8800abc47e54
[526800.161381] ffff8800abc47e48 ffff8800abc47e48 000000000000f9e0 ffff8800abc47fd8
[526800.161381] 0000000000015780 0000000000015780 ffff88009dbda350 ffff88009dbda648
[526800.161381] Call Trace:
[526800.161381] [<ffffffff812fc10b>] ? __mutex_lock_common+0x122/0x192
[526800.161381] [<ffffffff81153662>] ? cap_file_lock+0x0/0x3
[526800.161381] [<ffffffff8111f141>] ? sys_flock+0x12d/0x141
[526800.161381] [<ffffffff812fc233>] ? mutex_lock+0x1a/0x31
[526800.161381] [<ffffffff810efb21>] ? generic_file_llseek+0x22/0x51
[526800.161381] [<ffffffff810ee980>] ? sys_lseek+0x44/0x64
[526800.161381] [<ffffffff8111f149>] ? sys_flock+0x135/0x141
[526800.161381] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[526800.161381] INFO: task pickup:20508 blocked for more than 120 seconds.
[526800.161381] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[526800.161381] pickup D ffff8800da303880 0 20508 1747 0x00000000
[526800.161381] ffff8800da303880 0000000000000286 ffffffff8100e23f ffffffff810e729d
[526800.161381] ffff8800d8eda000 0000000000000000 000000000000f9e0 ffff8800978b1fd8
[526800.161381] 0000000000015780 0000000000015780 ffff8800b6b6f810 ffff8800b6b6fb08
[526800.161381] Call Trace:
[526800.161381] [<ffffffff8100e23f>] ? xen_restore_fl_direct_end+0x0/0x1
[526800.161381] [<ffffffff810e729d>] ? kmem_cache_free+0x72/0xa3
[526800.161381] [<ffffffff810f21e3>] ? cp_new_stat+0xe9/0xfc
[526800.161381] [<ffffffff812fc10b>] ? __mutex_lock_common+0x122/0x192
[526800.161381] [<ffffffff812fc233>] ? mutex_lock+0x1a/0x31
[526800.161381] [<ffffffff810efb21>] ? generic_file_llseek+0x22/0x51
[526800.161381] [<ffffffff810ee980>] ? sys_lseek+0x44/0x64
[526800.161381] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[526920.185396] INFO: task mysqld:1303 blocked for more than 120 seconds.
[526920.186634] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[526920.187761] mysqld D ffff880097998e20 0 1303 1127 0x00000000
[526920.188973] ffff880097998e20 0000000000000282 ffff880002ddde38 ffff880002ddde34
[526920.189349] ffff8800d96dee28 ffff880002ddde18 000000000000f9e0 ffff880002dddfd8
[526920.189349] 0000000000015780 0000000000015780 ffff8800d8f14db0 ffff8800d8f150a8
[526920.189349] Call Trace:
[526920.189349] [<ffffffffa0017e20>] ? log_wait_commit+0xbf/0x112 [jbd]
[526920.189349] [<ffffffff8106502a>] ? autoremove_wake_function+0x0/0x2e
[526920.189349] [<ffffffff8100e23f>] ? xen_restore_fl_direct_end+0x0/0x1
[526920.189349] [<ffffffff812fcc8a>] ? _spin_unlock_irqrestore+0xd/0xe
[526920.189349] [<ffffffffa00283f8>] ? ext3_sync_file+0x94/0xc8 [ext3]
[526920.189349] [<ffffffff8110c02c>] ? vfs_fsync_range+0x73/0x9e
[526920.189349] [<ffffffff8110c0d6>] ? do_fsync+0x28/0x39
[526920.189349] [<ffffffff8110c105>] ? sys_fsync+0xb/0x10
[526920.189349] [<ffffffff81010b63>] ? sysret_check+0x17/0x5a
[526920.189349] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[526920.189349] INFO: task qmgr:29078 blocked for more than 120 seconds.
[526920.189349] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[526920.189349] qmgr D ffff88000387ac40 0 29078 1747 0x00000000
[526920.189349] ffff8800db258e20 0000000000000282 ffff8800abc47e58 ffff8800abc47e54
[526920.189349] ffff8800abc47e48 ffff8800abc47e48 000000000000f9e0 ffff8800abc47fd8
[526920.189349] 0000000000015780 0000000000015780 ffff88009dbda350 ffff88009dbda648
[526920.189349] Call Trace:
[526920.189349] [<ffffffff812fc10b>] ? __mutex_lock_common+0x122/0x192
[526920.189349] [<ffffffff81153662>] ? cap_file_lock+0x0/0x3
[526920.189349] [<ffffffff8111f141>] ? sys_flock+0x12d/0x141
[526920.189349] [<ffffffff812fc233>] ? mutex_lock+0x1a/0x31
[526920.189349] [<ffffffff810efb21>] ? generic_file_llseek+0x22/0x51
[526920.189349] [<ffffffff810ee980>] ? sys_lseek+0x44/0x64
[526920.189349] [<ffffffff8111f149>] ? sys_flock+0x135/0x141
[526920.189349] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[526920.189349] INFO: task pickup:20508 blocked for more than 120 seconds.
[526920.189349] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[526920.189349] pickup D ffff8800da303880 0 20508 1747 0x00000000
[526920.189349] ffff8800da303880 0000000000000286 ffffffff8100e23f ffffffff810e729d
[526920.189349] ffff8800d8eda000 0000000000000000 000000000000f9e0 ffff8800978b1fd8
[526920.189349] 0000000000015780 0000000000015780 ffff8800b6b6f810 ffff8800b6b6fb08
[526920.189349] Call Trace:
[526920.189349] [<ffffffff8100e23f>] ? xen_restore_fl_direct_end+0x0/0x1
[526920.189349] [<ffffffff810e729d>] ? kmem_cache_free+0x72/0xa3
[526920.189349] [<ffffffff810f21e3>] ? cp_new_stat+0xe9/0xfc
[526920.189349] [<ffffffff812fc10b>] ? __mutex_lock_common+0x122/0x192
[526920.189349] [<ffffffff812fc233>] ? mutex_lock+0x1a/0x31
[526920.189349] [<ffffffff810efb21>] ? generic_file_llseek+0x22/0x51
[526920.189349] [<ffffffff810ee980>] ? sys_lseek+0x44/0x64
[526920.189349] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
** Model information
not available
** Loaded modules:
Module Size Used by
aufs 127769 3
ipt_REJECT 1953 9
xt_recent 5977 19
nf_conntrack_ipv4 9833 19
nf_defrag_ipv4 1139 1 nf_conntrack_ipv4
xt_state 1303 19
nf_conntrack 46535 2 nf_conntrack_ipv4,xt_state
xt_tcpudp 2319 19
iptable_filter 2258 1
ip_tables 13915 1 iptable_filter
x_tables 12845 5 ipt_REJECT,xt_recent,xt_state,xt_tcpudp,ip_tables
loop 11799 2
snd_pcm 60487 0
snd_timer 15598 1 snd_pcm
snd 46526 2 snd_pcm,snd_timer
soundcore 4598 1 snd
snd_page_alloc 6249 1 snd_pcm
pcspkr 1699 0
joydev 8459 0
evdev 7352 0
ext3 106710 2
jbd 37317 1 ext3
mbcache 5050 1 ext3
xen_netfront 15196 0
xen_blkfront 8358 1
** PCI devices:
not available
** USB devices:
not available
-- System Information:
Debian Release: 6.0.4
APT prefers stable
APT policy: (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.32-5-amd64 (SMP w/3 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages linux-image-2.6.32-5-amd64 depends on:
ii debconf [debconf-2.0] 1.5.36.1 Debian configuration management sy
ii initramfs-tools [linux-initra 0.98.8 tools for generating an initramfs
ii linux-base 2.6.32-41 Linux image base package
ii module-init-tools 3.12-2 tools for managing Linux kernel mo
Versions of packages linux-image-2.6.32-5-amd64 recommends:
pn firmware-linux-free <none> (no description available)
Versions of packages linux-image-2.6.32-5-amd64 suggests:
ii grub 0.97-64 GRand Unified Bootloader (dummy pa
pn linux-doc-2.6.32 <none> (no description available)
Versions of packages linux-image-2.6.32-5-amd64 is related to:
pn firmware-bnx2 <none> (no description available)
pn firmware-bnx2x <none> (no description available)
pn firmware-ipw2x00 <none> (no description available)
pn firmware-ivtv <none> (no description available)
pn firmware-iwlwifi <none> (no description available)
pn firmware-linux <none> (no description available)
pn firmware-linux-nonfree <none> (no description available)
pn firmware-qlogic <none> (no description available)
pn firmware-ralink <none> (no description available)
pn xen-hypervisor <none> (no description available)
-- debconf information:
linux-image-2.6.32-5-amd64/postinst/depmod-error-initrd-2.6.32-5-amd64: false
linux-image-2.6.32-5-amd64/prerm/removing-running-kernel-2.6.32-5-amd64: true
linux-image-2.6.32-5-amd64/postinst/missing-firmware-2.6.32-5-amd64:
linux-image-2.6.32-5-amd64/postinst/ignoring-do-bootloader-2.6.32-5-amd64:
Reply to: