Bug#513537: linux-image-2.6.26-1-openvz-amd64 hanging
On 7 February 2009 00:44:33 Tom Rathborne wrote:
It is very strange bug.
Can you specify a link where I can obtain vmliniX 2.6.26-1-openvz-amd64 image?
> Hi Daniel,
>
> I don't have much light to shed on your bug, except that I've got something
> similar without the nvidia kernel taint.
>
> You wrote:
> > XFS > LVM > dm_crypt > MD (RAID1) > { SATA AHCI, IDE PIIX }
>
> I'm running: ext3 loopback > ext3 > LVM > aacraid scsi.
>
> The bug was triggered while I was executing "invoke-rc.d vz stop" and
> simultaneously copying from the underlying ext3 to the ext3-loopback
> file.
>
> In every case the Call Traces end at: system_call_after_swapgs+0x8a/0x8f
>
> I've been trying to 'kill -9' all of the blocked processes, and that gives me,
> for example:
>
> [3594730.862500] BUG: soft lockup - CPU#1 stuck for 61s! [apache2:4576]
> [3594730.862500] Modules linked in: vzethdev vznetdev simfs vzrst vzcpt tun vzmon xt_length ipt_ttl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp ipt_REJECT xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack vzdquota vzdev ip_tables x_tables ipv6 loop snd_pcm snd_timer snd soundcore snd_page_alloc parport_pc parport pcspkr psmouse serio_raw k8temp i2c_amd8111 amd_rng rng_core i2c_amd756 i2c_core button shpchp pci_hotplug evdev ext3 jbd mbcache dm_mirror dm_log dm_snapshot dm_mod ide_cd_mod cdrom ide_pci_generic amd74xx ide_core sd_mod floppy ata_generic libata dock ohci_hcd tg3 aacraid scsi_mod thermal processor fan thermal_sys [last unloaded: simfs]
> [3594730.862500] CPU 1:
> [3594730.862500] Modules linked in: vzethdev vznetdev simfs vzrst vzcpt tun vzmon xt_length ipt_ttl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp ipt_REJECT xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack vzdquota vzdev ip_tables x_tables ipv6 loop snd_pcm snd_timer snd soundcore snd_page_alloc parport_pc parport pcspkr psmouse serio_raw k8temp i2c_amd8111 amd_rng rng_core i2c_amd756 i2c_core button shpchp pci_hotplug evdev ext3 jbd mbcache dm_mirror dm_log dm_snapshot dm_mod ide_cd_mod cdrom ide_pci_generic amd74xx ide_core sd_mod floppy ata_generic libata dock ohci_hcd tg3 aacraid scsi_mod thermal processor fan thermal_sys [last unloaded: simfs]
> [3594730.862500] Pid: 4576, comm: apache2 Not tainted 2.6.26-1-openvz-amd64 #1 036test001
> [3594730.862500] RIP: 0010:[<ffffffff804238de>] [<ffffffff804238de>] _spin_lock+0xc/0x15
> [3594730.862500] RSP: 0018:ffff810003367d10 EFLAGS: 00000293
> [3594730.862500] RAX: 0000000000001614 RBX: ffff81007e8e34a8 RCX: 0000000000000000
> [3594730.862500] RDX: ffffe200004e2968 RSI: 0000000000000002 RDI: ffff81007f5a67e0
> [3594730.862500] RBP: 0000000000000246 R08: 0000000000000008 R09: ffff810001101700
> [3594730.862500] R10: 0000000000000002 R11: 0000000000000000 R12: ffff810000010f80
> [3594730.862500] R13: ffffe200005612c0 R14: ffffffff8027a4ce R15: 0000000000000004
> [3594730.862500] FS: 00007f6e32f0f750(0000) GS:ffff81007f5a6a40(0000) knlGS:0000000000000000
> [3594730.862500] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [3594730.862500] CR2: 000000000183c1c8 CR3: 00000000549b5000 CR4: 00000000000006e0
> [3594730.862500] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [3594730.862500] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [3594730.862500]
> [3594730.862500] Call Trace:
> [3594730.862500] [<ffffffff802971d1>] ? shmem_free_blocks+0x27/0x42
> [3594730.862500] [<ffffffff80297a41>] ? shmem_truncate_range+0x763/0x808
> [3594730.862500] [<ffffffff80299a55>] ? shmem_delete_inode+0x65/0xde
> [3594730.862500] [<ffffffff802999f0>] ? shmem_delete_inode+0x0/0xde
> [3594730.862500] [<ffffffff802b398c>] ? generic_delete_inode+0xa3/0x115
> [3594730.862500] [<ffffffff802b0777>] ? d_kill+0x38/0x59
> [3594730.862500] [<ffffffff802b1964>] ? dput+0x119/0x14f
> [3594730.862500] [<ffffffff802a1d70>] ? __fput+0x14f/0x178
> [3594730.862500] [<ffffffff80288913>] ? remove_vma+0x53/0x88
> [3594730.862500] [<ffffffff80289633>] ? do_munmap+0x205/0x227
> [3594730.862500] [<ffffffff80423766>] ? __down_write_nested+0x12/0xa1
> [3594730.862500] [<ffffffff80289695>] ? sys_munmap+0x40/0x5a
> [3594730.862500] [<ffffffff8020bffa>] ? system_call_after_swapgs+0x8a/0x8f
> [3594730.862500]
>
> Things are continuing to block, e.g.:
>
> [3594580.540114] INFO: task sshd:23585 blocked for more than 120 seconds.
> [3594580.540182] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [3594580.540283] sshd D ffff81013d92d7d0 0 23585 15642
> [3594580.540288] ffff8101207a1bb0 0000000000000086 0000000000000000 0000000000000002
> [3594580.540294] ffff81013d92d7d0 ffff81007f5b2810 ffff81013d92da58 00000003bd68fcf8
> [3594580.540300] 0000000000000002 0000000000000000 00000000ffffffff 0000000000000000
> [3594580.540305] Call Trace:
> [3594580.540329] [<ffffffff80358ae3>] mix_pool_bytes_extract+0x5c/0x155
> [3594580.540338] [<ffffffff80422bd7>] __mutex_lock_slowpath+0x64/0x9b
> [3594580.540346] [<ffffffff80422a3c>] mutex_lock+0xa/0xb
> [3594580.540352] [<ffffffff803b97b3>] rtnetlink_rcv+0x9/0x1e
> [3594580.540357] [<ffffffff803c7a46>] netlink_unicast+0x215/0x28d
> [3594580.540362] [<ffffffff803aaa0b>] __alloc_skb+0x8d/0x153
> [3594580.540370] [<ffffffff803c8240>] netlink_sendmsg+0x25b/0x26e
> [3594580.540382] [<ffffffff803a46ae>] sock_sendmsg+0xcb/0xe3
> [3594580.540393] [<ffffffff80247be5>] autoremove_wake_function+0x0/0x2e
> [3594580.540405] [<ffffffff8022a8b9>] __wake_up+0x38/0x4f
> [3594580.540412] [<ffffffff803c7141>] netlink_insert+0x118/0x127
> [3594580.540420] [<ffffffff803a50aa>] sys_sendto+0xf3/0x127
> [3594580.540427] [<ffffffff803a5273>] move_addr_to_user+0x5d/0x78
> [3594580.540434] [<ffffffff803a5715>] sys_getsockname+0x72/0xa2
> [3594580.540440] [<ffffffff802b154f>] d_instantiate+0x52/0x5d
> [3594580.540453] [<ffffffff8020bffa>] system_call_after_swapgs+0x8a/0x8f
>
> I would be happy to provide any other information, but I'm not sure what would
> be useful at this point!
>
> Regards,
>
> Tom
>
--
Thank,
Vitaliy Gusev
Reply to: