Bug#518643: Same problem found in 2.6.26-2-amd64
Hi,
We experienced a similar lockup tonight.
At 3:30 the system started to have problems:
Aug 14 03:39:33 camelot kernel: [7646288.524370] ------------[ cut
here ]------------
Aug 14 03:39:33 camelot kernel: [7646288.524402] kernel BUG at
arch/x86/kvm/mmu.c:615!
Aug 14 03:39:33 camelot kernel: [7646288.524427] invalid opcode: 0000
[1] SMP
Aug 14 03:39:33 camelot kernel: [7646288.524459] CPU 1
Aug 14 03:39:33 camelot kernel: [7646288.524485] Modules linked in:
iptable_filter ip_tables x_tables tun kvm_intel kvm ipv6 bridge loop
snd_pcm snd_timer snd soundcore snd_page_al
loc parport_pc parport iTCO_wdt i2c_i801 pcspkr i2c_core rng_core
i5000_edac edac_core container button shpchp pci_hotplug evdev ext3 jbd
mbcache dm_mirror dm_log dm_snapshot dm_mo
d ide_cd_mod cdrom usb_storage piix ide_pci_generic ide_core ses
enclosure sd_mod ata_generic ehci_hcd libata dock uhci_hcd tg3 aacraid
scsi_mod thermal processor fan thermal_sys
Aug 14 03:39:33 camelot kernel: [7646288.525219] Pid: 6140, comm: kvm
Not tainted 2.6.26-2-amd64 #1
Aug 14 03:39:33 camelot kernel: [7646288.525219] RIP:
0010:[<ffffffffa027b844>] [<ffffffffa027b844>] :kvm:kvm_mmu_get_page
+0x2bd/0x3cf
Aug 14 03:39:33 camelot kernel: [7646288.525219] RSP:
0018:ffff81017ebc3b48 EFLAGS: 00010246
Aug 14 03:39:33 camelot kernel: [7646288.525219] RAX: ffff81011c2dc210
RBX: ffffc200049d1fd8 RCX: 0000000000000000
Aug 14 03:39:33 camelot kernel: [7646288.525219] RDX: 87e8c88be7c689e6
RSI: ffff8101c5f8b880 RDI: 0000000000000000
Aug 14 03:39:33 camelot kernel: [7646288.525219] RBP: ffff8102f6545e48
R08: 000000000000000a R09: 0000000000000002
Aug 14 03:39:33 camelot kernel: [7646288.525219] R10: 000000000000000d
R11: ffffffffa0296fb4 R12: ffff81017d8c0000
Aug 14 03:39:33 camelot kernel: [7646288.525219] R13: ffff81017e0d02c0
R14: 000000000001bcfb R15: 000000000001bcfb
Aug 14 03:39:33 camelot kernel: [7646288.525219] FS:
0000000000000000(0000) GS:ffff81033e48b8c0(0000) knlGS:000000009a39fb90
Aug 14 03:39:33 camelot kernel: [7646288.525219] CS: 0010 DS: 002b ES:
002b CR0: 0000000080050033
Aug 14 03:39:33 camelot kernel: [7646288.525219] CR2: 0000000068ba0000
CR3: 000000017ed88000 CR4: 00000000000026e0
Aug 14 03:39:33 camelot kernel: [7646288.525219] DR0: 0000000000000000
DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 03:39:33 camelot kernel: [7646288.525219] DR3: 0000000000000000
DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 14 03:39:33 camelot kernel: [7646288.525219] Process kvm (pid: 6140,
threadinfo ffff81017ebc2000, task ffff81031da6c1c0)
Aug 14 03:39:33 camelot kernel: [7646288.525219] Stack:
0000000000000000 0000000000000002 0000000000000000 ffff81017e0d02c0
Aug 14 03:39:33 camelot kernel: [7646288.525219] ffff8101d5009a28
0000000000000007 0000000000000000 ffffffffa027d137
Aug 14 03:39:33 camelot kernel: [7646288.525219] ffff8101d5009a28
ffff81017ebc3c64 0000000000000000 00000000000015de
Aug 14 03:39:33 camelot kernel: [7646288.525219] Call Trace:
Aug 14 03:39:33 camelot kernel: [7646288.525219]
[<ffffffffa027d137>] ? :kvm:paging64_page_fault+0x2b4/0x3bf
Aug 14 03:39:33 camelot kernel: [7646288.525219]
[<ffffffffa0274b01>] ? :kvm:kvm_read_guest_atomic+0x4b/0x6c
Aug 14 03:39:33 camelot kernel: [7646288.525219]
[<ffffffffa027ccf2>] ? :kvm:paging64_prefetch_page+0x3b/0x77
Aug 14 03:39:33 camelot kernel: [7646288.525219]
[<ffffffffa0295499>] ? :kvm_intel:vmx_set_cr3+0xe5/0xec
Aug 14 03:39:33 camelot kernel: [7646288.525219]
[<ffffffffa027bc16>] ? :kvm:kvm_mmu_page_fault+0x19/0x81
Aug 14 03:39:33 camelot kernel: [7646288.525219]
[<ffffffffa0279318>] ? :kvm:kvm_arch_vcpu_ioctl_run+0x463/0x615
Aug 14 03:39:33 camelot kernel: [7646288.525219] [<ffffffff8022c246>] ?
try_to_wake_up+0x118/0x129
Aug 14 03:39:33 camelot kernel: [7646288.525219]
[<ffffffffa027461b>] ? :kvm:kvm_vcpu_ioctl+0x36/0x3a9
Aug 14 03:39:33 camelot kernel: [7646288.525219]
[<ffffffffa02746d8>] ? :kvm:kvm_vcpu_ioctl+0xf3/0x3a9
Aug 14 03:39:33 camelot kernel: [7646288.525219] [<ffffffff802284eb>] ?
__wake_up_common+0x41/0x74
Aug 14 03:39:33 camelot kernel: [7646288.525219] [<ffffffff8024fb23>] ?
futex_wake+0x74/0xfa
Aug 14 03:39:33 camelot kernel: [7646288.525219] [<ffffffff80250718>] ?
do_futex+0xa6/0x777
Aug 14 03:39:33 camelot kernel: [7646288.525219] [<ffffffff80248d5e>] ?
ktime_get_ts+0x22/0x4b
Aug 14 03:39:33 camelot kernel: [7646288.525219]
[<ffffffffa027436e>] ? :kvm:kvm_vm_ioctl+0x19c/0x1b5
Aug 14 03:39:33 camelot kernel: [7646288.525219] [<ffffffff8022f0d5>] ?
hrtick_set+0x88/0xf7
Aug 14 03:39:33 camelot kernel: [7646288.525219] [<ffffffff802125e7>] ?
read_tsc+0x9/0x20
Aug 14 03:39:33 camelot kernel: [7646288.525219] [<ffffffff802c8180>] ?
compat_sys_ioctl+0xc8/0x33f
Aug 14 03:39:33 camelot kernel: [7646288.525219] [<ffffffff80244d9c>] ?
posix_ktime_get_ts+0xc/0x10
Aug 14 03:39:33 camelot kernel: [7646288.525219] [<ffffffff80224c52>] ?
sysenter_do_call+0x1b/0x66
Aug 14 03:39:33 camelot kernel: [7646288.525219]
Aug 14 03:39:33 camelot kernel: [7646288.525219]
Aug 14 03:39:33 camelot kernel: [7646288.525219] Code: e7 49 89 c6 e8 75
ed ff ff 31 d2 48 89 c3 48 89 c6 4c 89 e7 e8 04 e8 ff ff c7 44 24 04 00
00 00 00 eb 2e 48 8b 10 f6 c2 01 75 04 <0f> 0b eb fe f6 c2 02 74 0f 48
83 e2 fd 48 89 10 c7 44 24 04 01
Aug 14 03:39:33 camelot kernel: [7646288.525219] RIP
[<ffffffffa027b844>] :kvm:kvm_mmu_get_page+0x2bd/0x3cf
Aug 14 03:39:33 camelot kernel: [7646288.525219] RSP <ffff81017ebc3b48>
Aug 14 03:39:33 camelot kernel: [7646288.526757] ---[ end trace
0d943ec153fc0496 ]---
After that, system started to have problems every minute:
Aug 14 03:40:39 camelot kernel: [7646353.902579] BUG: soft lockup -
CPU#2 stuck for 61s! [kvm:6137]
Aug 14 03:40:39 camelot kernel: [7646353.902579] Modules linked in:
iptable_filter ip_tables x_tables tun kvm_intel kvm ipv6 bridge loop
snd_pcm snd_timer snd soundcore snd_page_al
loc parport_pc parport iTCO_wdt i2c_i801 pcspkr i2c_core rng_core
i5000_edac edac_core container button shpchp pci_hotplug evdev ext3 jbd
mbcache dm_mirror dm_log dm_snapshot dm_mod ide_cd_mod cdrom usb_storage
piix ide_pci_generic ide_core ses enclosure sd_mod ata_generic ehci_hcd
libata dock uhci_hcd tg3 aacraid scsi_mod thermal processor fan
thermal_sys
Aug 14 03:40:39 camelot kernel: [7646353.902579] CPU 2:
Aug 14 03:40:39 camelot kernel: [7646353.902579] Modules linked in:
iptable_filter ip_tables x_tables tun kvm_intel kvm ipv6 bridge loop
snd_pcm snd_timer snd soundcore snd_page_alloc parport_pc parport
iTCO_wdt i2c_i801 pcspkr i2c_core rng_core i5000_edac edac_core
container button shpchp pci_hotplug evdev ext3 jbd mbcache dm_mirror
dm_log dm_snapshot dm_mod ide_cd_mod cdrom usb_storage piix
ide_pci_generic ide_core ses enclosure sd_mod ata_generic ehci_hcd
libata dock uhci_hcd tg3 aacraid scsi_mod thermal processor fan
thermal_sys
Aug 14 03:40:39 camelot kernel: [7646353.902579] Pid: 6137, comm: kvm
Tainted: G D 2.6.26-2-amd64 #1
Aug 14 03:40:39 camelot kernel: [7646353.902579] RIP:
0010:[<ffffffff804295cc>] [<ffffffff804295cc>] _spin_lock+0x12/0x15
Aug 14 03:40:39 camelot kernel: [7646353.902579] RSP:
0018:ffff81017eff1cb0 EFLAGS: 00000297
Aug 14 03:40:39 camelot kernel: [7646353.902579] RAX: 0000000000008d8c
RBX: 0000000000000000 RCX: ffff81033e7d6880
Aug 14 03:40:39 camelot kernel: [7646353.902579] RDX: 0000000000000001
RSI: ffff81017e804338 RDI: ffff81017d8c0018
Aug 14 03:40:39 camelot kernel: [7646353.902579] RBP: 00000000ffffffff
R08: ffff81017e804280 R09: 0000000000000003
Aug 14 03:40:39 camelot kernel: [7646353.902579] R10: ffff81017eff0000
R11: ffffffff802f2039 R12: ffffffffa0294eb9
Aug 14 03:40:39 camelot kernel: [7646353.902579] R13: ffff8102295cc000
R14: 0000000200000202 R15: ffff81033b9be348
Aug 14 03:40:39 camelot kernel: [7646353.902579] FS:
0000000000000000(0000) GS:ffff81033e48b0c0(0063) knlGS:000000009aba3b90
Aug 14 03:40:39 camelot kernel: [7646353.902579] CS: 0010 DS: 002b ES:
002b CR0: 000000008005003b
Aug 14 03:40:39 camelot kernel: [7646353.902579] CR2: 00000000927b7004
CR3: 000000017ed88000 CR4: 00000000000026e0
Aug 14 03:40:39 camelot kernel: [7646353.902579] DR0: 0000000000000000
DR1: 0000000000000000 DR2: 0000000000000000
Aug 14 03:40:39 camelot kernel: [7646353.902579] DR3: 0000000000000000
DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 14 03:40:39 camelot kernel: [7646353.902579]
Aug 14 03:40:39 camelot kernel: [7646353.902579] Call Trace:
Aug 14 03:40:39 camelot kernel: [7646353.902579]
[<ffffffffa027aa68>] ? :kvm:mmu_free_roots+0x25/0xfb
Aug 14 03:40:39 camelot kernel: [7646353.902579]
[<ffffffffa02790fc>] ? :kvm:kvm_arch_vcpu_ioctl_run+0x247/0x615
Aug 14 03:40:39 camelot kernel: [7646353.902579]
[<ffffffffa027461b>] ? :kvm:kvm_vcpu_ioctl+0x36/0x3a9
Aug 14 03:40:39 camelot kernel: [7646353.902579]
[<ffffffffa02746d8>] ? :kvm:kvm_vcpu_ioctl+0xf3/0x3a9
Aug 14 03:40:39 camelot kernel: [7646353.902579] [<ffffffff802284eb>] ?
__wake_up_common+0x41/0x74
Aug 14 03:40:39 camelot kernel: [7646353.902579] [<ffffffff8024fb23>] ?
futex_wake+0x74/0xfa
Aug 14 03:40:39 camelot kernel: [7646353.902579] [<ffffffff8023e248>] ?
__dequeue_signal+0x19/0x15a
Aug 14 03:40:39 camelot kernel: [7646353.902579] [<ffffffff80248d5e>] ?
ktime_get_ts+0x22/0x4b
Aug 14 03:40:39 camelot kernel: [7646353.902579] [<ffffffff8023decf>] ?
recalc_sigpending+0xe/0x38
Aug 14 03:40:39 camelot kernel: [7646353.902579] [<ffffffff8023f1c3>] ?
dequeue_signal+0x8d/0x111
Aug 14 03:40:39 camelot kernel: [7646353.902579] [<ffffffff8025d482>] ?
compat_sys_rt_sigtimedwait+0xfc/0x26a
Aug 14 03:40:39 camelot kernel: [7646353.902579] [<ffffffff802125e7>] ?
read_tsc+0x9/0x20
Aug 14 03:40:39 camelot kernel: [7646353.902579] [<ffffffff802c8180>] ?
compat_sys_ioctl+0xc8/0x33f
Aug 14 03:40:39 camelot kernel: [7646353.902579] [<ffffffff80244d9c>] ?
posix_ktime_get_ts+0xc/0x10
Aug 14 03:40:39 camelot kernel: [7646353.902579] [<ffffffff80224c52>] ?
sysenter_do_call+0x1b/0x66
Aug 14 03:40:39 camelot kernel: [7646353.902579]
[<ffffffffa0296f60>] ? :kvm_intel:handle_io+0x0/0x54
Aug 14 03:40:39 camelot kernel: [7646353.902579]
We arrived at 8:15 to office. Server behaviour was clunky at best. It
ceased to respond tcp/ip connections shortly after that, after an
apt-get update was issued. We had to cold-reboot the system.
System has been working without problems since January. It has been
rebooted maybe 2 or 3 times since then.
Shall I submit another bug?
Thanks,
Eneko
Reply to: