Bug#552706: BUG: soft lockup - CPU#0 stuck for 61s! - :nfs:nfs_access_cache_shrinker
Package: linux-2.6
Version: 2.6.26-19
Severity: important
We get the following backtrace way too often, it repeats for all CPUs,
but I'm pasting it only once here.
[633582.838114] BUG: soft lockup - CPU#0 stuck for 61s! [init:1]
[633582.838114] Modules linked in: nfs lockd nfs_acl sunrpc ipip tunnel4
ipv6 bonding coretemp ipmi_devintf ipmi_si ipmi_watchdog ipmi_msghandler
loop snd_pcm snd_timer snd soundcore sn
d_page_alloc i2c_i801 serio_raw rng_core pcspkr i2c_core psmouse button
i5000_edac shpchp edac_core pci_hotplug joydev evdev ext3 jbd mbcache
dm_mirror dm_log dm_snapshot dm_mod sg sr_m
od cdrom ata_generic ata_piix ses sd_mod libata enclosur dock usbhid hid
ff_memless ide_pci_generic megaraid_sas ide_core ehci_hc bnx2
firmware_class uhci_hcd scsi_mod e1000e thermal
processor fan thermal_sys [last unloaded: scsi_wait_scan]
[633582.838114] CPU 0:
[63352.838114] Modules linked in: nfs lockd nfs_acl sunrpc ipip tunnel4
ipv6bonding coretemp ipmi_devintf ipmi_si ipmi_watchdog ipmi_msghandler
loop snd_pcm snd_timer snd soundcore sn
d_page_alloc i2c_i801 serio_ra rng_core pcspkr i2c_core psmouse button
i5000_edac shpchp edac_core pci_hotlug joydev evdev ext3 jbd mbcache
dm_mirror dm_log dm_snapshot dm_mod sg sr_m
od cdrom ata_generic ata_piix ses sd_mod libata enclosure dockusbhid hid
ff_memless ide_pci_generic megaraid_sas ide_core ehci_hcd bnx2firmware_cl=
ass uhci_hcd scsi_mod e1000e thermal
processor fan thermal_sys [last unloaded: scsi_wait_scan]
[633582.838114] Pid: 1, comm: initNot tainted 2.6.26-2-amd64 #1
[633582.838114] RIP: 0010:[<ffffffff8042a3c>] [<ffffffff8042a23c>]
_spin_lock+0x12/0x15
[633582.838114] RSP: 0018:ffff8101bf077b70 EFLAGS: 00000293
[633582.838114] RAX: 000000000001311 RBX: 0000000000000000 RCX: 0000000000153793
[633582.838114] RDX: 000000000000000 RSI: 00000000000000d0 RDI: ffffffffa0342714
[633582.838114] RBP: 0000000000000000 R08: 0000000000000064 R09: ffff810001101200
[633582.838114] R10: 000000000000000c R11: fffffffa016c378 R12: 0000000000000020
[633582.838114] R13: 0000000000000020 R14: 0000000000000020 R15: 000000000000000a
[633582.838114] FS: 00007f264a972770(0000) GS:fffffff8053c000(0000) knlGS:0000000000000000
[633582.838114] CS: 0010 DS: 0000 ES:0000 CR0: 000000008005003b
[633582.838114] CR2: 0000000000608f70 CR3: 00000001bc8d000 CR4: 00000000000006e0
[633582.838114] DR0: 0000000000000000 DR1: 000000000000000 DR2: 0000000000000000
[633582.838114] DR3: 0000000000000000 DR6: 00000000ffff0f0 DR7: 0000000000000400
[633582.838114]=20
[633582.838114] Call Trace:
[633582.838114] [ffffffffa0308539>] :nfs:nfs_access_cache_shrinker+0x26/0x1e9
[633582.838114] [<ffffffff8027b323>] shrink_slab+0x60/0x159
[633582.838114] [<ffffffff8027b76>] try_to_free_pages+0x25a/0x361
[633582.838114] [<ffffffff8027a6b7>] isolate_pges_global+0x0/0x2f
[633582.838114] [<ffffffff80276abd>] __alloc_pages_internal+026a/0x3bf
[633582.838114] [<ffffffff80295590>] kmem_getpages+0x96/0x15f
[633582.838114] [<ffffffff80295bfb>] fallback_alloc+0x146/0x1e1
[633582.838114] [<ffffffff80296260>] kmem_cache_alloc+0xc4/0xf6
[633582.838114] [<ffffffff802a33ba>] getname+0x25/0x1a7
[633582.838114] [<ffffffff802a501b>] __user_walk_fd+0x19/0x4c
[633582.838114] [<ffffffff8029e181>] vfs_stat_fd+0x1b/0x4a
[633582.838114] [<ffffffff8029e20c>] sys_newstat+0x19/0x31
[633582.838114] [<ffffffff802a7c48>] sys_select+0x123/0x183
[633582.838114] [<ffffffff8020beca>] system_call_after_swapgs+0x8a/0x8f
[633582.838114]=20
[633610.757164] BUG: soft lockup - CPU#1 stuck for 61s! [bb-iostat.sh:194=
00]
[633610.797166] Modules linked in: nfs lockd nfs_acl sunrpc ipip tunnel4
ipv6 bonding coretemp ipmi_devintf ipmi_si ipmi_watchdog ipmi_msghandler
loop snd_pcm snd_timer snd soundcore sn
d_page_alloc i2c_i801 serio_raw rng_core pcspkr i2c_core psmouse button
i5000_edac shpchp edac_core pci_hotplug joydev evdev ext3 jbd mbcache
dm_mirror dm_log dm_snapshot dm_mod sg sr_m
od cdrom ata_generic ata_piix ses sd_mod libata enclosure dock usbhid hid
ff_memless ide_pci_generic megaraid_sas ide_coe ehci_hcd bnx2
firmware_class uhci_hcd scsi_mod e1000e thermal
processorfan thermal_sys [last unloaded: scsi_wait_scan]
[633611.121155] CPU 1:
[633611.121155] Modules linked in: nfs lockd nfs_acl sunrpc ipip tunnl4
ipv6 bonding coretemp ipmi_devintf ipmi_si ipmi_watchdog ipmi_msghanler
loop snd_pcm snd_timer snd soundcore sn
d_page_alloc i2c_i801 serio_raw rng_core pcspkr i2c_core psmouse button
i5000_edac shpchp edaccore pci_hotplug joydev evdev ext3 jbd mbcache
dm_mirror dm_log dm_snapshot m_mod sg sr_m
od cdrom ata_generic ata_piix ses sd_mod libata enclosure dock usbhid hid
ff_memless ide_pci_generic megaraid_sas ide_core ehc_hcd bnx2
firmware_class uhci_hcd scsi_mod e1000e thermal
processor fan termal_sys [last unloaded: scsi_wait_scan]
[633611.121155] Pid: 19400, comm: bb-iostat.sh Not tainted 2.6.26-2-amd64=
#1
[633611.121155] RIP: 010:[<ffffffff8042a23c>] [<ffffffff8042a23c>] _spin_lock+0x12/0x15
[63611.121155] RSP: 0000:ffff8100b1d3dbc0 EFLAGS: 00000297
[633611.121155] RAX: 0000000000001211 RBX: 0000000000000000 RCX: 000000000015378c
[633611.121155] RDX: 0000000000000000 RSI: 00000000001200 RDI: ffffffffa032714
[633611.121155] RBP: 0000000000000000 R08: 0000000000000064 R09: ffff810001101480
[633611.121155] R10: 000000000000000c R11: ffffffffa016c378 R12: 0000000000000020
[633611.11155] R13: 0000000000000020 R14: 0000000000000020 R15: 000000000000000a
[633611.121155] FS: 00007f64a9cd76e0(0000) GS:ffff8101bf0918c0(0000) knlGS0000000000000000
[633611.121155] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[633611.121155] CR2: 00000000022546a4 CR3: 000000006f854000 CR4: 00000000000006e0
633611.121155] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[33611.121155] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[63611.121155]
[633611.121155] Call Trace:
[633611.121155] [<ffffffffa0308539>] :nfs:nf_access_cache_shrinker+0x26/0x1e9
[633611.121155] [<ffffffff8027b323>] shrink_slab+0x60/0x159
[633611.121155] [<ffffffff8027b676>] try_to_free_pages+0x2a/0x361
[633611.121155] [<ffffffff8027a6b7>] isolate_pages_global+0x0/0x2f
[63361.121155] [<ffffffff80276abd>] __alloc_pages_internal+0x26a/0x3bf
[633611.121155] [<ffffffff8027eb58>] do_wp_page+0x26e/0x5b2
[633611.121155] [<ffffffff80281ca0>] handle_mm_fault+0xdd/0x867
[633611.121155] [<ffffffff8031e033>] __up_read+0x13/0x8a
[633611.121155] [<ffffffff8042a599>] error_exit+0x0/0x60
[633611.121155] [<ffffffff80221fbc>] do_page_fault+0x5d8/0x9c8
[633611.121155] [<ffffffff8042a599>] error_exit+0x0/0x60
[633611.121155]
[633619.348092] BUG: soft lockup - CPU#3 stuck for 61s! [events/3:18]
[633619.384094] Modules linked in: nfs lockd nfs_acl sunrpc ipip tunnel4
ipv6 bonding coretemp ipmi_devintf ipmi_si ipmi_watchdog ipmi_msghandler
loop snd_pcm snd_timer snd soundcore sn
d_page_alloc i2c_i801 serio_raw rng_core pcspkr i2c_core psmouse button
i5000_edac shpchp edac_core pci_hotplug joydev evdev ext3 jbd mbcache
dm_mirror dm_log dm_snapshot dm_mod sg sr_m
od cdrom ata_generic ata_piix ses sd_mod libata enclosure dock usbhid hid
ff_memless ide_pci_generic megaraid_sas ide_core ehci_hcd bnx2
firmware_class uhci_hcd scsi_mod e1000e thermal
processor fan thermal_sys [last unloaded: scsi_wait_scan]
[633619.705134] CPU 3:
[633619.705134] Modules linked in: nfs lockd nfs_acl sunrpc ipip tunnel4
iv6 bonding coretemp ipmi_devintf ipmi_si ipmi_watchdog ipmi_msghandler
lop snd_pcm snd_timer snd soundcore sn
d_page_alloc i2c_i801 serio_raw rng_core pcspkr i2c_core psmouse button
i5000_edac shpchp edac_core pcihotplug joydev evdev ext3 jbd mbcache
dm_mirror dm_log dm_snapshot dm_md sg sr_m
od cdrom ata_generic ata_piix ses sd_mod libata enclosure dock usbhid hid
ff_memless ide_pci_generic megaraid_sas ide_core ehci_cd bnx2
firmware_class uhci_hcd scsi_mod e1000e thermal
processor fan therma_sys [last unloaded: scsi_wait_scan]
[633619.705134] Pid: 18, comm: events/3 Not tainted 2.6.26-2-amd64 #1
[633619.705134] RIP: 0010:[<fffffff80219dc3>] [<ffffffff80219dc3>] native_smp_call_function_mask+0xdb/0x18
[633619.705134] RSP: 0018:ffff8101bf16fe20 EFLAGS: 00000297
[633619.705134] RAX: 00000000000008fc RBX: 0000000000000003 RCX: 000000000000001
[633619.705134] RDX: 00000000000000fc RSI: 00000000000008fc RDI: 000000000000286
[633619.705134] RBP: 000000000002c5a3 R08: ffff8101bf16e000 R09: ffff8101bd97de00
[633619.705134] R10: ffff8100010598f0 R11:ffffffff80219ce8 R12: 0000000000000800
[633619.705134] R13: 000000000002c5a3R14: 0000000000058b46 R15: 000000000b168c00
[633619.705134] FS: 0000000000000000(0000) GS:ffff8101bf0a57c0(0000) knlGS:0000000000000000
[633619.705134] CS: 0010 DS: 0018 ES: 0018 CR0: 00000008005003b
[633619.705134] CR2: 00007f08ad3063f0 CR3: 00000001b7d31000 CR4: 00000000000006e0
[633619.705134] DR0: 0000000000000000 DR1: 000000000000000 DR2: 0000000000000000
[633619.705134] DR3: 0000000000000000 DR6: 00000000fff0ff0 DR7: 0000000000000400
[633619.705134]=20
[633619.705134] Call Trace:
[633619.70134] [<ffffffff802164ff>] ? mcheck_check_cpu+0x0/0x36
[633619.705134] [<ffffffff0428e9f>] ? thread_return+0x6b/0xac
[633619.705134] [<ffffffff80215e18>] ? mchecktimer+0x0/0x7c
[633619.705134] [<ffffffff802164ff>] ? mcheck_check_cpu+0x0/0x36
[63361.705134] [<ffffffff80238f87>] ? on_each_cpu+0x10/0x30
[633619.705134] [<ffffffff80215e35>] ? mcheck_timer+0x1d/0x7c
[633619.705134] [<ffffffff80243120> ? run_workqueue+0x82/0x111
[633619.705134] [<ffffffff802439ed>] ? worker_thread+xd5/0xe0
[633619.705134] [<ffffffff80246221>] ? autoremove_wake_function+0x0/0x2e=
[633619.705134] [<ffffffff80243918>] ? worker_thread+0x0/0xe0
[633619.705134] [<ffffffff802460fb>]? kthread+0x47/0x74
[633619.705134] [<ffffffff802301e9>] ? schedule_tail+0x27/0x5c
[633619.705134] [<ffffffff8020cf28>] ? child_rip+0xa/0x12
[633619.705134] [<ffffffff802460b4>] ? kthread+0x0/0x74
[633619.705134] [<ffffffff8020cf1e>] ? child_rip+0x0/0x12
[633619.705134]=20
-- System Information:
Debian Release: squeeze/sid
APT prefers unstable
APT policy: (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.31.1-think (SMP w/2 CPU cores; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Reply to: