Bug#517748: linux-image-2.6.26-1-686 System locks up under heavy disk load (MD Resync e.g.)
I have not been able to reproduce the problem on my testbox (Since I
test this onder VMware, hardware is not comparable to the production
machine). I did update the kernel with the newer version in te
repositories (Now running linux-image-2.6.26-1-686, 2.6.26-13lenny2)
but the problem still exists.
To keep the system workable (and prevent long lockups) I set the
resync speed to 2MB/s resulting in a two day resync time for the
array. Though even at 2MB/s the system still lockes up only for
shorter periods of time. Still getting the same messages in dmesg, see
attached txt file.
[1406330.496019] BUG: soft lockup - CPU#0 stuck for 85s! [snmpd:4612]
[1406330.496019] Modules linked in: ppdev parport_pc lp parport sit tunnel4 ipt_ MASQUERADE xt_multiport xt_tcpudp xt_state ip6table_filter iptable_nat nf_nat nf _conntrack_ipv4 iptable_filter ip_tables jfs dm_snapshot dm_mirror dm_log dm_mod ip6t_rt ip6table_mangle ip6_tables x_tables ipv6 cpufreq_ondemand acpi_cpufreq freq_table fuse nf_conntrack_ftp nf_conntrack usb_storage ntfs nls_base loop snd _pcm snd_timer snd soundcore snd_page_alloc pcspkr button i2c_viapro i2c_core sh pchp pci_hotplug via_agp agpgart evdev ext3 jbd mbcache raid456 async_xor async_ memcpy async_tx xor raid1 md_mod sd_mod ide_pci_generic usbhid hid ff_memless vi a82cxxx ide_core sata_promise via_rhine mii ata_generic libata ehci_hcd uhci_hcd usbcore scsi_mod dock thermal processor fan thermal_sys
[1406330.496019]
[1406330.496019] Pid: 4612, comm: snmpd Not tainted (2.6.26-1-686 #1)
[1406330.496019] EIP: 0060:[<c023db1e>] EFLAGS: 00200202 CPU: 0
[1406330.496019] EIP is at get_stats+0x2e/0x50
[1406330.496019] EAX: f74523c0 EBX: f74cc858 ECX: 00000000 EDX: c03c48c0
[1406330.496019] ESI: 00000000 EDI: 05a5734f EBP: 08badbff ESP: eb96df0c
[1406330.496019] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[1406330.496019] CR0: 8005003b CR2: b7fb0000 CR3: 2b951000 CR4: 00000690
[1406330.496019] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[1406330.496019] DR6: ffff0ff0 DR7: 00000400
[1406330.496019] [<c0257412>] ? dev_seq_show+0x1c/0x7b
[1406330.496019] [<c018a7b3>] ? seq_read+0x196/0x26f
[1406330.496019] [<c018a61d>] ? seq_read+0x0/0x26f
[1406330.496019] [<c01a12b2>] ? proc_reg_read+0x58/0x6b
[1406330.496019] [<c01a125a>] ? proc_reg_read+0x0/0x6b
[1406330.496019] [<c017499e>] ? vfs_read+0x81/0x11e
[1406330.496019] [<c0174def>] ? sys_read+0x3c/0x63
[1406330.496019] [<c0103853>] ? sysenter_past_esp+0x78/0xb1
[1406330.496019] =======================
[1406413.424020] BUG: soft lockup - CPU#0 stuck for 74s! [snmpd:4612]
[1406413.424020] Modules linked in: ppdev parport_pc lp parport sit tunnel4 ipt_ MASQUERADE xt_multiport xt_tcpudp xt_state ip6table_filter iptable_nat nf_nat nf _conntrack_ipv4 iptable_filter ip_tables jfs dm_snapshot dm_mirror dm_log dm_mod ip6t_rt ip6table_mangle ip6_tables x_tables ipv6 cpufreq_ondemand acpi_cpufreq freq_table fuse nf_conntrack_ftp nf_conntrack usb_storage ntfs nls_base loop snd _pcm snd_timer snd soundcore snd_page_alloc pcspkr button i2c_viapro i2c_core sh pchp pci_hotplug via_agp agpgart evdev ext3 jbd mbcache raid456 async_xor async_ memcpy async_tx xor raid1 md_mod sd_mod ide_pci_generic usbhid hid ff_memless vi a82cxxx ide_core sata_promise via_rhine mii ata_generic libata ehci_hcd uhci_hcd usbcore scsi_mod dock thermal processor fan thermal_sys
[1406413.424020]
[1406413.424020] Pid: 4612, comm: snmpd Not tainted (2.6.26-1-686 #1)
[1406413.424020] EIP: 0060:[<c01713ba>] EFLAGS: 00200246 CPU: 0
[1406413.424020] EIP is at kmem_cache_free+0x4c/0x4f
[1406413.424020] EAX: 00200246 EBX: d9463a5c ECX: f74045c0 EDX: d9463a5c
[1406413.424020] ESI: 00200246 EDI: d9463a5c EBP: d9463a5c ESP: eb96df60
[1406413.424020] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[1406413.424020] CR0: 8005003b CR2: b7fb0000 CR3: 2b951000 CR4: 00000690
[1406413.424020] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[1406413.424020] DR6: ffff0ff0 DR7: 00000400
[1406413.424020] [<c0182383>] ? d_kill+0x37/0x46
[1406413.424020] [<c0182631>] ? dput+0xb4/0xbb
[1406413.424020] [<c017518e>] ? __fput+0x111/0x135
[1406413.424020] [<c0172ad1>] ? filp_close+0x4d/0x53
[1406413.424020] [<c0173c0b>] ? sys_close+0x5b/0x8d
[1406413.424020] [<c0103853>] ? sysenter_past_esp+0x78/0xb1
[1406413.424020] =======================
[1406713.904021] BUG: soft lockup - CPU#0 stuck for 92s! [master:21417]
[1406713.904021] Modules linked in: ppdev parport_pc lp parport sit tunnel4 ipt_ MASQUERADE xt_multiport xt_tcpudp xt_state ip6table_filter iptable_nat nf_nat nf _conntrack_ipv4 iptable_filter ip_tables jfs dm_snapshot dm_mirror dm_log dm_mod ip6t_rt ip6table_mangle ip6_tables x_tables ipv6 cpufreq_ondemand acpi_cpufreq freq_table fuse nf_conntrack_ftp nf_conntrack usb_storage ntfs nls_base loop snd _pcm snd_timer snd soundcore snd_page_alloc pcspkr button i2c_viapro i2c_core sh pchp pci_hotplug via_agp agpgart evdev ext3 jbd mbcache raid456 async_xor async_ memcpy async_tx xor raid1 md_mod sd_mod ide_pci_generic usbhid hid ff_memless vi a82cxxx ide_core sata_promise via_rhine mii ata_generic libata ehci_hcd uhci_hcd usbcore scsi_mod dock thermal processor fan thermal_sys
[1406713.904021]
[1406713.904021] Pid: 21417, comm: master Not tainted (2.6.26-1-686 #1)
[1406713.904021] EIP: 0060:[<f8937474>] EFLAGS: 00000246 CPU: 0
[1406713.904021] EIP is at do_get_write_access+0x1/0x331 [jbd]
[1406713.904021] EAX: d81453ec EBX: d81453ec ECX: 00000000 EDX: d86b8da0
[1406713.904021] ESI: d86b8da0 EDI: f70376a0 EBP: f89acc90 ESP: d7dcfdac
[1406713.904021] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[1406713.904021] CR0: 8005003b CR2: b7ab2940 CR3: 191a7000 CR4: 00000690
[1406713.904021] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[1406713.904021] DR6: ffff0ff0 DR7: 00000400
[1406713.904021] [<f89377bc>] ? journal_get_write_access+0x18/0x26 [jbd]
[1406713.904021] [<f89aa4ff>] ? __ext3_journal_get_write_access+0x13/0x32 [ext3 ]
[1406713.904021] [<f899f71a>] ? ext3_reserve_inode_write+0x2d/0x5d [ext3]
[1406713.904021] [<f899f75b>] ? ext3_mark_inode_dirty+0x11/0x27 [ext3]
[1406713.904021] [<f89a1f04>] ? ext3_dirty_inode+0x50/0x63 [ext3]
[1406713.904021] [<c018cc2b>] ? __mark_inode_dirty+0x21/0x12a
[1406713.904021] [<c0184f0a>] ? touch_atime+0xc7/0xd1
[1406713.904021] [<c0158608>] ? generic_file_aio_read+0x493/0x4da
[1406713.904021] [<c017420e>] ? do_sync_read+0xbf/0xfe
[1406713.904021] [<c01318e0>] ? autoremove_wake_function+0x0/0x2d
[1406713.904021] [<f89ac90d>] ? ext3_xattr_security_get+0x31/0x3b [ext3]
[1406713.904021] [<c01b94df>] ? security_file_permission+0xc/0xd
[1406713.904021] [<c017414f>] ? do_sync_read+0x0/0xfe
[1406713.904021] [<c017499e>] ? vfs_read+0x81/0x11e
[1406713.904021] [<c017798e>] ? kernel_read+0x32/0x43
[1406713.904021] [<c0177a5f>] ? prepare_binprm+0xc0/0xc4
[1406713.904021] [<c017884b>] ? do_execve+0xd2/0x1c6
[1406713.904021] [<c010213b>] ? sys_execve+0x2a/0x4a
[1406713.904021] [<c0103853>] ? sysenter_past_esp+0x78/0xb1
[1406713.904021] =======================
[1406883.492020] BUG: soft lockup - CPU#0 stuck for 86s! [snmpd:4612]
[1406883.492020] Modules linked in: ppdev parport_pc lp parport sit tunnel4 ipt_ MASQUERADE xt_multiport xt_tcpudp xt_state ip6table_filter iptable_nat nf_nat nf _conntrack_ipv4 iptable_filter ip_tables jfs dm_snapshot dm_mirror dm_log dm_mod ip6t_rt ip6table_mangle ip6_tables x_tables ipv6 cpufreq_ondemand acpi_cpufreq freq_table fuse nf_conntrack_ftp nf_conntrack usb_storage ntfs nls_base loop snd _pcm snd_timer snd soundcore snd_page_alloc pcspkr button i2c_viapro i2c_core sh pchp pci_hotplug via_agp agpgart evdev ext3 jbd mbcache raid456 async_xor async_ memcpy async_tx xor raid1 md_mod sd_mod ide_pci_generic usbhid hid ff_memless vi a82cxxx ide_core sata_promise via_rhine mii ata_generic libata ehci_hcd uhci_hcd usbcore scsi_mod dock thermal processor fan thermal_sys
[1406883.492020]
[1406883.492020] Pid: 4612, comm: snmpd Not tainted (2.6.26-1-686 #1)
[1406883.492020] EIP: 0060:[<c025907f>] EFLAGS: 00200246 CPU: 0
[1406883.492020] EIP is at dev_load+0x1/0x3f
[1406883.492020] EAX: c041d060 EBX: eb96df30 ECX: 00000000 EDX: eb96df30
[1406883.492020] ESI: 00008933 EDI: c041d060 EBP: 00000000 ESP: eb96def8
[1406883.492020] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[1406883.492020] CR0: 8005003b CR2: b7fa7000 CR3: 2b951000 CR4: 00000690
[1406883.492020] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[1406883.492020] DR6: ffff0ff0 DR7: 00000400
[1406883.492020] [<c0259b3a>] dev_ioctl+0x2e1/0x571
[1406883.492020] [<c0183039>] d_alloc+0x138/0x17a
[1406883.492020] [<c0197c77>] inotify_d_instantiate+0xf/0x31
[1406883.492020] [<c024e8bb>] sock_ioctl+0x19f/0x1c1
[1406883.492020] [<c024e71c>] sock_ioctl+0x0/0x1c1
[1406883.492020] [<c017e404>] vfs_ioctl+0x1c/0x5d
[1406883.492020] [<c017e68f>] do_vfs_ioctl+0x24a/0x261
[1406883.492020] [<c017e6e7>] sys_ioctl+0x41/0x5a
[1406883.492020] [<c0103853>] sysenter_past_esp+0x78/0xb1
[1406883.492020] =======================
[1407427.664022] BUG: soft lockup - CPU#0 stuck for 101s! [pickup:21567]
[1407427.664022] Modules linked in: ppdev parport_pc lp parport sit tunnel4 ipt_ MASQUERADE xt_multiport xt_tcpudp xt_state ip6table_filter iptable_nat nf_nat nf _conntrack_ipv4 iptable_filter ip_tables jfs dm_snapshot dm_mirror dm_log dm_mod ip6t_rt ip6table_mangle ip6_tables x_tables ipv6 cpufreq_ondemand acpi_cpufreq freq_table fuse nf_conntrack_ftp nf_conntrack usb_storage ntfs nls_base loop snd _pcm snd_timer snd soundcore snd_page_alloc pcspkr button i2c_viapro i2c_core sh pchp pci_hotplug via_agp agpgart evdev ext3 jbd mbcache raid456 async_xor async_ memcpy async_tx xor raid1 md_mod sd_mod ide_pci_generic usbhid hid ff_memless vi a82cxxx ide_core sata_promise via_rhine mii ata_generic libata ehci_hcd uhci_hcd usbcore scsi_mod dock thermal processor fan thermal_sys
[1407427.664022]
[1407427.664022] Pid: 21567, comm: pickup Not tainted (2.6.26-1-686 #1)
[1407427.664022] EIP: 0060:[<c01318a8>] EFLAGS: 00000246 CPU: 0
[1407427.664022] EIP is at __wake_up_bit+0xc/0x2e
[1407427.664022] EAX: c17c3254 EBX: c17c3250 ECX: 00000000 EDX: c17bd800
[1407427.664022] ESI: 00000000 EDI: c13663ec EBP: ebb11580 ESP: dd01be90
[1407427.664022] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[1407427.664022] CR0: 8005003b CR2: b7a66048 CR3: 190fd000 CR4: 00000690
[1407427.664022] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[1407427.664022] DR6: ffff0ff0 DR7: 00000400
[1407427.664022] [<c01568dd>] ? unlock_page+0x19/0x23
[1407427.664022] [<c0161e72>] ? __do_fault+0x30e/0x34d
[1407427.664022] [<c0163d52>] ? handle_mm_fault+0x2d2/0x690
[1407427.664022] [<c0166dc4>] ? do_mmap_pgoff+0x266/0x2b9
[1407427.664022] [<c0115b4f>] ? do_page_fault+0x29b/0x5b8
[1407427.664022] [<c0106a5e>] ? sys_mmap2+0x96/0xa0
[1407427.664022] [<c01158b4>] ? do_page_fault+0x0/0x5b8
[1407427.664022] [<c02b9832>] ? error_code+0x72/0x78
[1407427.664022] =======================
[1408358.456019] BUG: soft lockup - CPU#0 stuck for 65s! [master:21848]
[1408358.456019] Modules linked in: ppdev parport_pc lp parport sit tunnel4 ipt_ MASQUERADE xt_multiport xt_tcpudp xt_state ip6table_filter iptable_nat nf_nat nf _conntrack_ipv4 iptable_filter ip_tables jfs dm_snapshot dm_mirror dm_log dm_mod ip6t_rt ip6table_mangle ip6_tables x_tables ipv6 cpufreq_ondemand acpi_cpufreq freq_table fuse nf_conntrack_ftp nf_conntrack usb_storage ntfs nls_base loop snd _pcm snd_timer snd soundcore snd_page_alloc pcspkr button i2c_viapro i2c_core sh pchp pci_hotplug via_agp agpgart evdev ext3 jbd mbcache raid456 async_xor async_ memcpy async_tx xor raid1 md_mod sd_mod ide_pci_generic usbhid hid ff_memless vi a82cxxx ide_core sata_promise via_rhine mii ata_generic libata ehci_hcd uhci_hcd usbcore scsi_mod dock thermal processor fan thermal_sys
[1408358.456019]
[1408358.456019] Pid: 21848, comm: master Not tainted (2.6.26-1-686 #1)
[1408358.456019] EIP: 0060:[<c0163c85>] EFLAGS: 00000216 CPU: 0
[1408358.456019] EIP is at handle_mm_fault+0x205/0x690
[1408358.456019] EAX: 19195067 EBX: c1000000 ECX: 19195067 EDX: ddc34bfc
[1408358.456019] ESI: 3c7ed067 EDI: c178fda0 EBP: c13232ac ESP: d92bfe7c
[1408358.456019] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[1408358.456019] CR0: 8005003b CR2: b7ab2940 CR3: 1ebe4000 CR4: 00000690
[1408358.456019] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[1408358.456019] DR6: ffff0ff0 DR7: 00000400
[1408358.456019] [<c01623a3>] ? follow_page+0x53/0x1d4
[1408358.456019] [<c0164342>] ? get_user_pages+0x232/0x334
[1408358.456019] [<c01774f9>] ? get_arg_page+0x2b/0x7b
[1408358.456019] [<c0177616>] ? copy_strings+0xcd/0x173
[1408358.456019] [<c01776d5>] ? copy_strings_kernel+0x19/0x27
[1408358.456019] [<c0178867>] ? do_execve+0xee/0x1c6
[1408358.456019] [<c010213b>] ? sys_execve+0x2a/0x4a
[1408358.456019] [<c0103853>] ? sysenter_past_esp+0x78/0xb1
[1408358.456019] =======================
[1408494.496018] BUG: soft lockup - CPU#0 stuck for 116s! [pickup:21850]
[1408494.496018] Modules linked in: ppdev parport_pc lp parport sit tunnel4 ipt_ MASQUERADE xt_multiport xt_tcpudp xt_state ip6table_filter iptable_nat nf_nat nf _conntrack_ipv4 iptable_filter ip_tables jfs dm_snapshot dm_mirror dm_log dm_mod ip6t_rt ip6table_mangle ip6_tables x_tables ipv6 cpufreq_ondemand acpi_cpufreq freq_table fuse nf_conntrack_ftp nf_conntrack usb_storage ntfs nls_base loop snd _pcm snd_timer snd soundcore snd_page_alloc pcspkr button i2c_viapro i2c_core sh pchp pci_hotplug via_agp agpgart evdev ext3 jbd mbcache raid456 async_xor async_ memcpy async_tx xor raid1 md_mod sd_mod ide_pci_generic usbhid hid ff_memless vi a82cxxx ide_core sata_promise via_rhine mii ata_generic libata ehci_hcd uhci_hcd usbcore scsi_mod dock thermal processor fan thermal_sys
[1408494.496018]
[1408494.496018] Pid: 21850, comm: pickup Not tainted (2.6.26-1-686 #1)
[1408494.496018] EIP: 0060:[<c010388c>] EFLAGS: 00000282 CPU: 0
[1408494.496018] EIP is at system_call+0x0/0x3b
[1408494.496018] EAX: 00000006 EBX: 00000007 ECX: b7bd8e64 EDX: b7f78ff4
[1408494.496018] ESI: 00000000 EDI: b7bf12d0 EBP: bff7a464 ESP: d89ddfe4
[1408494.496018] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[1408494.496018] CR0: 8005003b CR2: b7bd8dc8 CR3: 1c52c000 CR4: 00000690
[1408494.496018] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[1408494.496018] DR6: ffff0ff0 DR7: 00000400
[1408494.496018] =======================
Reply to: