Bug#536236: kernel: [56585.121000] BUG: soft lockup - CPU#1 stuck for 61s! [pdflush:210]
I thought I stated in my email that I had gotten this with both kernels. I have a server called web06 that has "Linux version 2.6.30-1-amd64 (Debian 2.6.30-1) (waldi@debian.org) (gcc version 4.3.3 (Debian 4.3.3-10) ) #1 SMP Sun Jun 14 15:00:29 UTC 2009" as it's kernel and then web02 which I sent the bug report from has "Linux version 2.6.30-bpo.1-amd64 (Debian 2.6.30-1~bpo50+1) (nobse@debian.org) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) #1 SMP Fri Jun 26 09:41:55 UTC 2009"
We can focus on web06 if you like, the errors from that are slightly different and I was using FSCACHE with my NFS. Here is the error from syslog.
Jul 2 16:35:40 web06 kernel: [ 6045.449709] BUG: unable to handle kernel NULL pointer dereference at 0000000000000040
Jul 2 16:35:40 web06 kernel: [ 6045.449779] IP: [<ffffffffa02737b2>] fscache_object_slow_work_execute+0x232/0x6bd [fscache]
Jul 2 16:35:40 web06 kernel: [ 6045.449857] PGD 7d516067 PUD 7d517067 PMD 0
Jul 2 16:35:40 web06 kernel: [ 6045.449898] Oops: 0002 [#2] SMP
Jul 2 16:35:40 web06 kernel: [ 6045.449933] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.3/i2c-adapter/i2c-0/name
Jul 2 16:35:40 web06 kernel: [ 6045.449989] CPU 1
Jul 2 16:35:40 web06 kernel: [ 6045.450016] Modules linked in: ip6table_filter ip6_tables iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_LOG ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_CONNMARK xt_connmark xt_CLASSIFY xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle iptable_filter ip_tables x_tables nfnetlink cachefiles nfs lockd fscache nfs_acl auth_rpcgss sunr
Jul 2 16:35:40 web06 kernel: pc tcp_bic loop evdev dcdbas psmouse snd_pcm snd_timer snd soundcore serio_raw i2c_i801 i2c_core rng_core snd_page_alloc pcspkr iTCO_wdt button processor i3000_edac edac_core shpchp pci_hotplug jfs nls_base raid1 md_mod sd_mod crc_t10dif piix ide_pci_generic ide_core ata_piix ata_generic libata scsi_mod ehci_hcd uhci_hcd tg3 libphy thermal fan thermal_sys
Jul 2 16:35:40 web06 kernel: [ 6045.450916] Pid: 4400, comm: kslowd Tainted: G D 2.6.30-1-amd64 #1 PowerEdge 850
Jul 2 16:35:40 web06 kernel: [ 6045.450971] RIP: 0010:[<ffffffffa02737b2>] [<ffffffffa02737b2>] fscache_object_slow_work_execute+0x232/0x6bd [fscache]
Jul 2 16:35:40 web06 kernel: [ 6045.451047] RSP: 0018:ffff8800acd01e90 EFLAGS: 00010202
Jul 2 16:35:40 web06 kernel: [ 6045.451079] RAX: 0000000000000040 RBX: ffff88007b0f3878 RCX: 0000000000000001
Jul 2 16:35:40 web06 kernel: [ 6045.451116] RDX: 000000000000000b RSI: 0000000000000003 RDI: ffff88007b0f381c
Jul 2 16:35:40 web06 kernel: [ 6045.451152] RBP: ffff88007b0f3830 R08: 0000000000000000 R09: ffff8800acd01df0
Jul 2 16:35:40 web06 kernel: [ 6045.451188] R10: 0000000000000032 R11: ffffffffa02dd569 R12: ffff88007b0f3878
Jul 2 16:35:40 web06 kernel: [ 6045.451225] R13: ffff88007b0f3800 R14: ffff8800acd01ed0 R15: 0000000000000000
Jul 2 16:35:40 web06 kernel: [ 6045.451262] FS: 0000000000000000(0000) GS:ffff880028037000(0000) knlGS:0000000000000000
Jul 2 16:35:40 web06 kernel: [ 6045.451317] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Jul 2 16:35:40 web06 kernel: [ 6045.451350] CR2: 0000000000000040 CR3: 000000007d515000 CR4: 00000000000006e0
Jul 2 16:35:40 web06 kernel: [ 6045.451387] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jul 2 16:35:40 web06 kernel: [ 6045.451423] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jul 2 16:35:40 web06 kernel: [ 6045.451460] Process kslowd (pid: 4400, threadinfo ffff8800acd00000, task ffff88010a061910)
Jul 2 16:35:40 web06 kernel: [ 6045.451515] Stack:
Jul 2 16:35:40 web06 kernel: [ 6045.451539] ffff88007b0f3878 0000000000000001 0000000000000032 0000000000000064
Jul 2 16:35:40 web06 kernel: [ 6045.451582] ffff8800acd01ed0 ffffffff8029100b 0000000000000000 ffffffff802348e4
Jul 2 16:35:40 web06 kernel: [ 6045.451646] 0000000000000000 ffff88010a061910 ffffffff802546e2 ffff8800acd01ee8
Jul 2 16:35:40 web06 kernel: [ 6045.451730] Call Trace:
Jul 2 16:35:40 web06 kernel: [ 6045.451756] [<ffffffff8029100b>] ? slow_work_thread+0x2a6/0x467
Jul 2 16:35:40 web06 kernel: [ 6045.451797] [<ffffffff802348e4>] ? __wake_up_common+0x44/0x73
Jul 2 16:35:40 web06 kernel: [ 6045.451836] [<ffffffff802546e2>] ? autoremove_wake_function+0x0/0x2e
Jul 2 16:35:40 web06 kernel: [ 6045.451874] [<ffffffff80290d65>] ? slow_work_thread+0x0/0x467
Jul 2 16:35:40 web06 kernel: [ 6045.451911] [<ffffffff80254326>] ? kthread+0x54/0x80
Jul 2 16:35:40 web06 kernel: [ 6045.451947] [<ffffffff80210aca>] ? child_rip+0xa/0x20
Jul 2 16:35:40 web06 kernel: [ 6045.451983] [<ffffffff802542d2>] ? kthread+0x0/0x80
Jul 2 16:35:40 web06 kernel: [ 6045.452018] [<ffffffff80210ac0>] ? child_rip+0x0/0x20
Jul 2 16:35:40 web06 kernel: [ 6045.452052] Code: b8 41 23 44 24 b0 0f bd c8 0f 44 ca 39 d1 0f 8d a9 03 00 00 e9 4e 04 00 00 49 8d 7d 1c e8 33 17 24 e0 49 8b 44 24 f0 48 83 c0 40 <f0> 0f ba 30 01 19 d2 85 d2 74 13 49 8b 7c 24 f0 be 01 00 00 00
Jul 2 16:35:40 web06 kernel: [ 6045.452309] RIP [<ffffffffa02737b2>] fscache_object_slow_work_execute+0x232/0x6bd [fscache]
Jul 2 16:35:40 web06 kernel: [ 6045.452376] RSP <ffff8800acd01e90>
Jul 2 16:35:40 web06 kernel: [ 6045.452404] CR2: 0000000000000040
Jul 2 16:35:40 web06 kernel: [ 6045.452799] ---[ end trace e37ac09e11c3d6a2 ]---
Here is the complete FSTAB:
# /etc/fstab: static file system information.
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
/dev/md2 / jfs noatime,errors=remount-ro 0 1
/dev/md0 /boot jfs noatime 0 2
/dev/md1 none swap sw 0 0
/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto 0 0
/dev/scd0 /media/floppy0 auto rw,user,noauto 0 0
10.0.0.1:/export/home /home nfs rw,rsize=8192,wsize=8192,intr,noatime 0 0
10.0.0.1:/export/sites/content /var/www nfs rw,rsize=32768,wsize=32768,intr,noatime 0 0
10.0.0.1:/export/sites/content/sessions /var/phpsessions nfs rw,rsize=32768,wsize=32768,intr,noatime 0 0
10.0.0.1:/export/sites/conf /etc/apache2/sites-available nfs rw 0 0
web06:~# l /proc/mounts
lrwxrwxrwx 1 root root 11 2009-07-08 10:34 /proc/mounts -> self/mounts
web06:~# mount
/dev/md2 on / type jfs (rw,noatime,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
procbususb on /proc/bus/usb type usbfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/md0 on /boot type jfs (rw,noatime)
10.0.0.1:/export/home on /home type nfs (rw,noatime,rsize=8192,wsize=8192,intr,addr=10.0.0.1)
10.0.0.1:/export/sites/content on /var/www type nfs (rw,noatime,rsize=32768,wsize=32768,intr,addr=10.0.0.1)
10.0.0.1:/export/sites/content/sessions on /var/phpsessions type nfs (rw,noatime,rsize=32768,wsize=32768,intr,addr=10.0.0.1)
10.0.0.1:/export/sites/conf on /etc/apache2/sites-available type nfs (rw,addr=10.0.0.1)
web06:~# fdisk -l
Disk /dev/sda: 164.6 GB, 164696555520 bytes
255 heads, 63 sectors/track, 20023 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000c08d3
Device Boot Start End Blocks Id System
/dev/sda1 * 1 12 96358+ fd Linux raid autodetect
/dev/sda2 13 510 4000185 fd Linux raid autodetect
/dev/sda3 511 20023 156738172+ fd Linux raid autodetect
Disk /dev/sdb: 164.6 GB, 164696555520 bytes
255 heads, 63 sectors/track, 20023 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000d1c60
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 12 96358+ fd Linux raid autodetect
/dev/sdb2 13 510 4000185 fd Linux raid autodetect
/dev/sdb3 511 20023 156738172+ fd Linux raid autodetect
Disk /dev/md0: 98 MB, 98566144 bytes
2 heads, 4 sectors/track, 24064 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000
Disk /dev/md0 doesn't contain a valid partition table
Disk /dev/md1: 4096 MB, 4096065536 bytes
2 heads, 4 sectors/track, 1000016 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000
Disk /dev/md1 doesn't contain a valid partition table
Disk /dev/md2: 160.4 GB, 160499761152 bytes
2 heads, 4 sectors/track, 39184512 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000
Disk /dev/md2 doesn't contain a valid partition table
Reply to: