Bug#278161: XFS panic under heavy loads on Debian Sarge
Title: Message
Package: kernel-image-2.6.8-1-686-smp
Version:
2.6.8-4
Error message:
-- start syslog dump --
2004-10-22T18:45:13+0100 <kern.emerg>
ispluxslin04 kernel: Bad page state at free_hot_cold_page (in process 'dbench',
page c134cb00)
2004-10-22T18:45:13+0100 <kern.emerg> ispluxslin04
kernel: flags:0x2000000c mapping:f7b4f610 mapcount:0
count:1
2004-10-22T18:45:13+0100 <kern.emerg> ispluxslin04 kernel:
Backtrace:
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
[<c0141fdb>] bad_page+0x6b/0xa0
2004-10-22T18:45:13+0100
<kern.warning> ispluxslin04 kernel: [<c01427d0>]
free_hot_cold_page+0x70/0x140
2004-10-22T18:45:13+0100 <kern.warning>
ispluxslin04 kernel: [<f8a2d9b7>] pagebuf_free+0x97/0xf0
[xfs]
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
[<f8a1eb4d>] xfs_trans_push_ail+0x14d/0x3a0
[xfs]
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
[<f8a0f7aa>] xlog_grant_push_ail+0x15a/0x1b0
[xfs]
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
[<f8a0e4e9>] xfs_log_reserve+0x69/0xf0 [xfs]
2004-10-22T18:45:13+0100
<kern.warning> ispluxslin04 kernel: [<f8a1d634>]
xfs_trans_reserve+0x84/0x1f0 [xfs]
2004-10-22T18:45:13+0100
<kern.warning> ispluxslin04 kernel: [<f8a06c16>]
xfs_itruncate_finish+0x2b6/0x460 [xfs]
2004-10-22T18:45:13+0100
<kern.warning> ispluxslin04 kernel: [<f8a2529c>]
xfs_inactive_free_eofblocks+0x28c/0x2d0 [xfs]
2004-10-22T18:45:13+0100
<kern.warning> ispluxslin04 kernel: [<f8a259cd>]
xfs_release+0x8d/0xe0 [xfs]
2004-10-22T18:45:13+0100 <kern.warning>
ispluxslin04 kernel: [<f8a30490>] linvfs_release+0x0/0x30
[xfs]
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
[<f8a304ad>] linvfs_release+0x1d/0x30 [xfs]
2004-10-22T18:45:13+0100
<kern.warning> ispluxslin04 kernel: [<c0160aaa>]
__fput+0x12a/0x140
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04
kernel: [<c015eec9>] filp_close+0x59/0x90
2004-10-22T18:45:13+0100
<kern.warning> ispluxslin04 kernel: [<c015ef6e>]
sys_close+0x6e/0xc0
2004-10-22T18:45:13+0100 <kern.warning>
ispluxslin04 kernel: [<c01061fb>]
syscall_call+0x7/0xb
2004-10-22T18:45:13+0100 <kern.emerg> ispluxslin04
kernel: Trying to fix it up, but a reboot is needed
2004-10-22T18:45:13+0100
<kern.warning> ispluxslin04 kernel: ------------[ cut here
]------------
2004-10-22T18:45:13+0100 <kern.alert> ispluxslin04
kernel: kernel BUG at include/linux/mm.h:312!
2004-10-22T18:45:13+0100
<kern.alert> ispluxslin04 kernel: invalid operand: 0000
[#1]
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
PREEMPT SMP
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04
kernel: Modules linked in: ipv6 cpqphp shpchp pciehp pci_hotplug ohci_hcd
usbcore serverworks sworks_agp agpgart evdev aic7xxx tg3 firmware_class xfs
dm_mod capability commoncap ide_cd ide_core cdrom rtc ext3 jbd mbcache cciss
scsi_mod unix font vesafb cfbcopyarea cfbimgblt
cfbfillrect
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04
kernel: CPU: 4
2004-10-22T18:45:13+0100
<kern.warning> ispluxslin04 kernel: EIP:
0060:[<f8a2d9b9>] Not
tainted
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
EFLAGS: 00010256 (2.6.8-1-686-smp)
2004-10-22T18:45:13+0100
<kern.warning> ispluxslin04 kernel: EIP is at pagebuf_free+0x99/0xf0
[xfs]
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel: eax:
00000000 ebx: 00000000 ecx: 00000001 edx:
c134cb00
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
esi: f5ce6580 edi: f5ce65e4 ebp: f61b4000
esp: f61b5ee4
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04
kernel: ds: 007b es: 007b ss:
0068
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
Process xfslogd/4 (pid: 533, threadinfo=f61b4000
task=f685b8b0)
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04
kernel: Stack: f4335050 f5ce6580 f4335050 f5ce6580 f89e7e49 f5ce6580 00000000
00000000
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
00000000 c2d8d000 f685a230 c2d8dca0 00000004 00000000 561f1180
000f4f04
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
c2d8e150 f685b8b0 f685ba60 f5ce6580 00000297 f5ce65e4 f8a2e9bd
f5ce6580
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
Call Trace:
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04
kernel: [<f89e7e49>] xfs_buf_iodone_callbacks+0x179/0x180
[xfs]
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
[<f8a2e9bd>] pagebuf_iodone_work+0x4d/0x50
[xfs]
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04 kernel:
[<c0131a26>] worker_thread+0x1f6/0x2e0
2004-10-22T18:45:13+0100
<kern.warning> ispluxslin04 kernel: [<f8a2e970>]
pagebuf_iodone_work+0x0/0x50 [xfs]
2004-10-22T18:45:13+0100
<kern.warning> ispluxslin04 kernel: [<c011c4f0>]
default_wake_function+0x0/0x20
2004-10-22T18:45:13+0100 <kern.warning>
ispluxslin04 kernel: [<c011c4f0>]
default_wake_function+0x0/0x20
2004-10-22T18:45:13+0100 <kern.warning>
ispluxslin04 kernel: [<c0131830>]
worker_thread+0x0/0x2e0
2004-10-22T18:45:13+0100 <kern.warning>
ispluxslin04 kernel: [<c0135f8a>]
kthread+0xba/0xc0
2004-10-22T18:45:13+0100 <kern.warning> ispluxslin04
kernel: [<c0135ed0>] kthread+0x0/0xc0
2004-10-22T18:45:13+0100
<kern.warning> ispluxslin04 kernel: [<c01042c5>]
kernel_thread_helper+0x5/0x10
2004-10-22T18:45:13+0100 <kern.warning>
ispluxslin04 kernel: Code: 0f 0b 38 01 49 9b a3 f8 eb b3 0f b7 96 cc 00 00 00 8b
46 5c
-- end syslog dump --
To reproduce:
-- start stresstest.sh
--
#!/bin/bash
PATH=/bin:/sbin:/usr/bin:/usr/sbin
export
PATH
if [ ! -d /root/results ] ;
then
mkdir
/root/results
fi
yes | mkfs -t xfs -f /dev/cciss/c0d0p5
echo "Filesystem created"
mount -t xfs
/dev/cciss/c0d0p5 /freespace
echo "Filesystem mounted"
mkdir
/freespace/clients
echo "Test directory created"
cd /freespace
echo -n "Stress test"
echo
"0 0" >
/root/results/dbench_xfs.txt
for u in 1 `seq 4 4 150` 150
do
echo
-n "."
dbench $u | awk ' /^Throughput/ { printf "%i\t%s\n",$4,$2 } '
>> /root/results/dbench_xfs.txt
done
echo "[done]"
cd /
umount
/freespace
echo "Filesystem unmounted"
-- end stresstest.sh --
Description of the problem:
When trying to compare the performance of different
filesystems for samba (using dbench), I hit a
problem with XFS when running
it with 80 "users". The server panics at that point. I tried with
different
filesystems (JFS/ReiserFS/Ext3/Ext2/XFS) and kernels (2.6.6-1 to 2.6.8-1) on two
similar
machines with different root filesystems (Ext3 and ReiserFS). The
problem only occurs when the
data filesystem is XFS.
Here comes the output of the script when running
against XFS:
-- start dbench_xfs.txt --
0
0
1
168.003
4
860.609
8
842.853
12
794.776
16
796.099
20
640.791
24
535.983
28
457.429
32
421.748
36
387.846
40
327.709
44
328.408
48
335.684
52
312.847
56
304.233
60
287.863
64
284.352
68
276.887
72
288.772
76 255.893
-- end dbench_xfs.txt --
Debian information:
release: Sarge
kernel:
2.6.8-1-686-smp
libc6: 2.3.2.ds1-13
Hardware information:
Server model: HP Proliant ML570
CPU model:
Intel(R) Xeon(TM) MP CPU 2.80GHz
Number of CPUs: 4
Memory installed:
4GB
Disk controller: RAID bus controller: Compaq Computer Corporation Smart
Array 64xx (rev 01)
Array configuration: Raid 1
Reply to: