XFS frequent crashes on PE1950 with perc 5/e and 2xMD1000
Hello!
I have a problem with system in the configuration described in subject
(Dell PE1950 III + PERC 5/E + 2xMD1000)
System is running Debian Lenny AMD64 version with all available updates.
I have 6 VD's 2TB each (for 32bit system compatibility). Each VD is a
LVM2 PV
I made a LVM2 volume and formatted this into XFS. Previously it was only
one MD1000 connected to PERC controller.
But two days ago i added next MD1000 added new PV's to LVM2 and extended
XFS with xfs_growfs.
After some time I got kernel panic like this:
[46925.374954] Filesystem "dm-0": XFS internal error xfs_trans_cancel at
line 11
63 of file fs/xfs/xfs_trans.c. Caller 0xffffffffa02b2e82
[46925.374954] Pid: 12269, comm: smbd Not tainted 2.6.26-2-amd64 #1
[46925.374954]
[46925.374954] Call Trace:
[46925.374954] [<ffffffffa02b2e82>]
:xfs:xfs_iomap_write_allocate+0x360/0x385
[46925.374954] [<ffffffffa02c095e>] :xfs:xfs_trans_cancel+0x55/0xed
[46925.374954] [<ffffffffa02b2e82>]
:xfs:xfs_iomap_write_allocate+0x360/0x385
[46925.374954] [<ffffffffa02b38f4>] :xfs:xfs_iomap+0x21b/0x297
[46925.374954] [<ffffffffa02c9637>] :xfs:xfs_map_blocks+0x2d/0x5f
[46925.374954] [<ffffffffa02ca74e>] :xfs:xfs_page_state_convert+0x2a2/0x54f
[46925.374954] [<ffffffffa02cab5a>] :xfs:xfs_vm_writepage+0xb4/0xea
[46925.374954] [<ffffffff802770db>] __writepage+0xa/0x23
[46925.374954] [<ffffffff802775a0>] write_cache_pages+0x182/0x2b1
[46925.374954] [<ffffffff802770d1>] __writepage+0x0/0x23
[46925.374954] [<ffffffff8027770b>] do_writepages+0x20/0x2d
[46925.374954] [<ffffffff80271900>] __filemap_fdatawrite_range+0x51/0x5b
[46925.374954] [<ffffffffa02cd2b0>] :xfs:xfs_flush_pages+0x4e/0x6d
[46925.374954] [<ffffffffa02c5548>] :xfs:xfs_setattr+0x695/0xd28
[46925.374954] [<ffffffff803b1326>] sock_common_recvmsg+0x30/0x45
[46925.374954] [<ffffffffa02d08cc>] :xfs:xfs_write+0x6de/0x722
[46925.374954] [<ffffffffa02cf1e3>] :xfs:xfs_vn_setattr+0x11c/0x13a
[46925.374954] [<ffffffff802add8f>] notify_change+0x174/0x2f5
[46925.374954] [<ffffffff80299f09>] do_truncate+0x5e/0x79
[46925.374954] [<ffffffff8029df53>] sys_newfstat+0x20/0x29
[46925.374954] [<ffffffff8029a00e>] sys_ftruncate+0xea/0x107
[46925.374954] [<ffffffff8020beca>] system_call_after_swapgs+0x8a/0x8f
[46925.374954]
[46925.374954] xfs_force_shutdown(dm-0,0x8) called from line 1164 of
file fs/xfs
/xfs_trans.c. Return address = 0xffffffffa02c0977
[46925.374954] Filesystem "dm-0": Corruption of in-memory data detected.
Shutti
ng down filesystem: dm-0
[46925.376874] Please umount the filesystem, and rectify the problem(s)
[46934.390143] Filesystem "dm-0": xfs_log_force: error 5 returned.
[46925.374954] Filesystem "dm-0": XFS internal error xfs_trans_cancel at
line 11
63 of file fs/xfs/xfs_trans.c. Caller 0xffffffffa02b2e82
[46925.374954] Pid: 12269, comm: smbd Not tainted 2.6.26-2-amd64 #1
[46925.374954]
[46925.374954] Call Trace:
[46925.374954] [<ffffffffa02b2e82>]
:xfs:xfs_iomap_write_allocate+0x360/0x385
[46925.374954] [<ffffffffa02c095e>] :xfs:xfs_trans_cancel+0x55/0xed
[46925.374954] [<ffffffffa02b2e82>]
:xfs:xfs_iomap_write_allocate+0x360/0x385
[46925.374954] [<ffffffffa02b38f4>] :xfs:xfs_iomap+0x21b/0x297
[46925.374954] [<ffffffffa02c9637>] :xfs:xfs_map_blocks+0x2d/0x5f
[46925.374954] [<ffffffffa02ca74e>] :xfs:xfs_page_state_convert+0x2a2/0x54f
[46925.374954] [<ffffffffa02cab5a>] :xfs:xfs_vm_writepage+0xb4/0xea
[46925.374954] [<ffffffff802770db>] __writepage+0xa/0x23
[46925.374954] [<ffffffff802775a0>] write_cache_pages+0x182/0x2b1
[46925.374954] [<ffffffff802770d1>] __writepage+0x0/0x23
[46925.374954] [<ffffffff8027770b>] do_writepages+0x20/0x2d
[46925.374954] [<ffffffff80271900>] __filemap_fdatawrite_range+0x51/0x5b
[46925.374954] [<ffffffffa02cd2b0>] :xfs:xfs_flush_pages+0x4e/0x6d
[46925.374954] [<ffffffffa02c5548>] :xfs:xfs_setattr+0x695/0xd28
[46925.374954] [<ffffffff803b1326>] sock_common_recvmsg+0x30/0x45
[46925.374954] [<ffffffffa02d08cc>] :xfs:xfs_write+0x6de/0x722
[46925.374954] [<ffffffffa02cf1e3>] :xfs:xfs_vn_setattr+0x11c/0x13a
[46925.374954] [<ffffffff802add8f>] notify_change+0x174/0x2f5
[46925.374954] [<ffffffff80299f09>] do_truncate+0x5e/0x79
[46925.374954] [<ffffffff8029df53>] sys_newfstat+0x20/0x29
[46925.374954] [<ffffffff8029a00e>] sys_ftruncate+0xea/0x107
[46925.374954] [<ffffffff8020beca>] system_call_after_swapgs+0x8a/0x8f
[46925.374954]
[46925.374954] xfs_force_shutdown(dm-0,0x8) called from line 1164 of
file fs/xfs
/xfs_trans.c. Return address = 0xffffffffa02c0977
[46925.374954] Filesystem "dm-0": Corruption of in-memory data detected.
Shutti
ng down filesystem: dm-0
[46925.376874] Please umount the filesystem, and rectify the problem(s)
[46934.390143] Filesystem "dm-0": xfs_log_force: error 5 returned.
[47112.408317] Pid: 15211, comm: umount Tainted: G D
2.6.26-2-amd64 #1
[47112.408317]
[47112.408317] Call Trace:
[47112.408317] [<ffffffff80234a20>] warn_on_slowpath+0x51/0x7a
[47112.408317] [<ffffffff802b6c5b>] __mark_inode_dirty+0xe0/0x179
[47112.408317] [<ffffffff802460ef>] bit_waitqueue+0x10/0x97
[47112.424115] [<ffffffff802461b4>] wake_up_bit+0x11/0x22
[47112.424196] [<ffffffff802b6364>] __writeback_single_inode+0x44/0x29d
[47112.424280] [<ffffffff802b6928>] sync_sb_inodes+0x1b1/0x293
[47112.424362] [<ffffffff802b6aa4>] sync_inodes_sb+0x9a/0xa6
[47112.424445] [<ffffffff8029c6ed>] __fsync_super+0xb/0x6f
[47112.424527] [<ffffffff8029c75a>] fsync_super+0x9/0x16
[47112.424608] [<ffffffff8029c976>] generic_shutdown_super+0x21/0xee
[47112.424692] [<ffffffff8029ca50>] kill_block_super+0xd/0x1e
[47112.424773] [<ffffffff8029cb0c>] deactivate_super+0x5f/0x78
[47112.424855] [<ffffffff802afe06>] sys_umount+0x2f9/0x353
[47112.424938] [<ffffffff80221fac>] do_page_fault+0x5d8/0x9c8
[47112.428111] [<ffffffff8029e0e4>] sys_newstat+0x19/0x31
[47112.428111] [<ffffffff8031dc73>] __up_write+0x21/0x10e
[47112.428111] [<ffffffff8020beca>] system_call_after_swapgs+0x8a/0x8f
[47112.428111]
[47112.428111] ---[ end trace ba717a82a77cfd6a ]---
[47112.428111] Filesystem "dm-0": xfs_log_force: error 5 returned.
[47112.428111] Filesystem "dm-0": xfs_log_force: error 5 returned.
[47112.428111] xfs_force_shutdown(dm-0,0x1) called from line 420 of file
fs/xfs/
xfs_rw.c. Return address = 0xffffffffa02c8d33
[47112.428111] Filesystem "dm-0": xfs_log_force: error 5 returned.
[47112.428111] Filesystem "dm-0": xfs_log_force: error 5 returned.
[47112.428111] xfs_force_shutdown(dm-0,0x1) called from line 420 of file
fs/xfs/
xfs_rw.c. Return address = 0xffffffffa02c8d33
[47112.428177] ------------[ cut here ]------------
[47112.428246] WARNING: at fs/fs-writeback.c:381
__writeback_single_inode+0x44/0
x29d()
[47112.428345] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler
ipv6 xfs
ext2 mbcache loop snd_pcm snd_timer snd soundcore snd_page_alloc
rng_core psmous
e i5000_edac iTCO_wdt button pcspkr serio_raw edac_core shpchp
pci_hotplug dcdba
s evdev reiserfs dm_mirror dm_log dm_snapshot dm_mod raid1 md_mod sg
sr_mod cdro
m ide_pci_generic ide_core ses enclosure ata_piix sd_mod e1000e
megaraid_sas bnx
2 firmware_class ata_generic libata dock uhci_hcd ehci_hcd mptsas
mptscsih mptba
se scsi_transport_sas scsi_mod thermal processor fan thermal_sys
[47112.432208] Pid: 15211, comm: umount Tainted: G D W
2.6.26-2-amd64 #1
[47112.432294]
[47112.432304] Call Trace:
[47112.432443] [<ffffffff80234a20>] warn_on_slowpath+0x51/0x7a
[47112.432528] [<ffffffff80278daa>] pagevec_lookup_tag+0x1a/0x21
[47112.432614] [<ffffffff80271846>] wait_on_page_writeback_range+0xc8/0x113
[47112.432709] [<ffffffff802b6c5b>] __mark_inode_dirty+0xe0/0x179
[47112.432794] [<ffffffff802460ef>] bit_waitqueue+0x10/0x97
[47112.432876] [<ffffffff802461b4>] wake_up_bit+0x11/0x22
[47112.432959] [<ffffffff802b6364>] __writeback_single_inode+0x44/0x29d
[47112.433073] [<ffffffffa02c8d33>] :xfs:xfs_bwrite+0xb0/0xbb
[47112.433161] [<ffffffffa02b5993>] :xfs:xfs_log_need_covered+0x15/0x8c
[47112.433242] [<ffffffff802b6928>] sync_sb_inodes+0x1b1/0x293
[47112.433328] [<ffffffff802b6aa4>] sync_inodes_sb+0x9a/0xa6
[47112.433411] [<ffffffff8029c75a>] fsync_super+0x9/0x16
[47112.433493] [<ffffffff8029c976>] generic_shutdown_super+0x21/0xee
[47112.433577] [<ffffffff8029ca50>] kill_block_super+0xd/0x1e
[47112.433661] [<ffffffff8029cb0c>] deactivate_super+0x5f/0x78
[47112.433743] [<ffffffff802afe06>] sys_umount+0x2f9/0x353
[47112.433825] [<ffffffff80221fac>] do_page_fault+0x5d8/0x9c8
[47112.433908] [<ffffffff8029e0e4>] sys_newstat+0x19/0x31
[47112.433994] [<ffffffff8031dc73>] __up_write+0x21/0x10e
[47112.434078] [<ffffffff8020beca>] system_call_after_swapgs+0x8a/0x8f
[47112.434147]
[47112.434147] ---[ end trace ba717a82a77cfd6a ]---
[47113.504506] Filesystem "dm-0": xfs_log_force: error 5 returned.
[47113.504506] Filesystem "dm-0": xfs_log_force: error 5 returned.
[47113.504506] Filesystem "dm-0": xfs_log_force: error 5 returned.
[47113.504506] Filesystem "dm-0": xfs_log_force: error 5 returned.
[47113.506718] Filesystem "dm-0": xfs_log_force: error 5 returned.
[47113.516457] VFS: Busy inodes after unmount of dm-0. Self-destruct in
5 second
s. Have a nice day...
I've found some similar issues but no solution :(
here is my
$omreport storage vdisk controller=0
output:
List of Virtual Disks on Controller PERC 5/E Adapter (Slot 1)
Controller PERC 5/E Adapter (Slot 1)
ID : 0
Status : Ok
Name : Array0
State : Ready
Progress : Not Applicable
Layout : RAID-5
Size : 1,953.12 GB (2097149902848 bytes)
Device Name : /dev/sdc
Type : SAS
Read Policy : Adaptive Read Ahead
Write Policy : Write Back
Cache Policy : Not Applicable
Stripe Element Size : 64 KB
Disk Cache Policy : Disabled
ID : 1
Status : Ok
Name : Array1
State : Ready
Progress : Not Applicable
Layout : RAID-5
Size : 1,951.13 GB (2095006613504 bytes)
Device Name : /dev/sdd
Type : SAS
Read Policy : Adaptive Read Ahead
Write Policy : Write Back
Cache Policy : Not Applicable
Stripe Element Size : 64 KB
Disk Cache Policy : Disabled
ID : 2
Status : Ok
Name : Array2
State : Ready
Progress : Not Applicable
Layout : RAID-5
Size : 1,953.12 GB (2097151737856 bytes)
Device Name : /dev/sde
Type : SAS
Read Policy : Adaptive Read Ahead
Write Policy : Write Back
Cache Policy : Not Applicable
Stripe Element Size : 64 KB
Disk Cache Policy : Disabled
ID : 3
Status : Ok
Name : Array3
State : Ready
Progress : Not Applicable
Layout : RAID-5
Size : 1,953.12 GB (2097151737856 bytes)
Device Name : /dev/sdf
Type : SAS
Read Policy : Adaptive Read Ahead
Write Policy : Write Back
Cache Policy : Not Applicable
Stripe Element Size : 64 KB
Disk Cache Policy : Disabled
ID : 4
Status : Ok
Name : Array4
State : Ready
Progress : Not Applicable
Layout : RAID-5
Size : 1,953.12 GB (2097151737856 bytes)
Device Name : /dev/sdg
Type : SAS
Read Policy : Adaptive Read Ahead
Write Policy : Write Back
Cache Policy : Not Applicable
Stripe Element Size : 64 KB
Disk Cache Policy : Disabled
ID : 5
Status : Ok
Name : Array5
State : Ready
Progress : Not Applicable
Layout : RAID-5
Size : 1,957.88 GB (2102253060096 bytes)
Device Name : /dev/sdh
Type : SAS
Read Policy : Adaptive Read Ahead
Write Policy : Write Back
Cache Policy : Not Applicable
Stripe Element Size : 64 KB
Disk Cache Policy : Disabled
I was thinking... maybe XFS on LVM2 requires some specific PERC VD
setup? I had same issue with gentoo 32-bit with 2.6.25 kernel and with
one MD1000. But the problem happend once a month. Now it's getting
worse. last 24 hours - 2 crashes :(
Any ideas?
Best regards,
Tomek Kruszona
Reply to: