Bug#274988: XFS crash in kernel-image-2.6.8-1-686-smp
Hi Jan,
Thanks for your bug report. The only xfs patch that is applied to the
Debian kernel is an ioctl patch which is pending a merge upstream, so I
guess this is a bug in the XFS code as present in Linus' tree. I have
CCed the linux-xfs mailing list in the hope that someone there can help
you out.
linux-xfs people, please feel free to either include or not include the
debian bug tracking system on any resulting thread by CCing
274988@bugs.debian.org as you feel fit. I have attached the one patch
that is applied to the debian 2.6.8 kernel for referance. Please let me
or the bug's address know if there is a resolution as I am not on the
linux-xfs list.
On Tue, Oct 05, 2004 at 09:43:03AM +0100, Jan Eringa wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Package: kernel-image-2.6.8-1-686-smp
> Version: 2.6.8-3
> Severity: Important
>
> Hardware layout
> - ---------------------
> Dual Xeon + Latest BIOS
> 1 GB ram
> 2 x 3ware SATA raid controllers + Latest Firmware
>
> All disks live on the 3ware 9xxx controllers
> Controllers provides 3 x 1.5TB raid-5 stripes
> One of which holds /, swap and /var.
>
> The rest of the free space I've built as a 4.5TB raid-0 stripe
> for the backup volume
>
> This is then carved into.....
> - ----------------------------------------------------------------------------------------
> backup-srv:~# df -k
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/sda1 3937220 2285880 1451336 62% /
> /dev/sda3 3937252 2497220 1240024 67% /var
> /dev/md0 4677217408 3090281796 1586935612 67% /backups
>
>
> - ----------------------------------------------------------------------------------------
>
>
> The /dev/md0 device is ....
> - ----------------------------------------------------------------------------------------
> backup-srv:~# cat /proc/mdstat
> md0 : active raid0 sdc1[2] sdb1[1] sda4[0]
> 4677348544 blocks 64k chunks
>
> unused devices: <none>
>
> - ----------------------------------------------------------------------------------------
>
> I had to use XFS as this was the only FS that would build that large.
> ext3 seems to barf at anything over 2TB
>
>
>
> Problem Description
> - -------------------------
> This machine is the production backup server for all the *nix machines on
> the network. cron runs rsync via ssh to grab the files from each client target
> The bulk of the systems are backed up weekly and a few daily
> The system seems to survive anywhere between a couple of days to no more
> that 2 weeks under this sort of heavy IO & network loading before
> giving up the ghost. dmesg dumps follow......
>
> This problem was also exhibited by 2.6.7 and 2.6.6
>
> I'm dropping back to 2.4.27 now & will let you know if pain persists
>
>
>
> - From Dmesg
> - ----------------------------------------------------------------------------------------
> Unable to handle kernel paging request at virtual address 20fda90c
> printing eip:
> f8b26144
> *pde = 00000000
> Oops: 0000 [#1]
> PREEMPT SMP
> Modules linked in: af_packet ipv6 piix hw_random uhci_hcd usbcore shpchp
> pciehp pci_hotplug floppy parport_pc parport pcspkr evdev e1000 xfs raid0 md
> dm_mod ide_cd ide_core cdrom rtc ext3 jbd mbcache sd_mod unix 3w_9xxx
> scsi_mod
> CPU: 3
> EIP: 0060:[<f8b26144>] Not tainted
> EFLAGS: 00010213 (2.6.8.20040927)
> EIP is at xfs_ail_insert+0x24/0xd0 [xfs]
> eax: 000003e7 ebx: 00000000 ecx: 000003e7 edx: 00000000
> esi: 20fda904 edi: f7198c18 ebp: c2005168 esp: f7703dd4
> ds: 007b es: 007b ss: 0068
> Process xfslogd/3 (pid: 604, threadinfo=f7702000 task=f7cb87d0)
> Stack: 0002050a 0000052a 549b2041 ed9cd202 c2005168 f7198c18 f7198c00 c0f1d30c
> f8b25e5d f7198c18 c2005168 00000000 c2005168 0002050a 0000052a 00000000
> c2005168 0002050a 0000052a f8b258bc f7198c00 c2005168 0002050a 0000052a
> Call Trace:
> [<f8b25e5d>] xfs_trans_update_ail+0x5d/0xf0 [xfs]
> [<f8b258bc>] xfs_trans_chunk_committed+0x17c/0x240 [xfs]
> [<f8b2566a>] xfs_trans_committed+0x4a/0x120 [xfs]
> [<f8b17743>] xlog_state_do_callback+0x2c3/0x3d0 [xfs]
> [<f8b178d0>] xlog_state_done_syncing+0x80/0xc0 [xfs]
> [<f8b15fe5>] xlog_iodone+0x55/0xf0 [xfs]
> [<f8b359bd>] pagebuf_iodone_work+0x4d/0x50 [xfs]
> [<c0131a26>] worker_thread+0x1f6/0x2e0
> [<f8b35970>] pagebuf_iodone_work+0x0/0x50 [xfs]
> [<c011c4f0>] default_wake_function+0x0/0x20
> [<c011c4f0>] default_wake_function+0x0/0x20
> [<c0131830>] worker_thread+0x0/0x2e0
> [<c0135f8a>] kthread+0xba/0xc0
> [<c0135ed0>] kthread+0x0/0xc0
> [<c01042c5>] kernel_thread_helper+0x5/0x10
> Code: 8b 46 08 8b 56 0c 89 44 24 08 89 54 24 0c 8b 55 0c 8b 45 08
> <6>note: xfslogd/3[604] exited with preempt_count 1
> - ----------------------------------------------------------------------------------------
>
>
> Machine locks up a little while after this & after a kick in the guts gives
> on next startup....
> - ----------------------------------------------------------------------------------------
> backup-srv:~# mount /backups/
> Oct 4 12:47:03 ouprci05 kernel: Filesystem "md0": XFS internal error
> xlog_clear_stale_blocks(2) at line 1253 of file fs/xfs/xfs_log_recover.c.
> Caller 0xf8b28876
> Oct 4 12:47:03 ouprci01 kernel: Filesystem "md0": XFS internal error
> xlog_clear_stale_blocks(2) at line 1253 of file fs/xfs/xfs_log_recover.c.
> Caller 0xf8b28876
> mount: Unknown error 990
> - ----------------------------------------------------------------------------------------
>
>
> So I try.....
> - ----------------------------------------------------------------------------------------
> backup-srv:~# xfs_repair /dev/md0
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
> - zero log...
> ERROR: The filesystem has valuable metadata changes in a log which needs to
> be replayed. Mount the filesystem to replay the log, and unmount it before
> re-running xfs_repair. If you are unable to mount the filesystem, then use
> the -L option to destroy the log and attempt a repair.
> Note that destroying the log may cause corruption -- please attempt a mount
> of the filesystem before doing this.
> - ----------------------------------------------------------------------------------------
>
>
> So I ....
> - ----------------------------------------------------------------------------------------
> backup-srv:~# xfs_repair -L /dev/md0
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
> - zero log...
> ALERT: The filesystem has valuable metadata changes in a log which is being
> destroyed because the -L option was used.
> - scan filesystem freespace and inode maps...
> - found root inode chunk
> Phase 3 - for each AG...
> - scan and clear agi unlinked lists...
> - process known inodes and perform inode discovery...
> - agno = 0
> LEAFN node level is 1 inode 2820138 bno = 8388608
>
> entry contains offset out of order in shortform dir 19126020
> corrected entry offsets in directory 19126020
> - agno = 1
> - agno = 2
> LEAFN node level is 1 inode 2147942164 bno = 8388608
> LEAFN node level is 1 inode 2148480815 bno = 8388608
> ....
> And so on for a few hours, for the rest of the 4.5TB file system check to
> complete :(
> ....
>
>
>
>
> ________________________________
> It is by caffeine alone I set my mind in motion,
> It is by the beans of Java that thoughts acquire speed,
> The hands acquire shaking, the shaking becomes a warning,
> It is by caffeine alone I set my mind in motion.
> (author unknown)
> with thanks and apologies to Frank Herbert
> ________________________________
> Jan Eringa
> Unix Admin
> Orbian Management Ltd
> ________________________________
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.4 (GNU/Linux)
>
> iD8DBQFBYl6XX4LWCZ7JjaMRAtH0AJwPIxdCA6xO88hHtJa27qo7UBlG/QCgigGI
> dhtLCXAxPd1W46KbnFMdMcY=
> =nuOo
> -----END PGP SIGNATURE-----
>
--
Horms
#! /bin/sh -e
## <PATCHNAME>.dpatch by <PATCH_AUTHOR@EMAI>
##
## All lines beginning with `## DP:' are a description of the patch.
## DP: Description: Add 32bit ioctl translations to XFS
## DP: Patch author: Christoph Hellwig <hch@lst.de>
## DP: Upstream status: pending submission in next XFS merge
. $(dirname $0)/DPATCH
@DPATCH@
--- 1.31/fs/xfs/Makefile 2004-06-18 04:05:01 +02:00
+++ edited/fs/xfs/Makefile 2004-08-11 10:20:52 +02:00
@@ -71,6 +71,7 @@
xfs-$(CONFIG_XFS_POSIX_MAC) += xfs_mac.o
xfs-$(CONFIG_PROC_FS) += linux-2.6/xfs_stats.o
xfs-$(CONFIG_SYSCTL) += linux-2.6/xfs_sysctl.o
+xfs-$(CONFIG_COMPAT) += linux-2.6/xfs_ioctl32.o
xfs-y += xfs_alloc.o \
--- 1.10/fs/xfs/xfs_fs.h 2004-02-27 07:28:05 +01:00
+++ edited/fs/xfs/xfs_fs.h 2004-08-11 18:27:01 +02:00
@@ -445,6 +445,13 @@
#define XFS_FSOP_GOING_FLAGS_NOLOGFLUSH 0x2 /* don't flush log nor data */
/*
+ * ioctl commands that are used by Linux filesystems
+ */
+#define XFS_IOC_GETXFLAGS _IOR('f', 1, long)
+#define XFS_IOC_SETXFLAGS _IOW('f', 2, long)
+#define XFS_IOC_GETVERSION _IOR('v', 1, long)
+
+/*
* ioctl commands that replace IRIX fcntl()'s
* For 'documentation' purposed more than anything else,
* the "cmd #" field reflects the IRIX fcntl number.
--- 1.27/fs/xfs/linux-2.6/xfs_ioctl.c 2004-05-24 05:08:10 +02:00
+++ edited/fs/xfs/linux-2.6/xfs_ioctl.c 2004-08-11 18:26:25 +02:00
@@ -73,14 +73,6 @@
#include <linux/pagemap.h>
/*
- * ioctl commands that are used by Linux filesystems
- */
-#define XFS_IOC_GETXFLAGS _IOR('f', 1, long)
-#define XFS_IOC_SETXFLAGS _IOW('f', 2, long)
-#define XFS_IOC_GETVERSION _IOR('v', 1, long)
-
-
-/*
* xfs_find_handle maps from userspace xfs_fsop_handlereq structure to
* a file or fs handle.
*
--- 1.84/fs/xfs/linux-2.6/xfs_super.c 2004-06-18 06:09:11 +02:00
+++ edited/fs/xfs/linux-2.6/xfs_super.c 2004-08-11 10:20:52 +02:00
@@ -66,6 +66,7 @@
#include "xfs_buf_item.h"
#include "xfs_utils.h"
#include "xfs_version.h"
+#include "xfs_ioctl32.h"
#include <linux/namei.h>
#include <linux/init.h>
@@ -857,6 +858,10 @@
goto undo_shaker;
}
+ error = xfs_ioctl32_init();
+ if (error)
+ goto undo_ioctl32;
+
error = register_filesystem(&xfs_fs_type);
if (error)
goto undo_register;
@@ -864,6 +869,9 @@
return 0;
undo_register:
+ xfs_ioctl32_exit();
+
+undo_ioctl32:
kmem_shake_deregister(xfs_inode_shaker);
undo_shaker:
@@ -882,6 +890,7 @@
vfs_exitquota();
XFS_DM_EXIT(&xfs_fs_type);
unregister_filesystem(&xfs_fs_type);
+ xfs_ioctl32_exit();
kmem_shake_deregister(xfs_inode_shaker);
xfs_cleanup();
pagebuf_terminate();
--- /dev/null 2004-08-10 00:19:24.000000000 +0200
+++ a/fs/xfs/linux-2.6/xfs_ioctl32.c 2004-08-11 18:27:20.008881680 +0200
@@ -0,0 +1,162 @@
+/*
+ * Copyright (c) 2004 Silicon Graphics, Inc. All Rights Reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it would be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+ *
+ * Further, this software is distributed without any warranty that it is
+ * free of the rightful claim of any third person regarding infringement
+ * or the like. Any license provided herein, whether implied or
+ * otherwise, applies only to this software file. Patent licenses, if
+ * any, provided herein do not apply to combinations of this program with
+ * other software, or any other product whatsoever.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write the Free Software Foundation, Inc., 59
+ * Temple Place - Suite 330, Boston MA 02111-1307, USA.
+ *
+ * Contact information: Silicon Graphics, Inc., 1600 Amphitheatre Pkwy,
+ * Mountain View, CA 94043, or:
+ *
+ * http://www.sgi.com
+ *
+ * For further information regarding this notice, see:
+ *
+ * http://oss.sgi.com/projects/GenInfo/SGIGPLNoticeExplan/
+ */
+
+#include <linux/config.h>
+#include <linux/compat.h>
+#include <linux/init.h>
+#include <linux/ioctl.h>
+#include <linux/ioctl32.h>
+#include <linux/syscalls.h>
+#include <linux/types.h>
+#include <asm/uaccess.h>
+
+#include "xfs_types.h"
+#include "xfs_fs.h"
+#include "xfs_dfrag.h"
+
+#if defined(CONFIG_IA64) || defined(CONFIG_X86_64)
+#define BROKEN_X86_ALIGNMENT
+#endif
+
+
+typedef struct xfs_fsop_bulkreq32 {
+ compat_uptr_t lastip; /* last inode # pointer */
+ __s32 icount; /* count of entries in buffer */
+ compat_uptr_t ubuffer; /* user buffer for inode desc. */
+ __s32 ocount; /* output count pointer */
+} xfs_fsop_bulkreq32_t;
+
+static int
+xfs_ioctl32_bulkstat(
+ unsigned int fd,
+ unsigned int cmd,
+ unsigned long arg,
+ struct file * file)
+{
+ xfs_fsop_bulkreq32_t __user *p32 = (void __user *)arg;
+ xfs_fsop_bulkreq_t __user *p = compat_alloc_user_space(sizeof(*p));
+ u32 addr;
+
+ if (get_user(addr, &p32->lastip) ||
+ put_user(compat_ptr(addr), &p->lastip) ||
+ copy_in_user(&p->icount, &p32->icount, sizeof(s32)) ||
+ get_user(addr, &p32->ubuffer) ||
+ put_user(compat_ptr(addr), &p->ubuffer) ||
+ get_user(addr, &p32->ocount) ||
+ put_user(compat_ptr(addr), &p->ocount))
+ return -EFAULT;
+
+ return sys_ioctl(fd, cmd, (unsigned long)p);
+}
+
+struct ioctl_trans xfs_ioctl32_trans[] = {
+ { XFS_IOC_DIOINFO, },
+ { XFS_IOC_FSGEOMETRY_V1, },
+ { XFS_IOC_FSGEOMETRY, },
+ { XFS_IOC_GETVERSION, },
+ { XFS_IOC_GETXFLAGS, },
+ { XFS_IOC_SETXFLAGS, },
+ { XFS_IOC_FSGETXATTR, },
+ { XFS_IOC_FSSETXATTR, },
+ { XFS_IOC_FSGETXATTRA, },
+ { XFS_IOC_FSSETDM, },
+ { XFS_IOC_GETBMAP, },
+ { XFS_IOC_GETBMAPA, },
+ { XFS_IOC_GETBMAPX, },
+/* not handled
+ { XFS_IOC_FD_TO_HANDLE, },
+ { XFS_IOC_PATH_TO_HANDLE, },
+ { XFS_IOC_PATH_TO_HANDLE, },
+ { XFS_IOC_PATH_TO_FSHANDLE, },
+ { XFS_IOC_OPEN_BY_HANDLE, },
+ { XFS_IOC_FSSETDM_BY_HANDLE, },
+ { XFS_IOC_READLINK_BY_HANDLE, },
+ { XFS_IOC_ATTRLIST_BY_HANDLE, },
+ { XFS_IOC_ATTRMULTI_BY_HANDLE, },
+*/
+ { XFS_IOC_FSCOUNTS, NULL, },
+ { XFS_IOC_SET_RESBLKS, NULL, },
+ { XFS_IOC_GET_RESBLKS, NULL, },
+ { XFS_IOC_FSGROWFSDATA, NULL, },
+ { XFS_IOC_FSGROWFSLOG, NULL, },
+ { XFS_IOC_FSGROWFSRT, NULL, },
+ { XFS_IOC_FREEZE, NULL, },
+ { XFS_IOC_THAW, NULL, },
+ { XFS_IOC_GOINGDOWN, NULL, },
+ { XFS_IOC_ERROR_INJECTION, NULL, },
+ { XFS_IOC_ERROR_CLEARALL, NULL, },
+#ifndef BROKEN_X86_ALIGNMENT
+ /* xfs_flock_t and xfs_bstat_t have wrong u32 vs u64 alignment */
+ { XFS_IOC_ALLOCSP, },
+ { XFS_IOC_FREESP, },
+ { XFS_IOC_RESVSP, },
+ { XFS_IOC_UNRESVSP, },
+ { XFS_IOC_ALLOCSP64, },
+ { XFS_IOC_FREESP64, },
+ { XFS_IOC_RESVSP64, },
+ { XFS_IOC_UNRESVSP64, },
+ { XFS_IOC_SWAPEXT, },
+ { XFS_IOC_FSBULKSTAT_SINGLE, xfs_ioctl32_bulkstat },
+ { XFS_IOC_FSBULKSTAT, xfs_ioctl32_bulkstat},
+ { XFS_IOC_FSINUMBERS, xfs_ioctl32_bulkstat},
+#endif
+ { 0, },
+};
+
+int __init
+xfs_ioctl32_init(void)
+{
+ int error, i;
+
+ for (i = 0; xfs_ioctl32_trans[i].cmd != 0; i++) {
+ error = register_ioctl32_conversion(xfs_ioctl32_trans[i].cmd,
+ xfs_ioctl32_trans[i].handler);
+ if (error)
+ goto fail;
+ }
+
+ return 0;
+
+ fail:
+ while (--i)
+ unregister_ioctl32_conversion(xfs_ioctl32_trans[i].cmd);
+ return error;
+}
+
+void
+xfs_ioctl32_exit(void)
+{
+ int i;
+
+ for (i = 0; xfs_ioctl32_trans[i].cmd != 0; i++)
+ unregister_ioctl32_conversion(xfs_ioctl32_trans[i].cmd);
+}
--- /dev/null 2004-08-10 00:19:24.000000000 +0200
+++ a/fs/xfs/linux-2.6/xfs_ioctl32.h 2004-07-11 16:42:03.000000000 +0200
@@ -0,0 +1,41 @@
+/*
+ * Copyright (c) 2004 Silicon Graphics, Inc. All Rights Reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it would be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+ *
+ * Further, this software is distributed without any warranty that it is
+ * free of the rightful claim of any third person regarding infringement
+ * or the like. Any license provided herein, whether implied or
+ * otherwise, applies only to this software file. Patent licenses, if
+ * any, provided herein do not apply to combinations of this program with
+ * other software, or any other product whatsoever.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write the Free Software Foundation, Inc., 59
+ * Temple Place - Suite 330, Boston MA 02111-1307, USA.
+ *
+ * Contact information: Silicon Graphics, Inc., 1600 Amphitheatre Pkwy,
+ * Mountain View, CA 94043, or:
+ *
+ * http://www.sgi.com
+ *
+ * For further information regarding this notice, see:
+ *
+ * http://oss.sgi.com/projects/GenInfo/SGIGPLNoticeExplan/
+ */
+
+#include <linux/config.h>
+
+#ifdef CONFIG_COMPAT
+extern int xfs_ioctl32_init(void);
+extern void xfs_ioctl32_exit(void);
+#else
+static inline int xfs_ioctl32_init(void) { return 0; }
+static inline void xfs_ioctl32_exit(void) { }
+#endif
Reply to: