Bug#295657: Another crash
Well, the crash happens still, up to twice a week. It is still somehow related to heavy disk use,
as it always happens when I'm copying multiple large files to RAIDSET-1, or during a nightly backup,
which copies lots of files from RAIDSET-1 to RAIDSET-0.
This morning the ext3-journal crashed again, and I found an extra interesting log-entry. See below.
I'm referring to the IO error a remount reports. Clues anyone?
Current kernel:
einstein:~# uname -a
Linux einstein 2.6.11-1-686-smp #1 SMP Mon Jun 20 20:18:45 MDT 2005 i686 GNU/Linux
einstein:~# dpkg -l | grep kernel-image-2.6.11
ii kernel-image-2.6.11-1-686-smp 2.6.11-7 Linux kernel image for version 2.6.11 on PPr
The log below was captured during nightly backup, copying a large number of files from RAIDSET-1 to RAIDSET-0.
From kern.log:
Aug 23 04:18:29 einstein kernel: EXT3-fs error (device dm-2): ext3_readdir: bad entry in directory #68272136: rec_len %% 4 != 0 - offset=0, inode=410831438
Aug 23 04:18:29 einstein kernel: Aborting journal on device dm-2.
Aug 23 04:18:30 einstein kernel: ext3_abort called.
Aug 23 04:18:30 einstein kernel: EXT3-fs error (device dm-2): ext3_journal_start_sb: Detected aborted journal
Aug 23 04:18:30 einstein kernel: Remounting filesystem read-only
Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1): ext3_free_blocks: Freeing blocks not in datazone - block = 4124995008, count = 1
Aug 23 04:18:40 einstein kernel: Aborting journal on device dm-1.
Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1): ext3_free_blocks: Freeing blocks not in datazone - block = 3676589613, count = 1
Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1): ext3_free_blocks: Freeing blocks not in datazone - block = 2129345344, count = 1
Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1): ext3_free_blocks: Freeing blocks not in datazone - block = 2512093470, count = 1
Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1): ext3_free_blocks: Freeing blocks not in datazone - block = 134217728, count = 1
Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1) in ext3_reserve_inode_write: Journal has aborted
Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1) in ext3_truncate: Journal has aborted
Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1) in ext3_reserve_inode_write: Journal has aborted
Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1) in ext3_orphan_del: Journal has aborted
Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1) in ext3_reserve_inode_write: Journal has aborted
Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1) in ext3_delete_inode: Journal has aborted
Aug 23 04:18:40 einstein kernel: __journal_remove_journal_head: freeing b_committed_data
Aug 23 04:18:40 einstein last message repeated 2 times
Aug 23 04:18:40 einstein kernel: ext3_abort called.
Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1): ext3_journal_start_sb: Detected aborted journal
Aug 23 04:18:40 einstein kernel: Remounting filesystem read-only
...REBOOT and REMOUNT...
Aug 23 05:42:30 einstein kernel: EXT3-fs warning (device dm-1): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Aug 23 05:42:30 einstein kernel: EXT3-fs warning (device dm-1): ext3_clear_journal_err: Marking fs in need of filesystem check.
Aug 23 05:42:30 einstein kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
Aug 23 05:42:30 einstein kernel: EXT3 FS on dm-1, internal journal
Aug 23 05:42:30 einstein kernel: EXT3-fs: recovery complete.
Aug 23 05:42:30 einstein kernel: EXT3-fs: mounted filesystem with ordered data mode.
Aug 23 05:43:17 einstein kernel: kjournald starting. Commit interval 5 seconds
Aug 23 05:43:17 einstein kernel: EXT3-fs warning (device dm-2): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Aug 23 05:43:17 einstein kernel: EXT3-fs warning (device dm-2): ext3_clear_journal_err: Marking fs in need of filesystem check.
Aug 23 05:43:17 einstein kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
Aug 23 05:43:17 einstein kernel: EXT3 FS on dm-2, internal journal
Aug 23 05:43:17 einstein kernel: EXT3-fs: recovery complete.
Aug 23 05:43:17 einstein kernel: EXT3-fs: mounted filesystem with ordered data mode.
Regards,
Jeroen
Reply to: