gluck (cvs, people, planet, etc.) downtime - ongoing raid problems
-----BEGIN PGP SIGNED MESSAGE-----
On Sunday evening gluck.debian.org started experiencing problems
writing to it's disks. The local admins investigated and after
physically power cycling machine it became apparent that the RAID
controller was deeply unhappy - it claimed to have lost 2 out of it's
6 RAID5'd drives. After reclaiming the drives it was left fscking
overnight for more than 9 hours.
Unfortunately, after it finished and came back up, it became apparent
that there's been extensive data corruption on the partition that
housed both /home and /org. The kernel is also deeply unhappy with
the file system, despite it being declared clean by e2fsck, with both
ls and rsync managing to cause the kernel to oops.
We've taken the box down for now and are trying to recover what we can
and get the file system to a state where the kernel is at least on
speaking terms. More information to come...
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.8 <http://mailcrypt.sourceforge.net/>
-----END PGP SIGNATURE-----