gluck (cvs, people, planet, etc.) downtime - ongoing raid problems
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
On Sunday evening gluck.debian.org started experiencing problems
writing to it's disks. The local admins investigated and after
physically power cycling machine it became apparent that the RAID
controller was deeply unhappy - it claimed to have lost 2 out of it's
6 RAID5'd drives. After reclaiming the drives it was left fscking
overnight for more than 9 hours.
Unfortunately, after it finished and came back up, it became apparent
that there's been extensive data corruption on the partition that
housed both /home and /org. The kernel is also deeply unhappy with
the file system, despite it being declared clean by e2fsck, with both
ls and rsync managing to cause the kernel to oops.
We've taken the box down for now and are trying to recover what we can
and get the file system to a state where the kernel is at least on
speaking terms. More information to come...
- --
James
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.8 <http://mailcrypt.sourceforge.net/>
iQIVAwUBQlGy9dfD8TGrKpH1AQJIIBAAjfaOcCcBLfDIDh1YlQ7T7/yRel06YGwj
FgGAnTwYQmHBEtSb2yRXWbJEMeyhYi5IvqkOVlFFY12lLrCs1jd5wqfX4WlST2pb
VNMqyRUQFYn5jkLP0Z8gLTShT5jqHarI4IJ0HQ3UbL6TOkrTfo/1PjTotSgOZalV
DLTAv7n/r7Mog21ZwbZHxR3Hdpj54DuNJ5AsicF0SpcmGIdkyRD+in+a9UehZ8mo
agdDv5iamGozncNCMpUhxK6dYNU3foexz/B06Lf721RArM/8pJF2MBeUzjPK1bBF
hv3rqHqRIETyGkr0L23VavtUovunP9OMFxBSdgQQMyYTcqDBK1ioP/RalAEQlKEl
W5ZUkBvZUbu3wh6HnFWGORo0ev4j9PfMKX6fX+uJNPYTz9t6RyJ07uVm8hdUJ8hZ
HydNEq+X87FukKGDGVE+phPCZc/tdoYYYnPRH6fh2DFvAqzmz1mHFhj2ypx8Qwq5
ksykX16p+4yBhTNDuxcSU8rPuc6uImo8Qgug09N9gDelHybwutIg/LpKdxVoPxNT
ADsId7yviCgdJOxZ4G5Ndo/dIN0RSo6TZh59gaQaRk2/CSOsX7C68ypaeIrWpA2U
H+U27dTZqP2ssJaT9pFNAF9439nOZbjnL9T7NVXcgCVgQchkAB+64JWa45nBUuLf
VboOE7/Ymp4=
=i8W6
-----END PGP SIGNATURE-----
Reply to: