[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

gluck available again / filesystem shaked

After gluck.debian.org started experiencing problems writing to its
disks on Sunday we have tried our best to get the machine back in
shape.  As we have checked all files that we could and fixed all
services that had broken files, best to our knowledge, we have decided
to bring the machine back online.  Please verify the files in your
home directory.

     !!!   Please check the files in your home directories!   !!!

After several runs of e2fsck the machine was running stable again.
However, unfortunately, the filesystem that contains /home (and hence
/org as /home/org) was totally screwed.  Many inode numbers were
exchanged by a near random permutation.

As a result of this, former files were turned into directories which
contained files that weren't supposed to be where we found them.
Other files were named properly but had wrong content.  Also several
files contained blocks in the middle which weren't supposed to be
there, hence, broken content.

We have restored the services that were running on gluck.debian.org as
good as we were able to.  We have also removed spurious files in users
home directories.  We are unable to check the content of the remaining
files in users home directories, though.

This machine hosted the following service (some only as local copy)

cdimage.debian.org            recovered
cvs.debian.org                checked, see below
ddtp.debian.org               disabled before
ftp.debian.org                only a copy, recreated
keyring.debian.org            only a copy, recreated
lintian.debian.org            broken, needs to be rebuilt
packages.debian.org           moved to another machine
admin.debian.org              verified
people.debian.org             need to be checked
planet.debian.org             fixed
popcon.debian.org             verified
release.debian.org            fixed
www.debian.org                restored
woody chroot                  recreated

The following CVS repositories are currently hosted on gluck:

/cvs/webwml                   restored from backup
/cvs/dak                      needs to be verified
/cvs/debian-admin             not affected
/cvs/debian-doc               restored from backup
/cvs/debian-boot              restored from backup
/cvs/debian-openoffice        checked, should be mostly fine
/cvs/debian-planet            fixed
/cvs/debian-policy            looks fine
/cvs/debbugs                  looks fine
/cvs/deity                    looks fine
/cvs/dpkg                     looks fine
/cvs/qa                       looks fine
/cvs/tetex                    fixed (one corrupt file to be replaced)

There are still problems in developers home directories.  These need
to be verified from the developers on their own.  In particular, the
public files in the public_html directories need to be verified to be
not broken.  The .ssh directory has been disabled for all users.

In the directory /home/__Corrupt-R-US/$LOGNAME you'll find the
spurious files with wrong ownership that were found in the developer's
home directory.  Feel free to take a look at them.  We will remove
these on May 1st, so don't wait too long with checking whether
anything useful is included.

     !!!   Please check the files in your home directories!   !!!

If you have any questions, feel free to contact the Debian admin team
at <debian-admin@lists.debian.org>.



PS: There are 23 accounts that had a current .ssh and a disabled one
    from the break in as well.  If you instate a new .ssh directory,
    it's probably a good idea to remove the old one(s).

It's time to close the windows.

Attachment: signature.asc
Description: Digital signature

Reply to: