After gluck.debian.org started experiencing problems writing to its
disks on Sunday we have tried our best to get the machine back in
shape. As we have checked all files that we could and fixed all
services that had broken files, best to our knowledge, we have decided
to bring the machine back online. Please verify the files in your
home directory.
!!! Please check the files in your home directories! !!!
After several runs of e2fsck the machine was running stable again.
However, unfortunately, the filesystem that contains /home (and hence
/org as /home/org) was totally screwed. Many inode numbers were
exchanged by a near random permutation.
As a result of this, former files were turned into directories which
contained files that weren't supposed to be where we found them.
Other files were named properly but had wrong content. Also several
files contained blocks in the middle which weren't supposed to be
there, hence, broken content.
We have restored the services that were running on gluck.debian.org as
good as we were able to. We have also removed spurious files in users
home directories. We are unable to check the content of the remaining
files in users home directories, though.
This machine hosted the following service (some only as local copy)
cdimage.debian.org recovered
cvs.debian.org checked, see below
ddtp.debian.org disabled before
ftp.debian.org only a copy, recreated
keyring.debian.org only a copy, recreated
lintian.debian.org broken, needs to be rebuilt
packages.debian.org moved to another machine
admin.debian.org verified
people.debian.org need to be checked
planet.debian.org fixed
popcon.debian.org verified
release.debian.org fixed
www.debian.org restored
woody chroot recreated
The following CVS repositories are currently hosted on gluck:
/cvs/webwml restored from backup
/cvs/dak needs to be verified
/cvs/debian-admin not affected
/cvs/debian-doc restored from backup
/cvs/debian-boot restored from backup
/cvs/debian-openoffice checked, should be mostly fine
/cvs/debian-planet fixed
/cvs/debian-policy looks fine
/cvs/debbugs looks fine
/cvs/deity looks fine
/cvs/dpkg looks fine
/cvs/qa looks fine
/cvs/tetex fixed (one corrupt file to be replaced)
There are still problems in developers home directories. These need
to be verified from the developers on their own. In particular, the
public files in the public_html directories need to be verified to be
not broken. The .ssh directory has been disabled for all users.
In the directory /home/__Corrupt-R-US/$LOGNAME you'll find the
spurious files with wrong ownership that were found in the developer's
home directory. Feel free to take a look at them. We will remove
these on May 1st, so don't wait too long with checking whether
anything useful is included.
!!! Please check the files in your home directories! !!!
------------------------------------------------
If you have any questions, feel free to contact the Debian admin team
at <debian-admin@lists.debian.org>.
Regards,
Joey
PS: There are 23 accounts that had a current .ssh and a disabled one
from the break in as well. If you instate a new .ssh directory,
it's probably a good idea to remove the old one(s).
--
It's time to close the windows.
Attachment:
signature.asc
Description: Digital signature