[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Update on compromise of gluck.debian.org, lock down of other debian.org machines

Hash: SHA1


To any press/general public type folks who might be reading this: this
mail is mostly aimed at developers - you might want to read Joey's
post[1] on debian-news instead.


			    Status Update

gluck.debian.org is back up and most services have been restored[2].
It has a new SSH key, which is attached at the end of this email[3].

Short version: A developer's debian.org account was compromised some
time ago.  This account was then used to exploit the recent prctl
vulnerability (CVE-2006-2451)[4] on gluck and gain root privileges.

Longer version follows...


Beginning at 02:43 UTC on 2006-07-12, 3 mails were sent as the result
of cron jobs running as root on gluck.debian.org.  These mails
were... obviously wrong and Matt Taggart contacted Ryan Murray and
myself at about 03:30.

			    What happened

We started investigating and discovered the following:

 o The cron emails referenced a specific user account and based on the
   (geographic) location of logins to this account it was clear that
   the account was compromised and had been for some time.

 o The attackers had then apparently obtained root via the recent
   prctl vulnerability (CVE-2006-2451)[4]; specifically via the
   exploit (or something very close to it) that had very recently been
   published on the full-disclosure mailing list[5].

 o The compromised account did not have access to any restricted
   Debian hosts (i.e. mailing lists, archive, security, etc.) and
   these machines had not been compromised.

We contacted the developer whose account had been compromised and he
responded.  It's not yet clear how that developer's account was

We also notified the contact people for other machines that we
suspected/knew were involved where possible.

As far as we can tell, due to the short window between the attacker
gaining root and us noticing it, they hadn't had time/inclination to
do a great deal.  The only obviously compromised binary we found was
'ping', which we're passing off to a forensics expert to look at.


We took gluck offline at 04:30 to boot it off of trusted media and
continue investigating.  We also started upgrading our other
i386/amd64 boxes and confirming that they hadn't been compromised.

In order to get services back online, we reinstalled gluck from
scratch, keeping only /home and /org intact.

			   What's been done

 o Any obvious secret keys (GPG or SSH) have been purged from gluck.

 o Anyone who kept their (Debian) GPG secret key on gluck has had
   their account locked and key removed from the keyring.
 o Accounts with weak passwords have been locked.

We'll be contacting the developers involved in the latter two points

			 How did this happen?

gluck was running Linux  Unfortunately it had not yet been
updated to or both of which were released on

	       How do I make sure my machines are safe?

If you're running sarge's kernel, you are not vulnerable to this
exploit as the first vulnerable kernel version was 2.6.13 and sarge is
only at 2.6.8.

If you're running a more modern kernel, make sure you're running
at least or

		     Lock down of other machines

We will be unlocking machines as and when they've been:

 (1) Updated to run a non-vulnerable kernel and...
 (2) Verified that they haven't been compromised.

You can see the status of this at:


Bear in mind though that this may take some time and that for a lot of
the !x86 machines, we rely on the local admin or a friendly porter to
provide us with a suitable kernel for that architecture so the work
may be blocked on them in some cases.


The following people deserve thanks for their efforts in managing this

  Matt Taggart, Dann Frazier, Ryan Murray, Anthony Towns, Paul Bame,
  Martin 'Joey' Schulze

- -- 

[1] http://lists.debian.org/debian-news/debian-news-2006/msg00030.html

[2] Except for CVS pserver, which needs a patched CVS package that
    we're still in the process of updating/restoring.


ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAsI8lJrAmf/xBOynwTpxXJ8c2X/4PCFTfx6CL17s6tJYPGBqZotMf63au4NETmkPNpD7+Ej4+79GVDh8omnYTEnctNlPQ0L2J7oga4yjL/KS37rA5W5pbwkmwhwSYp6PCM7yqBZUQIUmXGw82aLPSExD1KONBlPjEfXzcYWNL+KE= root@gluck

[4] http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-2451
[5] http://archives.neohapsis.com/archives/fulldisclosure/2006-07/0234.html
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.8+ <http://mailcrypt.sourceforge.net/>


Reply to: