Update on compromise of gluck.debian.org, lock down of other debian.org machines
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
To any press/general public type folks who might be reading this: this
mail is mostly aimed at developers - you might want to read Joey's
post[1] on debian-news instead.
================================================================================
Status Update
-------------
gluck.debian.org is back up and most services have been restored[2].
It has a new SSH key, which is attached at the end of this email[3].
Short version: A developer's debian.org account was compromised some
time ago. This account was then used to exploit the recent prctl
vulnerability (CVE-2006-2451)[4] on gluck and gain root privileges.
Longer version follows...
Detection
---------
Beginning at 02:43 UTC on 2006-07-12, 3 mails were sent as the result
of cron jobs running as root on gluck.debian.org. These mails
were... obviously wrong and Matt Taggart contacted Ryan Murray and
myself at about 03:30.
What happened
-------------
We started investigating and discovered the following:
o The cron emails referenced a specific user account and based on the
(geographic) location of logins to this account it was clear that
the account was compromised and had been for some time.
o The attackers had then apparently obtained root via the recent
prctl vulnerability (CVE-2006-2451)[4]; specifically via the
exploit (or something very close to it) that had very recently been
published on the full-disclosure mailing list[5].
o The compromised account did not have access to any restricted
Debian hosts (i.e. mailing lists, archive, security, etc.) and
these machines had not been compromised.
We contacted the developer whose account had been compromised and he
responded. It's not yet clear how that developer's account was
compromised.
We also notified the contact people for other machines that we
suspected/knew were involved where possible.
As far as we can tell, due to the short window between the attacker
gaining root and us noticing it, they hadn't had time/inclination to
do a great deal. The only obviously compromised binary we found was
'ping', which we're passing off to a forensics expert to look at.
Response
--------
We took gluck offline at 04:30 to boot it off of trusted media and
continue investigating. We also started upgrading our other
i386/amd64 boxes and confirming that they hadn't been compromised.
In order to get services back online, we reinstalled gluck from
scratch, keeping only /home and /org intact.
What's been done
----------------
o Any obvious secret keys (GPG or SSH) have been purged from gluck.
o Anyone who kept their (Debian) GPG secret key on gluck has had
their account locked and key removed from the keyring.
o Accounts with weak passwords have been locked.
We'll be contacting the developers involved in the latter two points
shortly.
How did this happen?
--------------------
gluck was running Linux 2.6.16.18. Unfortunately it had not yet been
updated to 2.6.16.24 or 2.6.17.4 both of which were released on
2006-07-06.
How do I make sure my machines are safe?
----------------------------------------
If you're running sarge's kernel, you are not vulnerable to this
exploit as the first vulnerable kernel version was 2.6.13 and sarge is
only at 2.6.8.
If you're running a more modern kernel, make sure you're running
at least 2.6.16.24 or 2.6.17.4.
Lock down of other machines
---------------------------
We will be unlocking machines as and when they've been:
(1) Updated to run a non-vulnerable kernel and...
(2) Verified that they haven't been compromised.
You can see the status of this at:
http://db.debian.org/machines.cgi
Bear in mind though that this may take some time and that for a lot of
the !x86 machines, we rely on the local admin or a friendly porter to
provide us with a suitable kernel for that architecture so the work
may be blocked on them in some cases.
Thanks
------
The following people deserve thanks for their efforts in managing this
incident:
Matt Taggart, Dann Frazier, Ryan Murray, Anthony Towns, Paul Bame,
Martin 'Joey' Schulze
- --
James
[1] http://lists.debian.org/debian-news/debian-news-2006/msg00030.html
[2] Except for CVS pserver, which needs a patched CVS package that
we're still in the process of updating/restoring.
[3]
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAsI8lJrAmf/xBOynwTpxXJ8c2X/4PCFTfx6CL17s6tJYPGBqZotMf63au4NETmkPNpD7+Ej4+79GVDh8omnYTEnctNlPQ0L2J7oga4yjL/KS37rA5W5pbwkmwhwSYp6PCM7yqBZUQIUmXGw82aLPSExD1KONBlPjEfXzcYWNL+KE= root@gluck
[4] http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-2451
[5] http://archives.neohapsis.com/archives/fulldisclosure/2006-07/0234.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.8+ <http://mailcrypt.sourceforge.net/>
iQIVAwUBRLaL9NfD8TGrKpH1AQKwChAAgrQsVOj85W5EWwa6bd97PQbqWU3wfURP
uddC52dteyePi84jqMTRyBs6J0haJ2ukuIEQWdxSJMSdMZQjlkjhxsgJrRD6dAbu
/TkY9KBNSicbOBgvbAAaVsYukFzVSkEy+li5rudGg22D3B2SrCLIrQup0woFOobk
U4TmUq9voIXwWz1dHmF/ruQajBwzKLMLzz98Osd0ICLU+IoLXB6JnR78/cLJj8nS
1Wg6G516sAj3N//VODST9oBJqag+PhEn6h4YgwXku8JJRVK4vJGIu5QNEBbTZu7j
BS9yBNENII73lGpNJlfbWyjDi+bQaEjng99D5POSrksvb7yAB+VeBLhpv4VZlley
incAxxR/ov3wjPRSBykUd09nim1RL0AvzN8i5GXr+SjwYVB7CPtqHeOWujVOBbIb
mrxmF0dcq/SdEkXvDTXa+7rWHTBoJyQnfrAM48tmweNqmHiR27vQcuKxU0xGJAba
pWyo2zatX1/CCDL/E0KdKf2ciuG9oGhkiGQ19UxEx7IEVxyYs0u7/nytlPbZF9wg
1v2Y1HsjZMFrkls91S8mM3wP8f36XcYtiEy6k97LktnxNvZQ3vIVNT62Tv+SSrp0
zmhIHr/6Qm2v/gn9BhGYKtsurQsV/SuZif6tWfT5Fwf1kRuR+8m9CwKP8lrfPYHg
cLttlYQ/hHM=
=KXR0
-----END PGP SIGNATURE-----
Reply to: