recovery status update
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
[To any press/general public type folks who might be reading this:
this mail is really aimed at developers, hopefully a more (coherent
etc.) public announce will go out soon through the normal channels
for the "relevant to non-developer" bits (e.g. the new signing key,
the archive scripts being up etc.)]
Hi,
Sorry it's taken a little bit longer than planned, but here's an
update on the recovery process.
- --
James [- as always, speaking only for myself.]
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The archive and uploads
=======================
The archives' integrity (all 3 of them) has been verified in multiple
independent ways[1] and we're confident that they weren't tampered
with.
The archive scripts on ftp-master have been audited and re-enabled and
a mirror pulse went out tonight. For now, non-ftpmaster local
accounts haven't been re-enabled on ftp-master. At the moment this is
temporary, but see the "Where can I login?" section of the "Accounts
and Services" chapter below.
The archive scripts for security.d.o are audited but are being run
byhand at the moment, simply because without the buildds, the security
infrastructure is painfully crippled. I haven't even looked at the
archive scripts for non-US; it's extremely low priority (seeing as how
it's mostly an orphanage these days).
New Automatic Signing Key
-------------------------
There's a new automatic signing key used to sign the Release.gpg files:
pub 1024D/30B34DD5 2003-12-03 Debian Archive Automatic Signing Key (2003 v2) <ftpmaster@debian.org>
Key fingerprint = ACFA DE85 7807 A173 34D2 692C 2DB1 C725 30B3 4DD5
It's available from the keyring[2] in the 'debian-role-keys.gpg' keyring or from
http://ftp-master.debian.org/ziyi_key_2003v2.asc
The key (and this message) are signed by my key.
How do I upload?
----------------
Use the anonymous upload queue on ftp-master. I believe you want to
use something like "dupload --to anonymous-ftp-master foo.changes" for
dupload and "dput ftp-master foo.changes" for, err, dput. But I don't
use either of these programs and haven't tested that.
buildds & testing
-----------------
Neither the buildds nor testing are running yet so any uploads you
make won't be recompiled for other architectures or progress into
sarge. Testing should be back soon, but the buildds will take some
time, there are a lot of them (esp. since several architectures have
more than one) and they each have to be updated, hardened, re-LDAPed
etc.
further keyring removals
------------------------
Due to the Elgamal Sign&Encrypt weakness[3], a bunch of keys got
pro-actively removed from the keyring[4]. This purge didn't take into
account any keys that had updates that got sent to the keyserver on
keyring.debian.org post-compromise.
If you have a subkey that's type 20, and you want to revoke it and
generate a new subkey, please send the updated key to
keyring-maint@debian.org.
If you have a pubkey that's type-20, you lose. Please follow the
instructions on http://keyring.debian.org/replacing_keys.html .
NB: signing a new key with only your old type-20 pubkey is not a valid
trust path.
Please also be aware that keyring-maint processing is going to take a
back seat for a while (needs of the many vs. the few and all that);
apologies for the inconvenience.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Accounts and Services
=====================
When are logins going to be re-enabled?
---------------------------------------
We're working on this as fast as we can. I hope to have accounts on
db.debian.org unlocked in the next 24 hours or so. This'll allow
people to log back onto a subset of previous debian.org machines that
have been sanitized, hardened etc. At the moment, these machines are:
master, gluck, merkel [unrestricted]
klecker, auric [restricted - for now]
murphy, newsamosa, newraff, saens [restricted]
Once things are ready, I'll post a follow-up message to let people
know (including details of how exactly to re-enable your account).
What to do when you can login
-----------------------------
Unfortunately, the attacker had control of at least the 4 machines for
upto a couple of days and there's no way to be 100% sure of what he
used his root access to do. So, we have to assume the worst. There's
obviously no way for us to verify the contents of your home
directories properly, so what we've done is tried to sanitize them so
that you guys can login and check it out yourself.
Please, pretty please with a cherry on top, take this seriously and
either check the files you have there, or if you can, just remove
anything you don't need. Please don't just move stuff back or make
stuff executable again blindly.
What was done to home dirs
--------------------------
o Any obvious "secret" files (gpg secret keys, ssh private keys) have
been removed.
[NB: this was done purely by name - if you had a GPG secret key on
*.d.o in a file called s3kr1t.gpg, we won't have found it.
Please assume that anything that relies on the secrecy of the
file is compromised and clean it up yourself.]
o All s{g,u}id files have been de-s{g,u}id-ized.
o The following files have been moved out of the way (i.e. moved to
"DSA_disabled-$file"):
- public_html
- .ssh
- bash, {t,}csh, zsh startup files
[NB: the script used to do this was fairly mindless; in retrospect
it's obviously moved entirely innocuous files out of the way
(e.g. ".bash*" covers .bash_history) - I didn't want to spend
too much time obsessing over details...]
o All files in /home/ have been made non-executable.
o crontabs have been removed and copied into to ~/DSA_disabled-crontab.
New SSH keys
------------
All the re-LDAPified boxes have new SSH keys. They only have RSA [v2]
keys; no DSA or RSA1 and sshd will only allow protocol 2 to connect.
The keys for logon boxes are below:
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAlc/iz5Vj5//Bd4FCLB+L2Nj3Bkre/E4EhjNRLXoykbsKDMvrRlA0ZAu3hvqpHFbfJ04W4k+b1ejhgoM/HvLF8/mz7XCFvXFcaovLhrDxQc//5wF7gHbKrkB/0oULarBu1CsqCgHFs1D4t9/G3HALy3KxiW8Q/7wEj913LSStJR0= root@master
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA4wA+cdq6tS3hk+/4+EvkZQ4YMJBM4eI7j5ooGk9a1IwvlXsWD37Yg4S2xqWmqOXWhIgTlLZoFetdCRPyc/IBw+8xlCTEssxh7lSkBnJxpCgS+0ITCtDdYaKC9q3j/KbOB3EHglIo9vOTPTAEd3PkHI6jSfLxwWbdgYm5s9ZUVV8= root@gluck
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA1BMDgFcnGmv+fhwNCeIVxn/tx6wtVBDdAqdQn+gIsfFm3X9jkoj0pelo7pjoX2ASE2Bg1ETnx2vfcjWseb25PH3wIlAdjKoTDQrjIGwtTy3SqUmN0AdeE3fWKQHz0Lf7AIEipVZJhkK8tLpP2Bq82L1Da6BGpQHC5uO5cuqGcuM= root@merkel
Host public keys are also available through db.debian.org; see the
"db.debian.org LDAP changes" chapter below for more info on that.
========================================
Where can I login?
------------------
There's been a fair bit of talk post-compromise about restricting
access to machines running (core) services. At the moment, the only
thing I'm (personally) doing is not enabling non-services accounts on
auric (ftp-master) and klecker (security, non-US, qa, nm, www-master)
immediately. Obviously, it's useful for random developers to have
access to e.g. the postgres database of the archive, so the current
plan if the restricted nature of auric becomes permanent is to mirror
the system daily to another box that would be unrestricted. [This
would have the added bonus of giving us a hot spare for
disasters/arson attacks etc.]
Basically the whole issue of what, if anything, to restrict is still
up in the air. I'm looking for input/opinions/discussion on this. If
you need access to the machines running the archives, please tell me
(or probably better yet, start a thread on debian-devel) why.
On a similar note some of our boxes are currently overloaded and
services are generally inelegantly distributed; there's certainly
going to be some juggling of them coming up. It's not decided
what/when/where/how yet though, more details before it happens.
======================================================================
"When is [my/foo] service coming back(, you [etc.])?"
-----------------------------------------------------
Services need much the same scrutiny; if you run a service, contact us
(DSA). If it's on one of the known compromised machines (and it
probably is), it'll probably not be in /org anymore, and you'll need
to check & vouch for it before it can go back. (Obviously once
accounts comes back. [Although temporary accounts have already been
given to the admins of some of the more urgent/important services to
speed this process along.])
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
db.debian.org LDAP changes
==========================
As part of the recovery process, db.debian.org has migrated, both to a
new host (because the old box was on it's last legs and HP kindly
donated a shiny new one to replace it), and to a newer version of LDAP
(because it was still using potato's OpenLDAP).
======================================================================
New Certificate
---------------
As part of the "assume the worst" approach to the recent compromise, a
new SSL key has been generated. As an added bonus, this time around
it's signed/issued by SPI. Here's the public certificate:
- -----BEGIN CERTIFICATE-----
MIIEMzCCA5ygAwIBAgIBBDANBgkqhkiG9w0BAQQFADCBvjELMAkGA1UEBhMCVVMx
EDAOBgNVBAgTB0luZGlhbmExFTATBgNVBAcTDEluZGlhbmFwb2xpczEoMCYGA1UE
ChMfU29mdHdhcmUgaW4gdGhlIFB1YmxpYyBJbnRlcmVzdDETMBEGA1UECxMKaG9z
dG1hc3RlcjEgMB4GA1UEAxMXQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxJTAjBgkq
hkiG9w0BCQEWFmhvc3RtYXN0ZXJAc3BpLWluYy5vcmcwHhcNMDMxMjA0MTcxODEx
WhcNMDQxMjAzMTcxODExWjCBsTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0luZGlh
bmExFTATBgNVBAcTDEluZGlhbmFwb2xpczEoMCYGA1UEChMfU29mdHdhcmUgaW4g
dGhlIFB1YmxpYyBJbnRlcmVzdDEPMA0GA1UECxMGRGViaWFuMRYwFAYDVQQDEw1k
Yi5kZWJpYW4ub3JnMSYwJAYJKoZIhvcNAQkBFhdkZWJpYW4tYWRtaW5AZGViaWFu
Lm9yZzCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEApaK6O7TdIoEQaGmcum4n
eVcN3hZWcI5pD/HOeYkgThbgDbb8JwXnrX/dcesGxtGucHQtoKuPsqLBIYKau7u7
I550LxdXcyRJwfak26gpsLHftUp1xA9z29OrEGS4/Z0BvqzqZnOQOYJsp58FrPqb
gCVb9DXPXftWgzTrkNLBKyUCAwEAAaOCAUowggFGMAkGA1UdEwQCMAAwLAYJYIZI
AYb4QgENBB8WHU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1UdDgQW
BBSCf6x2+nhzk5LULS4hjj6TcpMzeTCB6wYDVR0jBIHjMIHggBQHrehBHX+91r8b
gXo/jEuI3gTS+qGBxKSBwTCBvjELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0luZGlh
bmExFTATBgNVBAcTDEluZGlhbmFwb2xpczEoMCYGA1UEChMfU29mdHdhcmUgaW4g
dGhlIFB1YmxpYyBJbnRlcmVzdDETMBEGA1UECxMKaG9zdG1hc3RlcjEgMB4GA1UE
AxMXQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxJTAjBgkqhkiG9w0BCQEWFmhvc3Rt
YXN0ZXJAc3BpLWluYy5vcmeCAQAwDQYJKoZIhvcNAQEEBQADgYEANapO5Qc/0kS0
v5gGbtxmGtK0IsQGf00cjdaOLDtl6VuqMJY+wN6TxRp9+Yv6b6yQipwWnv9+026j
+UWoADEPuzLgigIWQRBxUMEDqNtkj3fqIgr/Lc0qkOezO0gosXTo/OEIQPybTenL
LXAHc/lxr/yg7iqAixrv9g9cFbDSdc8=
- -----END CERTIFICATE-----
======================================================================
Schema and database changes
---------------------------
Since OpenLDAP 2.x is a lot more fascist about what data it would
import, we took the opportunity to upgrade userdir-ldap to use a
proper schema and enable strict schema checking. This required some
changes to the database:
o allowedHosts -> allowedHost
o labeledURL -> labeledURI
o objectClass: account -> objectClass: inetOrgPerson
o objectClass: posixAccount -> objectClass: debianAccount
o objectClass: posixGroup -> objectClass: debianGroup
o hosts now have objectClass: debianServer
o icqUin fields have been sanitized to meet the schema definition.
o telephoneNumber field likewise.
o jabberJID has been added but, NB, the scripts, mail gateway and web
pages have not been updated to support it yet.
o All fields using non-7Bit ASCII data have been (re)encoded as UTF-8.
The new schema is available, for now, from:
http://db.debian.org/userdir-ldap.schema
but it will in due course be part of the userdir-ldap source.
======================================================================
Thanks
------
Many thanks to Steve Langasek for his extensive work on the 1.x -> 2.x
migration.
But it's all broken
-------------------
Prior to the compromise, the plan for the LDAP/db.debian.org migration
was to do it slowly and only switch over after everything was tested
and working. That obviously hasn't happened and a lot of stuff broke.
Most of it should be working now, but please report any problems you
see with it (especially when it's unlocked) to
debian-admin@lists.debian.org.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
[1] Please don't ask me personally for details, I didn't do it and
don't know the specifics.
[2] Available from the usual places, i.e.
o rsync://keyring.debian.org::keyrings/
o debian/doc/debian-keyring.tar.gz on your nearest (up-to-date) mirror.
[3] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CAN-2003-0971
[4] 1024D/0B35E52E Anthony Towns <aj@azure.humbug.org.au>
1024D/66468D05 Marcelo E. Magallon <mmagallo@debian.org>
1024D/17BA45CE J.H.M. Dassen (Ray) <jdassen@debian.org>
1024D/9591557A Bart Warmerdam <bartw@xs4all.nl>
1024D/B1E054D7 Martin Mitchell <martin@debian.org>
1024D/A3D7B9BC Chris Lawrence <lawrencc@debian.org>
2048G/5AF274F7 Hanno Wagner (Debian-Key) <wagner@debian.org>
1024D/54782B9A James LewisMoss (Dres) <dres@debian.org>
1024D/9823336C Jim Westveer (jwest) <jwest@netnw.com>
1024D/CDAB933D Peter Teichman <peter@teichman.org>
1024D/2705CD5A Nicholas Flintham <nick@flinny.demon.co.uk>
1024D/3E70D097 Ron Farrer <rbf@octanecs.net>
1024G/948562BC Tomas Pospisek <tpo_deb@sourcepole.ch>
1024D/44703332 Eric Gillespie, Jr. <epg@debian.org>
1024D/07ACF6FA Thomas Seyrat <tomasera@debian.org>
1024G/07E2EA40 Juan Alvarez <jalvarez@fluidsignal.com>
1024D/68DA0F3E Marcel Harkema <marcel@nl.linux.org>
2048G/B6C7087B Aigars Mahinovs <aigarius@debian.org>
1024G/2BE2FE81 Mark Johnson (Harco) <mrj@debian.org>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.8 <http://mailcrypt.sourceforge.net/>
iEYEARECAAYFAj/P5KgACgkQgD/uEicUG7DeKACeJLyb+NruC7wnmE2q/q8laBQ/
2+MAnjqhdRyvcIENTIZencbpHweC34Jp
=X6uf
-----END PGP SIGNATURE-----
Reply to: