Re: ssl security desaster (was: Re: SSH keys: DSA vs RSA)
On Thu, May 15, 2008 at 09:15:57AM -0700, Mike Bird wrote:
> The rollout of information and updates was appalling - even adding in
> the material from Ubuntu the information was piecemeal and inadequate
> to properly secure systems within the limited time before crackers
> might be expected to have exploits.
I think part of the problem here was that the coordinated release date
for the advisory was simply too soon after the relevant parties were
notified.
I understand that the news broke on vendor-sec on Thursday, and I found
out about it on Friday when I was brought into it (with my Ubuntu hat
on, though partly due to my position as Debian's openssh maintainer).
Tuesday was simply too soon to get all our ducks in a row. I worked
through the weekend on it and still only just managed to get something
ready in time (I'd have had no chance if I hadn't been able to drop
everything and work on it full-time), with essentially no opportunity
for anyone to review the code. The Ubuntu security team was even
harder-pressed, variously working through the night and while on
holiday. I'm sure the Debian security team weren't sitting on their
hands either, of course - these are just the stress points of which I
have direct experience.
The sheer length of
http://lists.debian.org/debian-devel-changes/2008/05/msg02311.html is
largely composed of items that should have been dealt with before the
advisory was released, but there just wasn't time. In these conditions
mistakes are going to be made.
Of course, there was a real risk that the news would leak before
advisories were ready, particularly since the fix was already in
unstable. It was also felt to be important to be somewhat cagey about
the nature of the vulnerability in an attempt to extend the time until
an exploit would be released (during which time automatic upgrades would
at least deal with replacing ssh host keys and blacklisting ssh user
keys), which meant that some documentation was a bit on the cryptic
side. It's certainly clear that trade-offs were involved. Still, I know
hindsight is 20:20, but I think an extra day or two on the embargo
period would very likely have produced a better result.
There was a problem with the Debian web site not being updated quickly
enough. I believe this was due to it running off a cron job which was
rather infrequent, and people noticed the gap between the advisory and
the cron job firing (please correct me if I'm wrong). If this is true
then we should make sure that documented emergency measures are in place
for this kind of thing.
> The vulnerability scanner didn't handle keys in many forms (e.g.
> Apache keys) and gave false negatives because it doesn't use
> ~/.ssh/config to check the correct port in the common case where
> ssh is running on a port other than 22. In the wonderful light
> of hindsight, it would probably have been better to devote less
> effort to the scanner and more effort to documenting all the
> kinds of key replacements that are needed, and to simply assume
> that all keys are potentially compromised.
I think the effort spent on key blacklisting was worthwhile; that
provided a degree of automatic protection that would not have been
available with a response that relied primarily on sysadmins' own
actions.
> Serious efforts are needed on two fronts. Second, we must ensure
> that nothing like this ever happens again. This calls for a thorough
> investigation and carefully updated policies and procedures. It will
> take a while to do properly. It must be apparent to both the Debian
> community and the world that the effort is authoritative, sincere,
> and open.
For major crises such as this, Canonical has a procedure of creating
incident reports, which at first largely serve the purpose of collecting
an authoritative timeline and information regarding whatever went wrong,
before going on to produce recommendations for change. Obviously this is
the sort of thing that works better in a company where you can tell
people to do this tedious work :-), but I think Debian could do worse
than to adopt a similar procedure for crises.
--
Colin Watson [cjwatson@debian.org]
Reply to: