on Wed, Nov 26, 2003 at 09:30:05AM +0100, Martin Schulze (joey@infodrom.org) wrote:
> Dan Jacobson wrote:
> > To us debian users, the most notable thing during this break in or
> > whatever episode, is how the communication structures crumbled.
>
> It had to be re-installed. You probably know that since you've read
> the announcement we were able to send out before the machine was taken
> down for reinstallation.
That announcement wasn't delivered for all users until _after_ murphy
was resurrected. I myself got the debian-security-announce message
mailed Nov 21 on 25 Nov 2003 15:16:56 -0800.
> > debian-announce had one message on the 21st, five days ago, saying for
> > more information, see www.debian.org.
>
> You'll find the same information linked on the front-page. Since the
> web infrastructure was affected as well, but you already knew that
> since it was mentioned in the announcement, it was not that easy
> updating the web server. However, after a day we finally managed to
> do that.
>
> > Nothing special there, so I checked http://www.debian.org/security/,
> > same problem.
>
> As you know http://www.debian.org/security/ if for security
> announcements regarding the packages Debian distributes. It has
> nothing to do with the security on the Debian machines. Hence, it's
> the wrong place.
First I want to say that the Debian project, in extremely adverse
circumnstances, comported itself well, disseminated information, if not
fully effectively, well beyond its nominal capacity with both web and
email services offline. Disclosures were timely, informative, and
helpful, while restraining themselves to established facts and working
within constraints of an as yet ongoing investigation. Very few
organizations can claim as much. Not only this, but it appears at this
point that the crown jewels -- the Debian archives and mirrored
distribution points themselves -- were _not_ compromised. Commendable.
Some bits could be improved, which is what I'm focusing on below.
I'll disagree with Martin's comment that the server compromise didn't
constitute a security issue despite the lack of an archive compromise.
For someone well versed in Debian procedures, it might have been
plausible that the archives themselves weren't compromised. For a
typical user, I don't think this was the case. For the typical user's
management or clients, it's very likely _not_ the case, and a timely
positive statement of status would be very, very helpful.
Security affecting Debian servers _potentially_ affects Debian packages.
As it was, I cleared my locale package cache and stopped updates on
hearing about the compromise. It wasn't for another few hours that I
was aware that the archive was reportedly _not_ compromised.
In the absense of any information, the security status of Debian project
packages in the event of a known or rumored server compromise is at best
unknown.
Communications in an emergency sitation is paramount, and a number of
people clearly _didn't_ get informed through back channels. I myself
was _on_ IRC as word started leaking out, and still wasn't fully certain
of what was going on or what to trust. Wichert's website (which I only
learned was his the 27th!) was very helpful, as was the coverage
provided by Slashdot and elsewhere.
Discussion this with Manoj on IRC, my suggestion as summarized by him is
that Debian should have an emergency response plan, part of which is a
communications policy in the event a similar future compromise or
systems failure. Specifically:
- Triggering events. There are thresholds below which notifications
needn't be triggered, and above which they very much should.
Suggested: any event significantly affecting perceptions of
security of the Debian archives or servers. Any outage of mail,
web, or archive services anticipated to last beyond <n> <time
units>. E.g.: 6-12 hours, across core servers (but not mirrors).
Any core server root compromise. *Not* single-package issues.
Nuclear war or asteroid strike: you're on your own.
- Where to provide information. Personal websites and news channels
served well, but an advance statement of "here's where you should
turn in the event of an emergency" would be useful.
- What information to provide.
Specifically,
- the known (or unknown) status of archive or package compromise.
- diagnostic checks; and/or
- cleanup procedures.
Wichert's pages on this would be a good template.
By "known (or unkown)", I mean: if the archives are reasonably
known to be safe, or are known to be compromised, this is
communicated. If an assessment cannot be made with confidence,
_that_ fact should be stated, e.g.: "the current security of the
archives is unknown".
By diagnostics and cleanup: pointers to tools or documentation
explaining how to assess and/or secure a system. Wipe and rebuild
if necessary. Again, wiggy.net is a good model.
- If possible: a procedure for ensuring that reasonably current
subscriber lists for debian-announce and/or debian-security-announce
at the very least (add other critical lists as appropriate,
including developers and admin teams) are maintained in multiple
locations _off_ Debian project servers, and can be used by DPL,
Listmaster, and/or other key persons.
By reasonably current -- say, a once-a-week cron job that mails the
list(s) to the identified persons, who are responsible for keeping
these maintained offline. There's a balance between spamming
unsubscribed addresses (though with a single message, risk is
relatively small), and keeping subscribers informed.
I'm not talking about mirroring the entire list machine, I'm talking
about the ability to send mail to people in the event of an
emergency. A list, possibly apportioned, and one or more systems
with a reasonably high speed connection to distribute it.
- If possible: where to post questions or clarifications. An
emergency "Ask Slashdot" or equivalent might be suitable.
Essentially a place where reasonable questions can be posted,
without overburdening an already overworked, stressed security team
attempting system damage control, assessment, and recovery.
- An update schedule -- say, posts every 2-3 days for a prolonged
outage. Enough to keep people informed, but not swamp developers or
the fall back emergency list infrastructure, or at least a "expect
updates within <n> <time units>".
Including this information in a document distributed with Debian (say,
Debian Policy or another document along the lines of "What To Do In The
Event Of A Debian Emergency", would be useful.
Peace.
--
Karsten M. Self <kmself@ix.netcom.com> http://kmself.home.netcom.com/
What Part of "Gestalt" don't you understand?
I was never no good after that night, Charlie.
- M. Brando
Attachment:
pgpoSqc3mjBOm.pgp
Description: PGP signature