[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

The New Security Build Infrastructure

Hello world,

So, there's probably a whole bunch of people out there who're interested
in the form new security infrastructure will take. If you're not, feel
free to skip this message.

Historically, which is to say, currently, the security.debian.org archive
has been managed pretty manually: a member of the security team will
patch a program with a security bug and test it; then ssh to machines
of each different architecture in the stable release and build the new
source there; then, once this is done everywhere, prepare an advisory
including the md5sums of all the newly built packages; then move the
packages into the appropriate place on security.debian.org; then run
dpkg-scanpackages and dpkg-scansources; and, finally, sign and send out
the advisory. (The above is the general idea aiui, anyway. I'm sure if
anyone cares the security team can correct the details)

This has a whole bunch of problems: since almost everything's done
by hand, it's easy to forget to do something or to make mistakes,
and this has occasionally resulted in broken security updates or
confusing advisories. Additionally, it takes a lot of time and care,
which can delay updates, and certainly makes the job a lot harder than
it might otherwise be. But the real problem is it doesn't scale well:
the ever growing number of packages alone gives the security team an ever
increasing amount of work to do, with woody having twice the number of
ports as potato we've made every single security update require twice
as much effort from the security team, and that's just too much.

Well, that's the problem statement anyway: managing security updates,
and especially getting them built, needs to be much easier.

So, what's the solution? For a fair while (at least a few months that
I've been aware of), the security team were attempting to get "rbuilder"
[0] installed on machines of each architecture so that they could use
that to build security updates, but for various reasons didn't have much
success. As such, in early to mid April, the buck was passed, although
it wasn't for a couple of weeks after that that we figured out who it
could reasonably be passed to.

The new buildd infrastructure consists of two components that've been
on the agenda for a while now:

 (1) Converting security.debian.org to be run by katie

 (2) Modifiying the central wanna-build infrastructure to do

For the former, the thought is that it's a waste to have all this neat
software for managing Debian archives, and then not use it for our
security archive -- but until now, nothing's been actually done, since
making it happen requires some significant changes to both the security
archive itself (poolising it amongst other things), and to generalise
the katie code itself.

Meanwhile, the latter is a nifty idea that the archive changes for
crypto-in-main have allowed: since packages are checked (for signatures
and such) within about fifteen minutes of them being uploaded, there's
no longer any reason to have the autobuilders wait until the daily
archive run before trying to build them. So it's basically a matter
of setting up the buildds' sources.list to point to a new url [0], and
add newly accepted packages to the wanna-build database, and -- presto
chango! -- packages get automatically built thirty minutes or an hour
or so after they're uploaded! As it happens, there're a bunch of little
complications beyond this, to the point where the current setup is the
third reimplementation of the above basic description in the past month
or so, but you get the idea.

While the above makes for a good answer to the scalability problems
the security team are facing -- the archive code and the autobuilders
already cope with much larger numbers of packages than the security team
produce -- there are a number of additional difficulties in trying to
manage security uploads that have to be taken into account.

The major difference is that security updates frequently shouldn't be
made public as soon as they're ready; in some cases they only need to
wait until they can be tested further or an advisory written, in others
they need to wait for a week or more to avoid publicising the flaw until
all vendors have had a reasonable chance to fix it. This is pretty much
in direct opposition to the way normal uploads work: we want those to
be publically available as soon as possible. And, of course, there's an
additional problem here in that in order to do accepted autobuilding,
packages need to be made available to the autobuilders as early as
possible, which is well before thay should be made available to the
wider public.

So! How does it all work? Here's the general idea:

 a) Someone finds a security problem.

 b) Someone fixes the problem, and makes an upload to
    security.debian.org's incoming. That is to say, a package is
    uploaded to:

          (aka ftp://security.debian.org/pub/SecurityUploadQueue)

    Its changelog needs to point it at "stable-security" or
    "testing-security" in place of the usual "unstable". It should be
    signed in the usual manner. It should, obviously, be tested fairly
    thoroughly. Anyone can do this, but presumably it'll usually be a
    member of the security team. If you're _not_ a member of the security
    team, you should definitely contact them before doing this. Further,
    while anyone can upload files here, only the security team can see
    what people have uploaded.

 c) The upload gets checked and processed by "jennifer" and moved into
    queue/accepted, and the buildds are notified. Files in here can
    be accessed by the security team and (somewhat indirectly) by the

 d) Security-enabled buildds pick up the source package (prioritized over
    normal builds), build it, and send the logs to the security team.

 e) The security team reply to the logs, and the newly built packages
    are uploaded to queue/unchecked, where they're processed by jennifer,
    and moved into queue/accepted.

 f) When the security team are happy with the source package (ie, that
    it's been correctly built for all applicable architectures and that
    it fixes the security hole and doesn't introduce new problems of its
    own) they run a script (viz, "amber") which: 
	* installs the package into the security archive
	* updates the Packages, Sources and Release files in the usual way
	* sets up a template advisory that the security team can finish off
	* (optionally) forwards the packages to the appropriate 
	  proposed-updates so that it can be included in the real archive ASAP

And that's pretty much it.

All of the above has been implemented and tested in one way or another,
although some of this hasn't been in entirely realistic conditions yet.
It does, though, seem pretty safe to say that any further changes that'll
be needed will be small bugfixes rather than redesigns or rewrites. katie
for security.debian.org has been setup on satie.debian.org [1] and
basically works, and the buildd changes are being live tested on the
alpha, i386, powerpc and sparc autobuilders.

Credit for the katie implementation work go to James Troup, for the
buildd changes, Ryan Murray, and for all the -devel and other flamewars,
yours truly. ;)

What you can expect next is for the security infrastructure to move
over to satie permanently and into the hands of the security team, and
an update on all the other things that affect woody which have come up
over the past month and a bit.

It's possible that some of these changes will let us handle security
updates in better ways in future (I'm hopeful it'll let us does security
updates for testing in a more timely manner, eg), but that probably
won't become clear until the security team's had more of a chance to
get comfortable with it all.

aj (woody release manager)

[0] http://incoming.debian.org/buildd/, as it happens. These are IP/password
    restricted and shouldn't be used except by autobuilders. If you want to
    poke around just for curiousity's sake, auric:/org/incoming.debian.org/
    is one place to start.

[1] Also accessible via a reference to this project's codename, "security
    by obscurity", as obscurity.debian.org.

Anthony Towns <ajt@debian.org>

Attachment: pgpCc9HaBEst1.pgp
Description: PGP signature

Reply to: