Hello world,
Sorry: this is long, and it's probably all relevant. Get a snack and
something to drink while you read it maybe.
So, it's time we start doing more than think about the next release. Since
I'm all for aggressive goals, let's aim for sometime in December -- how
about 2003-12-01 00:00:00 UTC?
Perhaps I shouldn't have suggested getting the drink. It's okay, get a
cloth, I can wait, and you're not going to be able to read the rest of
this email until you clean that monitor up.
Anyway, obviously this is aggressive, but I think it's entirely
achievable. Exactly how we achieve it may be a little bit unclear, but
hopefully the following will address some of that. To setup the context,
we'll begin by discussing some of our longer term goals wrt releases,
and the problems we're having with those.
One reasonable question some people ask is why we have stable releases at
all. A fair number of people are happily running testing and unstable on
their desktops and servers, and if they're not already as secure, reliable
and featureful as stable is, they can certainly be made to be. The
key difference that we want to keep is updates: testing and unstable
will change the way your system works in small ways quite regularly,
requiring constant attention from a sysadmin to ensure the system still
does what it's supposed to. By contrast, the regular security updates and
occassional point releases of stable generally make a point of not adding
any new features, nor breaking *any* behaviour admins or users might
be relying on. Basically, it's the difference between having a sysadmin
spending fifteen minutes every day or week tweaking your server to keep
it running, or having a sysadmin come in for a week once a year to do a
major upgrade. In some environments the latter is by far the more feasible
option, and it's there that Debian stable releases are most useful.
One major issue for these types of users is predictability: you don't
want to schedule your admins to come on site, then find you need to do
a major Debian upgrade a month later, and have to bring them back out.
The same argument applies even more so to people trying to pre-install
Debian and ship it to people: if features you need are only available
in testing/unstable, you either need to wait an indeterminate amount of
time until a release happens or maintain a possibly large set of security
updates in a timely fashion yourself.
So one thing we've been working on, and will continue to work on, is
ensuring we can plan a release well in advance of it happening, and that
our plans actually pan out. One of the functions of setting as aggressive
a goal as we are this time is to see just how effective we can be.
The other goal of our "releases", using the term more loosely to refer to
our daily releases of packages in suites other than stable, is to support
people who *are* willing to tweak their system on a daily or weekly
basis. The tradeoff still to be made here is one of "features" versus
"reliability"; the value of new changes, versus the risk that the changes
will break something you need. Essentially we want to allow people to make
a choice between newer software that we're confident works fairly well,
and software that's as new as possible, possibly so new we can't make any
promises at all about it. We're trying to make this distinction with the
"testing" and "unstable" suites, letting people choose to take all their
software one way or the other, or using pinning, just some of it.
Our primary failure here is that in order to keep testing as current as
we can, we're skimping on keeping unstable as new as we might otherwise.
This includes being slow to upload released software that we're not as
confident with, and being very hesitant to upload CVS versions of packages
at all. These aren't necessarily bad choices. They are, however, choices
we should be aiming to leave up to our users, not making ourselves.
Hopefully none of the above should be particularly controversial. People's
priorities will obviously lie in different areas, but keeping the above in
mind should allow us to avoid getting in conflict too much.
Because conflict is probably going to be coming up. One thing that an
aggressive schedule implies is that we're going to have to experiment a
little: if we could just keep doing the same old thing, it'd be an easy
schedule, if we only had to try a little harder, it'd be a difficult
schedule. This isn't either of those things, so in order to try to
achieve it, we're going to mix things up a bit.
("Try to achieve it"? Tsk, what would Yoda say?)
The first new feature we have for this release is distributed release
management. It's hierarchial, not peer to peer, but hey, it's a start.
After replying to the "Help wanted" mail from March, and surviving a
series of grueling challenges on the debian-release list, Joey Hess
<joeyh@debian.org>, Steve Langasek <vorlon@debian.org> and Colin
Watson <cjwatson@debian.org> (Kamion) have each been granted powers
above and beyond mere mortal developers and the grand title of Release
Assistant. In particular if you don't understand why your packages aren't
getting into testing, they'll usually be able to either explain why, or
fix it. They're also able to offer good advice on all manner of things
should you be feeling confused, or need some help. They all IRC, and
amongst all of us, we should cover most timezones. So don't be afraid
to ask for help or advice. If you don't know what to work on, don't be
afraid to ask for suggestions.
In order to ease some of the pressure on unstable, we're encouraging
greater usage of experimental. The plan here is that you should upload the
latest, release-quality packages to unstable; and the latest development
packages to experimental. This means daily snapshots, CVS versions,
alphas, pre-releases and so forth. If you're currently maintaining
a foo-snapshot package in unstable, you should consider dropping the
-snapshot, and uploading it to experimental. It also means you should make
an extra effort to ensure that what you put in unstable is maintained at
the quality you'd expect from a Debian stable release, although obviously
with far more frequent changes. You won't always succeed, unless you're
some sort of packaging God, but that should definitely be your aim.
For people uploading to experimental, you should be aware of the following:
* experimental uses the same overrides as unstable. Packages
already in unstable can be uploaded to experimental without
requiring any delay while an ftpmaster reviews licenses and
such; and similarly once a NEW package uploaded to experimental
has been approved, no further delay will be required when it's
uploaded to unstable.
* there is an "experimental" tag in the BTS you can use to help
differentiate bugs in the unstable and experimental versions
of your package. Automated version tracking for the BTS is
under development, but in the meantime this is your best bet.
* Bugs fixed in uploads to experimental will be automatically tagged
as "fixed-in-experimental", rather than closed.
* experimental is not, and will not be, automatically built by our
standard buildd infrastructure. Instead, you should ask other
developers who own machines of the various architectures to
do a build for you; please wait for your source upload to hit
the mirrors, and include the package name, version, and a list
of any dependencies that should be satisfied from experimental
rather than unstable in your request.
We hope that eventually we'll be able to incorporate this
mechanism into the standard buildd infrastructure, but it's
expected experimental will only ever be built-on-request.
On-request experimental builders are encouraged to make an
unsigned .changes file, a build log, and, ideally, the finished
build tree (with .o files, etc) available to the maintainer
along with the compiled .debs, rather than uploading straight to
the archive.
Developer accessible porting machines are listed at
http://db.debian.org/machines.cgi, and unstable chroots are
available for almost all architectures. Contact debian-admin
about any packages you need installed.
* apt-get will not automatically install packages from
experimental when you add it to your sources.list, you'll need
to select them manually, or make use of pinning.
* experimental is otherwise the same as the other suites, with the
same restrictions. Once you upload an .orig.tar.gz, eg, you
cannot change it, including when you're eventually ready to
upload to unstable.
Note that once you upload to unstable, you're committed to making what
you've uploaded release-quality. If you have any doubts, try to get rid
of them by uploading to experimental first.
Please be careful when you're using experimental -- we haven't done much
with it now, so some things can get screwed up. We've already had one
instance of this with a sparc build of an experimental version of glibc
making its way into unstable; katie should REJECT that sort of thing now,
but there may yet be other kinks that haven't been worked out. Yet.
Obviously, we also need to massively reduce the release-critical bug
count. As such, you should consider every weekend from now until the
release a "bug squash party" in the usual sense. If you've got some free
time, look through the release-critical bug list and help reproduce bugs,
supply patches, test fixes, or do NMUs. Of interest, if you haven't been
paying close attention, might be the little extra bits of information
in the RC bug list: bugs tagged help, patch or pending are now coloured
purple, green and orange to make them a little easier to spot, and
the number of bugs marked patch and pending are listed at the top to
give an overview of how we're going. At the rate we'd like to be going,
we'll need to shorten the RC bug list by between 50 and 100 bugs a week,
or close about 10-20 RC bugs a day. On average, every developer will
need to at least help resolve one RC bug every two weeks from now until
release, and since simply reading this puts you slightly above average
in the activeness stakes, you'll need to do slightly more than this for
us to meet our goal.
To help coordinate RC bug fixing and BSPs, you can make use of the
"BugSquash Claims" page, at:
http://bugs.debian.org/cgi-bin/claims.cgi
It's populated by creating files in
master:/org/bugs.debian.org/bugsquash/claims/
named after your username @debian.org, and entering one bug number per
line. Developers who're participating in the BSP should enter the bugs
they're working on into there, and might like to check to see if anyone
else has "claimed" the bug to avoid duplication of work. Once you've
got the bug patched and NMUed, it'll remain on the page for a while,
for bragging rights. Also interesting might be urls such as:
http://bugs.debian.org/cgi-bin/claims.cgi?include=ajt@d.o
http://bugs.debian.org/cgi-bin/claims.cgi?exclude=pending,fixed
Remember, we're not prospecting for gold here: claims are advisory
only; if you really want to fix a bug, and can't seem to get a hold of
whoever's got a claim, don't worry about it -- fix the bug yourself, and
mail your findings to the bug report. If you want to do duplicate work,
that's your business.
While the above is nice and all, and will hopefully be useful during
future releases as well, it's probably not going to shake things up
quite enough to get sarge out by December. As stronger medicine, we're
going to spend the next few weeks -- in particular, the period between
Saturday 23rd August and Sunday 14th September -- experimenting with
an excessively liberal policy for NMUs, namely, "NMU whatever you like,
whenever you like". You can look at this in a few ways:
* If you're uploading NMUs for the BSP during this period,
you should skip DELAYED, and just upload straight to
queue/unchecked (f.k.a Incoming).
* You should consider yourself a co-maintainer of every package
in the distribution, and feel obliged to fix the bugs you see,
and apply any outstanding patches.
* You should consider this perhaps your one and only chance to get
some fix you desperately need into sarge, whether it be to a
release-critical bug, an important bug, a normal bug, a minor
bug, or in many cases even a wishlist bug.
There are caveats. You should follow the usual traditions wrt NMUs:
use a point version number, make sure the problems you're fixing are
reported, mail patches to the BTS, make sure the maintainer knows what
you're doing, don't make irrelevant changes to the package, don't make
major changes to the package, be respectful of the maintainer at all
times especially when disagreeing, fix any bugs that you cause even
more promptly than you would in your own packages, and generally act
according to the highest standards.
If you're going to mention this 0-day NMU rule in passing to anyone,
make sure you cover the entire paragraph above, and emphasise it. We're
temporarily reducing the level of courtesy and bureaucracy involved in
making NMUs, not the quality we expect of them.
Additionally, you should realise that this is a strictly limited offer,
so while you should take advantage of it, don't waste your time or our
users' time, adding features that you know the maintainer will simply
remove in his or her next upload.
On the other hand, as a maintainer, while you're certainly encouraged
to spend the next few days fixing any outstanding issues that you think
might be targetted come the 23rd, one of the points to this experiment
is to encourage you not to see an NMU as a particularly bad thing. So
try not to be too NMU-averse.
If you find you have a particularly good or bad experience with this,
as either a maintainer or an NMUer (or possibly as a user), please keep
a note of your thoughts and post them to debian-devel on the 15th of
September, so that we can do an experiment report, determine what worked
and what didn't, and see if we want to update our regular NMU policy
(or if individual maintainers might want to update their personal NMU
policies).
This is a risky policy. That's why it ends early, so we still have time
to clean up any mess. Do take care.
Continuing on in the "no pain, no gain" school of thought, we'll also
be attempting to reduce the number of release-critical bugs affecting
sarge by removing a significant number of packages. Candidates for
removal will be all packages with release-critical bugs: if the version
of your package in testing has release-critical bugs it will probably
be removed from testing; if it has release-critical bugs in unstable,
and looks otherwise unmaintained under a guilty-until-proven-innocent
policy, there's a real chance it will also be removed from unstable.
This is the only warning you'll receive; the best way to ensure this
doesn't happen to you is to do a better job maintaining your packages than
I do maintaining mine. Keep your RC bug count down to approximately zero,
and make sure you don't leave any open for more than about a week.
Reply to bugs, and use tags to reflect their status. It's not a
particularly high standard, so you really shouldn't have any problems.
Should your package be removed from testing, you'll probably have time
to fix the bugs in the unstable version, and let that propogate back
into sarge. If you're worried about this, contact myself or any of the
release assistants to find out what's going on. The status of packages
being considered for inclusion in testing is available at:
http://ftp-master.debian.org/testing/update_excuses.html.gz
Ask around if you don't understand what it some of the cryptic comments in
the contents mean.
Next on the list of changes to the way we do things is working out
what's a "severe policy violation". In particular, we're going to try
moving away from subtle semantic distinctions in the forty thousand word
document that is Debian Policy, and switching towards a simpler, bullet
point list of release-critical issues. This is located at
http://people.debian.org/~ajt/sarge_rc_policy.txt
and should be considered canonical. Please briefly skim it just to make
sure you're familiar with all the issues listed. Also of interest may
be the introduction of the "sarge-ignore" tag, which will be used to
indicate which release-critical issues will be ignored for sarge. Note
that that tag should *only* be used by the release manager.
Going back to our original problem, namely, release predictability,
we have three things to address. One is making it clear what we've
decided. Another is making it clear that we can achieve what we've decided
-- that's what the above attempts. The other is making it clear whether
or not our plans are actually working out. In aid of that, here is a
rough dateline of what we're aiming to achieve, and when.
Dateline!
* August 19th (now)
0-day NMUs (as of the 23rd)
Development versions of packages expecting to include
major changes in sarge uploaded to experimental.
Drop the RC bug list by ~150 bugs to ~700
(via removals and fixes)
Beta testing of debian-installer by adventurous users
(subscribe to debian-boot, and try CVS or the images at
http://people.debian.org/~tfheen/d-i/images/daily/)
Beta testing of upgrades over the network and with sarge
CD images
(available under http://gluck.debian.org/cdimage/ . May
be bootable, depending on your luck. Only for the truly
adventurous)
* September 1st
0-day NMUs
Last major changes to toolchain packages
Drop the RC bug list by ~200 bugs to ~500
(via removals and fixes)
HOWTO use debian-installer to install sarge posted to
-devel-announce (volunteers appreciated)
Beta testing of installation with sarge CD images by
adventurous users
* September 15th
Drop the RC bug list by ~150 bugs to ~350
(via better accounting, removals and fixes)
Beta testing of the installation (debian-installer, tasksel,
base-config, package installs, CD images, everything)
debian-installer hackfest at Oldenburg, Germany
Last major changes to major packages uploaded to unstable
* October 1st
Drop the RC bug list by ~150 bugs to ~100
(via removals, fixes and workarounds)
1st test cycle, public request for comments
Last minute fixes and changes to the installer
* October 15th
Drop the RC bug list by ~100 bugs to ~0
(via fixes, workarounds, wishful thinking and ignorance)
Final, last minute, low-risk bug fixes only
Get support for sarge up and running on security.debian.org
2nd test cycle, public request for comments
* November 1st
Get any remaining security bugs patched
3rd and final test cycle, public request for comments
Minimal package updates -- manual approval by RM only
* November 15th
Final tweaks
Minimal package updates -- manual approval by RM only
* December 1st
Release
You're strongly encouraged to work out your own schedule for packages you
work on, to make sure you don't leave it too late to get any changes you
want properly tested before the release. Please don't forget to include
the lag from the 10-day testing delay. Also make sure to include some
leg room if you depend on packages that have a tendency to be buggy
(glibc, for example).
So, that's the plan, and the end of this email. If the heart attack at
the beginning didn't get you, perhaps the exhaustion will.
What's that? You're hungry? Well, I did say you should get a snack. That's
not what you meant? You're hungry for *action*, you say? Well, get to it!
Cheers,
a "corny lines 'r' us" j
--
Anthony Towns <ajt@debian.org>
Debian Release Manager
Attachment:
pgprW7HKEXWK6.pgp
Description: PGP signature