[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

The sarge release disaster - some thoughts


I'm a former Debian developer, and this mail contains some subjective 
observations of mine regarding what lessions Debian might learn from 
mistakes during the sarge release cycle.

- Introduction
- Have a second plan - Discover problems early and react
- RC bugs - only a metric
- Dump testing?


These are just my personal thoughts.
If you think 90% of this email are bullshit, I'm glad to hear that you 
think 10% of this email contain valid points.

I'm talking about issues with the release management.
This isn't meant as a personal offence against former release manager
Anthony Towns and the current release managers Steve Langasek and
Colin Watson and their assistants. Everyone makes mistakes and I might 
have made more mistakes in my life than all these people together.
I do believe that the Debian release managers tried their best.
But things went wrong. It wasn't bad luck - mistakes were made.
And the mistakes should be evaluated to avoid them in the future.

The situation today is that several announced release dates for sarge 
have passed, with the first one being December 1st 2003 [1].

Debian stable is horribly outdated - e.g. it's nowadays non-trivial 
finding new hardware completely supported by Debian 3.0 .

Why don't I send this mail after the release of sarge?
Well, I thought exactly this way several months ago.
But much time passed since then, and the release of sarge is still not 
in the past.

Have a second plan - Discover problems early and react

Some pretty simple rule might have avoided several delays in the sarge 
release cycle:
If there are risks that might cause great delays, habe a second plan if 
it doesn't work out as planned.

Two examples:

The Debian installer

The timeline for the first officially announced release date was:
- August 19th 2003: announcement 
- October 1st 2003: installer has to be in a state that it only 
                    requires "last minute fixes"
- December 1st 2003: announced release date
- December 1st 2003: announcement that sarge isn't being released

I don't know whom of the people responsible to the installer had 
promised to Anthony that the installer would be ready that fast. Anthony 
said in his announcement that the timeline he set was an "aggressive 
goal". With this in mind, extra care would have been required to have a 
second plan if any part of his release plan would fail.

Not after October 1st 2003 it sould have been clear that the progress 
of the installer wasn't as good as expected. This was 2 months before 
the announced release date.

What would have been a second plan?
Nobody likes boot-floppies.
But considering the choice between releaseing Debian 3.1 with the new 
installer in 2005 or releasing Debian 3.1 with boot-floppies in 2003, it 
might have been possible finding some Debian developers hacking 
boot-floppies to use them for Debian 3.1 .

The new installer would have been ready in time for Debian 3.2 .

Would this have been an ideal solution?
But it's quite possible that it might have worked - and that it might 
have benefitted the users of Debian.

The timeline for another failed release date:
- August 2nd 2004: announcement
- August 8th 2004: "Official security support for sarge begins"
- September 15th 2004: announced release date

The milestone that included the start of the official security support 
for sarge was only 6 days after the announcement, but is was missed by 
more than 6 months.

Whyever it was expected to get testing-security for sarge that quick, it 
should have been obvious 6 days later that it wasn't possible that 

What would have been a second plan?
Use testing-proposed-updates.

Using testing-proposed-updates for security fixes, users might have 
gotten security updates one or two days after the DSA on some 

Would this have been an ideal solution? 
But it would have worked without a great impact on the release date.

RC bugs - only a metric

Nowadays, it seems the main metric to measure the quality of a release 
inside Debian is the RC bug count.

As with any metrics, work on improving the metric might make the metric 
look much better, but doesn't at the same time imply that the overall 
quality improved that much.

An example:

A major problem in Debian are MIA developers.

Consider a MUA maintained by a MIA developer with the following bugs:
- #1 missing build dependency (RC)
- #2 MUA segfaults twice a day (not RC)

Consider the two possible solutions:
1. a NMU fixing #1
2. - ensure that the maintainer is MIA
   - orphan all packages of the MIA maintainer
   - new maintainer adopts MUA
   - new maintainer fixes both bugs

The first solution has a quick positive effect on the "RC bug count" 
The second solution looks worse in this metric, but it's actually better 
for the users of Debian.

Dump testing?

It seems noone asks the following question:
Testing - is it worth it?

Several people have stated that with the size of Debian today, it 
wasn't possible to manage a release without testing with a "traditional" 
freeze (unstable will be frozen at a date, announced several months 
before), and that only testing makes releasing possible.

I have yet to see any objective evaluation of this claim.

The number of packages has increased since potato times, but at the same 
time, the number of release managers and assistants has increased from 
one to five. If the number of members has increased that much despite 
testing, perhaps the same number of people might be able to handle a 
traditional freeze? 

I remember that when testing was introduced, it was said that testing 
might always be in a releasable state. History has shown that testing 
was sometimes in a better shape as unstable, but also sometimes in a 
worse shape. Testing has some advantages over unstable (always 
fulfillable dependencies, some kinds of brown paperbag bugs are very 
unlikely), but serious data loss bugs like #220983 are always possible.

testing was expected to make shorter freezes possible.
Neither the woody nor the sarge freeze support this claim.
This might not only be the fault of testing, but the positive effects of 
testing (if any) aren't visible.

Regarding the release process, testing offers some advantages like 
having a relatively low number of RC bugs and packages built on all 

But the question is whether the same can't be achieved using a 
traditional freeze of unstable and sorting these things out during the 
first one or two weeks after the freeze.

This might make a freeze a bit longer?
But consider the disadvantages of testing:
- Testing causes additional work for both the release team and all 
  Debian developers.
  As an example, library transitions are always a pain due to testing.
  And RC bugs already fixed in unstable but not in testing need to be
- There's some collateral damage of removing packages from testing
  that depend on another package that gets removed due to some RC bug
  or that gets removed during some library transition but isn't able
  to come back due to some other reasons that might come from quite
  different causes (e.g. the blocking of a more recent KDE from testing
  makes it impossible for new vim packages to enter testing).
  Yes, there is a logic behind how testing works, but the result is 
  often chaotic and surprisingly.
- Architectures have to be in sync due to testing.
  It should be noted that all problems with architectures not being in 
  sync are only caused by testing.
  An architecture without an autobuilder is dead.
  But if an architecture doesn't has any autobuilder for two weeks
  this wouldn't cause any problems if testing wouldn't exist.
- Bugs fixed long ago in unstable might still be in testing.
  RC bugs might be tracked, but e.g. blocking a more recent KDE from
  testing might prevent any new package of vim that might fix some
  nasty non-RC bugs from entering testing.

Yes, some of these things can be done better with testing (e.g. via 
version tracking in the BTS).

But this still leaves the question whether introducing testing actually 
was an improvement compared to the previous release process or not.

Thanks for reading this email

[1] http://lists.debian.org/debian-devel-announce/2003/08/msg00010.html


       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Reply to: