[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

d-i beta4 release process retrospective (long)

In case you haven't heard, it's released!

The big bad news is no mips yet, but it will be back very soon I think.
Days, not weeks. The lack of s390 is not suprising at this point, but
does mean we've missed the goal of supporting all target arches in this
release. Sparc is also somewhat problematical with both CD and
framebuffer problems.

But I think it'll turn out to be a pretty good release, and the sheer
number of architectures, languages, fixes, and enhancements is stunning.
Thank you all for putting it together, I look at d-i and am amazed that
I started this project, it has so many good things in it that I never

This beta was by far the hardest for me (and probably others) of the
four we've done, which is not too suprising, because each has been
significantly harder then the one before it.

I've kept a log of the major events of this past week leading up to the
release, which I recommend reading to get an idea of some of the
problems and processes: http://kitenet.net/~joey/blog/entry/beta4/

I'm afraid things have reached the point where we'll have to handle the
next d-i release differently if we want to be able to release it with
less than two solid weeks like the one described above; and I suspect
that two weeks of this would completly burn me out. I was rather burnt
out by Thursday. (And I apoligise for the times I was snappish with
various people.)

As I see it, some of the general problems with how this past week went
are these:

        - Lack of timely testing.
        - Quick-fixing one problem often leads to a new one.
        - Image builds are slow and somewhat fragile, and we only get
	  one build per day max, so only so many chances to get it
        - Freezing the base dependencies helped, but did not completly
          alleviate debootstrap problems. The two debootstrap issues (one
          indirectly caused by the fix to the other one) were a major
          problem still.

The testing problem is the big one, we had some things that were simply
not tested, at all, ever, until 2 days before release. I'm amazed these
things got in in a working state. Most of the week was spent playing
wack-a-mole, I'd think I had a releasable set of images, and then a new
major problem would pop up. This was very frustrating, and wasted a lot
of people's time, not least the tireless ftp-masters.

I have some ideas about solving some of the problems, some of them
mutually exclusive. In approximate order from the no-brainers to the
serious blue sky:

Someone to deal with stuff while I sleep.

	Most of my work during a release is coordination, collating
	problems, deciding which are release blockers, making sure
	someone is working on each, keeping track of what images are
	working. This is fairly parelizable with good communication
	between people doing it, and always having someone awake who has
	the whole release in their head would help to speed things up.
	Mail me if you're interested in doing this.

Automated BYHANDling of the d-i tarball.

	The ftp-masters are doing an awesome job of manually untarring
	the d-i tarballs as they're autobuilt and uploaded, but there is
	still a delay here, there are still hours when no ftp-master is
	available to do our bidding right now, and I think this could be
	The main problem is not getting the images into the archive, but
	getting the upload accepted ASAP so the autobulders can start on
	it. It takes some autobuilders a long time to build d-i, 12
	hours is not uncommon to get a full set of builds. Even a few
	hours shaved off the front of this can be significant.

	A two stage queue might work, with stage one (byhand)
	automatically accepting d-i builds, and dumping them in stage
	two, where the ftp-masters could still check and untar them

Speeding up our pulse.

	Our pulse is the daily mirror sync, we currently can only test
	one set of images and udebs per day, and we spend a fair amount
	of time just waiting for dinstall.

	I've talked about this with James Troup before. James seemed to
	think that we could switch to two per day, but of course this
	has large ramifications, both in impact on the mirrors, and on
	Debian as a whole, right down to our users who may be suprised
	to see this change. I think it will have largely good
	consequences, and if d-i had not eaten my life lately, I might
	have given a talk about this and related issues at DebConf 4. As
	it is, I'd like to maybe have a BOF dealing it, especially if
	some of the archive people are available there.

	Short of making that large change, there are things we can do
	such as setting up our own repository and automated builds
	against it that happen more often than the daily oficial one.
	Remember, we did something similar at DebConf 3, and rather
	accellerated the pace while we were there. This will involve a
	fair bit of work though.

Facilitate better testing.

	We need to work on getting people with the time to do testing
	together with machines to do useful testing on. 
	I'd like to see a remote test lab that could be used for tests
	of most arches. I think that Bdale has something close to this
	for several arches, but it's not yet publically available. 
	We also should see how Debian can help meet other testing needs
	of d-i developers. That's still mostly in tbm's court, I know
	he's been doing this kind of thing already.

	We should also look at recruiting our user base. In fact, we've
	done this fine for i386, but installation reports for most other
	arches are too rare to get a daily impression of how those
	architectures are doing. Most arches have a community around
	them, and we need to do better at tapping into that community.
	If you're involved in such an arch, and such a community, please
	consider what you can do about this.

Freeze base during d-i freeze.

	It's very hard to aim for a release when half of the system
	being released is a moving target. This gave us the tty
	permissions problem, for example. It could be fixed by
	temporarily halting any base package progression to testing
	while d-i was in the throes of release.

	There are probably all kinds of problems with doing this, what
	do you think, AJ?

	The flip side of this coin is we could just ignore the base system
	that d-i delivers, with similar results, but a much lower
	release quality. Bear in mind that installation reports
	increasinly report on the whole Debian install right down to X,
	and since noone else is collecting those reports for the sarge
	release (the debian-testing list is dead), this is a good thing.
	Releasing with a prossibly fubar base would lose this.

Swtich to testing-style udeb propigation.

	It would really be better if instead of copying a release to
	testing in one lump, we copied individual udebs to testing when
	they're ready. I have outlined some of the technical problems
	with using purely automated system as used for deb testing
	propigation in an earlier mail, but maybe we could do a manual
	propigation. It would probably involve a lot of work, and would
	probably slow down what did get in, but that may not be
	inappropriate at this point.

Don't try for such a perfect and well-tested release.

	It could be argued that beta4 is really over-tested for an
	interim beta release. I suppose time will tell. If we set our
	sights lower, getting the next release out will be easier. On the
	other hand, we need to get a stable-quality release out
	eventually, and one point of the betas has been to learn how to
	do so, and improve the process and each time, which we've done so
	far, at the expense of some considerable pain.

Require successful installs on all arches and major platforms before

	This idea turns the release process on its head. Instead of
	freezing and trying to hit a release at a given fixed date in
	the future, this has us beginning to seriously test, and collate
	reports, well before we'll probably release. If a build goes
	green on all arches, start the (probably very short) release

	The big problems with this are that the release team needs to be
	able to plan ahead, they have enough problem with there being no
	set date for the Debian release; and that this will need a lot
	more testing that we currently seem to muster. In fact, it
	almost calls out for automated testing -- automated installs on
	11 architectures with regression testing! -- to work.

	Of courss that could be phased in gradually. I'd be happy to see
	automated testing available on any single arch.

see shy jo

Attachment: signature.asc
Description: Digital signature

Reply to: