Need for a test suite
Hi,
I was just watching a discussion on #debian-devel and the subject of QA
has come up yet again.
The questions that pop up are basically the same as always, namely, how
can we have some sort of QA procedure that *actually* helps in speeding
up the release process? The answer is at least partially evident: we
need people actually doing QA. It looks like Raphael, whom, AFAIK, was
playing QA coodinator, got caught up in RL, and other people are in a
similar situation. Can we have an informal heads count here?
Once we have the people, the other part of the answer is not that
clear. IMO, a possible solution would be to have a sort of test suite.
Since the size of Debian together with the size of the QA team makes
the very idea of a generalized test suite laughable at the moment.
Phil did develop such a beast, but if you look in /usr/lib/debian-test,
unless you have a particular set of packages, you'll note it's empty.
Woody's release is hopefully too close to attempt getting these scripts
written for a significant part of Debian.[1]
Last year a group of developers demed the idea of using a task manager
as a good one, in order to help introduce some order into the apparent
chaos that QA is right now. One example of such software is
SourceForce (SF is more than a task manager, but the task manager
contained there in is a good one, IMO). Roland Mas has packaged
SourceForce for Debian, and as far as I understand it's installed on
some machine (SPI's, not Debian's). The idea here would be to create
tasks that can be taken by different people, which would gain us two
things: splitting the work into small realistic chunks and giving us a
scoreboard where we can come and say "yes, this was checked". This can
also serve as a system where people can simply report "I tested this
particular task on such a system". Of course there's at least one
flaw: we need to push other developers arround in order to get the task
list populated.
For now, it seems the most urgent task in this yet non-existant list is
to test the boot-floppies. For *woody*. No, not debian-installer.
The woody boot-floppies. The people doing QA should forget about
debian-installer at the moment. AFAIK, only Adam di Carlo is really
working on them (I'll be glad if I get corrected). The first thing to
do to get this going is *building* the boot floppies out of CVS. I'm
not sure if that's doable atm. Adam and the b-f team have done a great
job documenting all this stuff, so it's not that hard to get up and
going. Probably the best way to get this started is having the boot
floppies built on a regular basis, with a list of changes wrt to the
last built (trivial with cvs2cl) with, perhaps, an addiotional list of
stuff to actually test.
The other question that popped up is "what's holding up packages in
unstable and what can QA do to help that?"
It's not hard to tell that I've made a lot of this mail on the spot,
mostly triggered by ideas thrown on #d-d. I'm not trying to get things
going on *right now*, but to get a discussion started about how to go
from here. The general consensus is woody won't get out of the door
unless it starts to get (systematically) tested soon.
TIA,
Marcelo
[1] Good candidates for these tests are security fixes tests, such as
the one provided by rsync, to ensure that bugs that once existed don't
crop up again.
Reply to: