building test suites from bug reports

[Raul Miller <rdm@test.legislate.com>]

I'm interested in the opportunities afforded by our bug tracking
system for building test suites.

Informally, there seem to be several classes of bugs for this purposes:

(1) Bugs which may be fairly directly used to specify a repeatable
test against the software represented in a package.

(2) Bugs against the package where it's difficult to derive a test
[for example: where we need a X-11 test harness to replicate the
bug without manual intervention.]

(3) Bug reports which don't directly lead to a test, but which maybe
are FAQ material (either because it's not a bug, or because it's a
documentation issue).

(4) Bug reports which, when considered from a testing viewpoint, are
more indicative of something systemic about debian as a whole than
about the particular package.

Now, the question is, what should we do about this?

To bring to bear the parallelism that we should, we need some very simple
concepts which just about everyone can easily grasp.

At the moment, I'm favoring extending the bug tracking system -- that is,
when closing the bug report the package maintainer should assign the bug
to the appropriate "testing" class.  [Presumably we could add more testing
classes if that becomes important.]  If the class of test is specific to
the package then building the test remains the maintainer's responsibility
[though, of course, anyone else could write up a test and submit it --
and this should be encouraged for cases which have hung around for a
long time].  If the class of test is more pertinent to debian as a whole
then it should be assigned to the appropriate entity (ftp.debian.org,
policy, lintian, dpkg, ...).

What do people think about this strategy?  Can we do better?

Thanks,

-- 
Raul


--  
To UNSUBSCRIBE, email to debian-testing-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org