Automated testing - design and interfaces
Note: that this is one of two messages on roughly the same topic.
This message will deal solely with TECHNICAL issues. If you have some
technical followup then please go ahead and reply here. If you have a
complaint or comment about my or Ubuntu's approach, please reply in
debian-*project* to my other message, titled `Automated testing -
politics, information, and Ubuntu's plans'.
The first two paragraphs are common to the two messages:
One thing that the Debian universe is lacking is a good way to
automatically test packages. This makes it hard to spot regressions,
and difficult to be systematic.
Ubuntu are proposing to invent a system to allow us to do automated
regression tests. The full plan consists of a number of pieces and
can be read here:
If you're interested in this subject I'd appreciate it if you could go
and take a look. Feel free to make your comments here on -policy, or
any other appropriate channel.
One of the important parts needed here is an interface for packages to
supply tests. That is, a standard way for a Debian-format package to
enumerate and describe the tests it supplies, and to allow a test
harness to invoke them.
Note that the point is to be able to test the _actual package_, _as
installed_ (eg on a testbed system). This is much better than testing
the package from the source treeu during build time because it will
detect packaging mistakes as well as program source problems and as we
know packaging mistakes of one kind or another are one of the main
causes of problems.
It seems to me that the right way for packages to offer their tests in
the source tree. As I say in the wiki page:
Other possibilities include a special .deb generated by the source
(which is a bit strange and what happens to this .deb and will make
it even harder to reuse upstream test suites), or putting them in
the .deb to be tested (definitely wrong - most people won't want the
tests and they might be very large) or having them floating about
separately somewhere (which prevents us from sharing and exchanging
tests with other parts of the Free Software community). The source
package is always available when development is taking place.
To save you looking at the wiki page for the details of the draft
interface spec. I have reproduced it here:
The source package provides a test metadata file debian/tests/
control. This is a file containing zero or more RFC822-style
stanzas, along these lines:
Tests: fred bill bongo
Restrictions: needs-root breaks-computer
This means execute debian/tests/fred, debian/tests/bill, etc.,
each with no arguments, expecting exit status 0 and no stderr. The
cwd is guaranteed to be the root of the source package which will
have been built (but note that the tests must test the installed
If the stanza contains:
then we execute path-from-source-root/fred, path-from-source-root/
bar, etc. This allows tests to live outside the debian/ metadata
area, so that they can more palatably be shared with non-Debian
Any unknown thing in Restrictions, or any unknown field in the
RFC822 stanza, causes the tester core to skip the test with a
message like `test environment does not support "blames-canada"
restriction of test "simpsons"'.
Restrictions: modifies-global-data needs-x-display
etc. - moves complexity from individual packages into central
A basic test could be simply running the binary and checking the
result status (or other variants of this). Eventually every
package would to be changed to include at least one test.
Ideally eventually where possible the upstream regression tests
could be massaged so that they test the installed version. Whether
this is possible and how best to achieve it has to be decided on a
Even integration tests can be represented like this: if one
package's tests Depend on the other's, then they are effectively
integration tests. The actual tests can live in whichever package
is most convenient.
So, what do you think ?
(wearing both my Debian and Ubuntu hats)