Bug#41902: Test suite proposal

[Ian Jackson <ian@davenant.greenend.org.uk>]

Package: debian-policy
Version: 3.0.0.0

I think most of us will agree that we need some kind of way of
distributing and automatically using regression test suites.  We need
to do this in a way that allows people to download just the .deb and
the regression tests, without the full source archive, and we need to
do it in a way that support automatic tests of various kinds.

Regression tests for many of our programs will be hard to execute
under some circumstances, because they may want to be root, change
configurations of things, etc.

I suggest that we come up with a standard scheme for
 (a) distributing the tests
and
 (b) expressing which ones are appropriate to run.

Regarding (a), I think a separate part of the source package is
appropriate.  I suggest
   <package>_<version>.tests.tar.gz
which contains a directory
   <package>-<version>-tests/
This is supposed to work when unpacked anywhere.

Inside <package>-<version>-tests there should be some indication of
what tests are available.  I suggest a directory `controls' or some
such, containing one file per test.

Each file would contain some basic information about the test:

* Whether it needs to run as any particular user, and if so which.

* How much change it will make to the system, chosen from perhaps
   - none visible
   - reconfigure or change systemwide data belonging to the package in
     question or some closely related package (scorefiles, logfiles
     etc. do not count unless they're deleted or stuffed full of junk)
   - reconfigure or change systemwide configuration belonging to other
     packages, or shared between packages.  This includes randomly
     installing other packages (except test packages if they are
     harmless and are removed again afterwards).  It also includes
     rebooting the machine or anything like that.

* Whether the test is supposed to work given only the package and its
dependencies, or whether it would need the Recommends as well.

* Whether the test requires a tty and/or an X display, and whether it
interacts with the user.  (NB a package may be noninteractive but
require X and/or a tty).

* Other package(s) which must be installed for the test to be
meaningful, or which must not be installed.  (Depends/Conflicts)

* Any other things we think are relevant here.  The format must be
extensible.

* A shell script to run.  The current directory will be set to the
base of the test suite.  The test script is allowed to use any file in
the test suite, and is allowed to create files etc. inside the test
suite.  The test is considered to have succeeded if this script exits
with a zero exit status and without printing anything on stderr.


There should also be a `cleanups' directory in the same format, but
whose scripts just clean up the side-effects of the tests (even if the
tests failed), if possible.  This includes deleting any files the
tests created.


To assist in writing test scripts, there should perhaps be support
package(s) which contain useful utilities for (eg) applying regexps to
the output of things, etc.

Ian.