Test suite proposal
I am currently sorting through the bugs in debian-policy, and came
across this one by Ian Jackson. Any thoughts?
Julian
> I think most of us will agree that we need some kind of way of
> distributing and automatically using regression test suites. We need
> to do this in a way that allows people to download just the .deb and
> the regression tests, without the full source archive, and we need to
> do it in a way that support automatic tests of various kinds.
>
> Regression tests for many of our programs will be hard to execute
> under some circumstances, because they may want to be root, change
> configurations of things, etc.
>
> I suggest that we come up with a standard scheme for
> (a) distributing the tests
> and
> (b) expressing which ones are appropriate to run.
>
> Regarding (a), I think a separate part of the source package is
> appropriate. I suggest
> <package>_<version>.tests.tar.gz
> which contains a directory
> <package>-<version>-tests/
> This is supposed to work when unpacked anywhere.
>
> Inside <package>-<version>-tests there should be some indication of
> what tests are available. I suggest a directory `controls' or some
> such, containing one file per test.
>
> Each file would contain some basic information about the test:
>
> * Whether it needs to run as any particular user, and if so which.
>
> * How much change it will make to the system, chosen from perhaps
> - none visible
> - reconfigure or change systemwide data belonging to the package in
> question or some closely related package (scorefiles, logfiles
> etc. do not count unless they're deleted or stuffed full of junk)
> - reconfigure or change systemwide configuration belonging to other
> packages, or shared between packages. This includes randomly
> installing other packages (except test packages if they are
> harmless and are removed again afterwards). It also includes
> rebooting the machine or anything like that.
>
> * Whether the test is supposed to work given only the package and its
> dependencies, or whether it would need the Recommends as well.
>
> * Whether the test requires a tty and/or an X display, and whether it
> interacts with the user. (NB a package may be noninteractive but
> require X and/or a tty).
>
> * Other package(s) which must be installed for the test to be
> meaningful, or which must not be installed. (Depends/Conflicts)
>
> * Any other things we think are relevant here. The format must be
> extensible.
>
> * A shell script to run. The current directory will be set to the
> base of the test suite. The test script is allowed to use any file in
> the test suite, and is allowed to create files etc. inside the test
> suite. The test is considered to have succeeded if this script exits
> with a zero exit status and without printing anything on stderr.
>
>
> There should also be a `cleanups' directory in the same format, but
> whose scripts just clean up the side-effects of the tests (even if the
> tests failed), if possible. This includes deleting any files the
> tests created.
>
>
> To assist in writing test scripts, there should perhaps be support
> package(s) which contain useful utilities for (eg) applying regexps to
> the output of things, etc.
>
> Ian.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Julian Gilbey, Dept of Maths, QMW, Univ. of London. J.D.Gilbey@qmw.ac.uk
Debian GNU/Linux Developer, see http://www.debian.org/~jdg
Reply to: