Bug#741330: autopkgtest: please add ability to wrap a script/runner/adverb around existing tests
autopkgtest is not the only implementation of "as-installed" package testing.
Other options include:
* GNOME InstalledTests (a test is a .desktop-style file in a subdirectory
* Autotools "make installcheck"
* NetBSD kyua <http://code.google.com/p/kyua/wiki/About>
* Perl-style TAP tests
* whatever other ad-hoc frameworks people have come up with,
e.g. /usr/lib/telepathy-gabble-tests/twisted/run-test.sh vaguely mimics
Autotools output and was designed to be screen-scraped by Linaro's LAVA
It would be nice to be able to wrap tests designed for one of these other
frameworks to make them conform to autopkgtest syntax and requirements.
For instance, some tests are executable as-is, but should be wrapped
in "adverb" commands like env, xvfb-run and/or dbus-run-session; and the .test
files representing GNOME installed-tests are not directly executable at all,
but can be run by wrapping them in gnome-desktop-testing-runner.
autopkgtest currently requires that each test is a separate
executable in debian/tests (or some other single directory in the source
tarball). In practice, this results in the entire upstream test-suite
becoming a single autopkgtest, for instance
making it impossible for a generic autopkgtest-based framework to track
failures over time: instead of "tests 'foo' and 'bar' have always passed,
but test 'baz' regressed between versions 1.2 and 1.3" the best available
report would be "the only test is 'installed-tests' and it regressed between
1.2 and 1.3".
Straw-man syntax proposal:
If the Interpreter for a stanza is specified, the autopkgtest
implementation will look for it in the Tests-Directory (defaulting to
debian/tests as usual). If it is found there, it will be made executable
and executed once for each test-case listed in Tests, passing the
test-case's name as an argument.
If the Interpreter does not exist as a file in the Tests-Directory,
it is taken to be a shell command-line to which the
test-case's name will be appended; commands which can typically be
used in this way include env, dbus-run-session and xvfb-run.
If an Interpreter is specified, the Tests are not required to exist
as files in debian/tests, and may contain "/" in addition to the
characters allowed in Debian package names.
Example: wrapping non-executable tests in commands from $PATH
Interpreter: xvfb-run gnome-desktop-testing-runner
Depends: gnome-desktop-testing, xvfb, ...
Example: wrapping tests that need a special environment
Tests: fred bill bongo
Depends: dbus, dpkg-dev, xvfb, ...
exec dbus-run-session -- xvfb-run -- env TEST_OPTIONS=all,verbose -- \
It would also be nice if the Tests could be the output of a script, or
listed in a file that may be created by the package build, or something,
so that the list of test cases doesn't have to be hard-coded in
debian/tests/control. That way, GNOME packages could use something like
gnome-desktop-testing-runner --list glib/ | sed -e 's/ (.*)$//' | LC_ALL sort -u
to list all their test-cases automatically.
-- System Information:
Debian Release: jessie/sid
APT prefers proposed-updates
APT policy: (500, 'proposed-updates'), (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 3.13-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages autopkgtest depends on:
ii apt-utils 0.9.15.5+b1
ii debhelper 9.20140228
ii pbuilder 0.215
ii python 2.7.5-5
ii python-debian 0.1.21+nmu2
autopkgtest recommends no packages.
Versions of packages autopkgtest suggests:
pn autopkgtest-xenlvm <none>
ii lxc 0.9.0~alpha3-2+deb8u1
ii qemu-system 1.7.0+dfsg-3
ii qemu-utils 1.7.0+dfsg-3
ii schroot 1.6.8-1
-- no debconf information