Re: package testing, autopkgtest, and all that
Lucas Nussbaum writes ("Re: package testing, autopkgtest, and all that"):
> If there's a packaged tool to run the test suite on a given package,
> then it's quite easy to integrate it into my infrastructure. But I
> clearly do not have the time to get autopkgtest's code back in shape
> first.
Yes, there is such a packaged tool, "adt-run". It worked last time I
tried it.
The options are a bit complicated because you have to specify exactly
what you want to do; the script "adt-testreport-onepackage" is what
I was using to invoke adt-run, and it runs it in broadly speaking two
ways:
adt-run --source <some source package> ---...
to check that the source builds with dependencies satisfied
from the archive, and if it contains tests, run those tests
with the resulting binary packages
adt-run --binaries=install --binary <some binary package> ---...
to check that the package in question is installable
But another useful one would probably be:
adt-run --built-binaries-filter=_ --source <some source package> ---...
to check that the package's tests pass with binary packages
currently in the archive (which almost other things tests that
the package's dependencies haven't changed so as to break it)
I've enclosed a copy of the formatted manpage for adt-run, below. It
seems to be missing the --instantiate option, whose usage message is:
instantiate testbed now (during testing phase) and install
packages selected for automatic installation, even if this might
apparently not be required otherwise
> Well, it has rarely worked like that. Most of the time, I just do the
> log analysis + bug filing alone. That means that the tool to run the test
> suite must be built with filing bugs in mind: it should provide all the
> needed info in the logfile, so that developers can easily reproduce the
> failure without asking the bug reporter.
It's possible that autopkgtest could do with some work in this area;
currently the relevant output might be in more than one logfile.
Ian.
adt-run(1) Linux Programmer's Manual adt-run(1)
NAME
adt-run - test an installed binary package using the package's tests
SYNOPSYS
adt-run options... --- virt-server [virt-server-arg...]
DESCRIPTION
adt-run is the program for invoking the autopkgtest package testing
machinery.
autopkgtest is a facility for testing binary packages, as installed on
a system (such as a testbed system). The tests are those supplied in
the source package.
adt-run runs each test supplied by a particular package and reports the
results. It drives the specified virtualisation regime as appropriate,
and parses the test description metadata, and arranges for data to be
copied to and from the testbed as required.
adt-run should be invoked (unless options to the contrary are supplied)
in the top level directory of the built source tree, on the host. The
package should be installed on the testbed.
PROCESSING INSTRUCTIONS
--built-tree directory
Specifies that tests from the built source tree directory should
be run. Note that the packages that would normally be installed
as a result of * in the tests' Depends field (which includes the
case where the Depends field is not specified) are not
installed. The caller must explicitly instruct adt-run to
install any relevant packages.
--source dsc
Builds dsc. The resulting binaries will (by default) be used to
satisfy dependencies. The tests from that built tree will also
be run (by default). The ordering is significant: each --source
option should precede options whose dependencies are to be sat-
isfied by the binaries it produces.
--unbuilt-tree directory
Specifies that tests from the unbuilt source tree directory
should be run. This is very similar to specifing --source
except that a directory tree (which should be pristine) is sup-
plied, instead of a source package.
--binary deb
Specifies that deb should be used. By default it will be used
to satisfy dependencies, both during building and testing, but
not necessarily installed. The ordering is significant, as for
--source.
filename
Bare filename arguments are processed as if --built-tree,
--source, --unbuilt-tree or --binary was specified; the nature
of the argument is guessed from the form of the filename. In
the case of --built-tree, either the option must be specified,
or the filename must end in a slash; two slashes at the end are
taken to mean --unbuilt-tree.
PROCESSING OPTIONS
These affect modify processing instructions. Unless stated otherwise,
they affect all subsequent options.
--paths-testbed|--paths-host
Specifies that subsequent pathnames in command-line arguments
refer to files on the testbed, or on the host, respectively.
The default is --paths-host.
--sources-tests|--sources-no-tests
Specifies that the tests in subsequent --source and --unbuilt-
tree arguments should (or should not) be run.
--built-binaries-filter=pattern,pattern,...
Specifies that only binaries whose package names match one of
the specified patterns should be used; others will be ignored.
This option applies to subsequent --source and --unbuilt-tree
arguments.
--no-built-binaries
Specifies that all built binaries should be ignored completely;
equivalent to --built-binaries-filter=_ (since no package name
ever contains _).
--binaries=ignore | --binaries=auto | --binaries=install
Specifies that binary package (in subsequently specified
--binary arguments, or resulting from subsequently specified
--source or --unbuilt-tree arguments and not filtered out)
should be ignored, used only to satisfy dependencies, or
installed unconditionally, respectively. Equivalent to specify-
ing both --binaries-forbuilds and --binaries-fortests.
--binaries-forbuilds=...
Like --binaries= but only changes the handling during package
building: packages will be ignored, used for dependencies, or
unconditionally installed, when a source package is built.
--binaries-fortests=...
Like --binaries= but only changes the handling during testing:
packages will be ignored, used for dependencies (including as
the package under test), or unconditionally installed, when
tests are run (as a result of --source, --built-tree or
--unbuilt-tree).
OTHER OPTIONS
--output-dir output-dir
Specifies that stderr and stdout from the tests should be placed
in output-dir. These files are named argid-test-stderr and
argid-test-stdout for each test test, and log for the log tran-
script. If no output-dir is specified, or the path is specified
to be on the testbed (ie, if --output-dir follows --paths-
testbed), then the log file is instead written to the temporary
directory tmpdir if one was specified, or otherwise no separate
copy is made. Note that the log transcript output will also be
sent to adt-run's stderr unless --quiet is specified.
--user=user
Run builds and tests as user on the testbed. This needs root on
the testbed; if root on the testbed is not available then builds
and tests run as whatever user is provided.
--gain-root=gain-root
Prefixes debian/rules binary with gain-root.Thedefaultisnot-
touseanything,exceptthatif --user is supplied or root on the
testbed is not available the default is fakeroot.
--tmp-dir=tmpdir
Specifies that tmpdir should be used instead of a fresh tempo-
rary directory on the host. tmpdir will be created if neces-
sary, and emptied of all of its contents before adt-run starts,
and it will not be cleaned out afterwards. tmpdir is not
affected by --paths-testbed. NOTE again that all of the con-
tents of tmpdir will be deleted.
--log-file=logfile
Specifies that the trace log should be written to logfile
instead of to log in output-dir or tmpdir. log-file is not
affected by --paths-testbed.
--summary=summary
Specifies that a summary of the outcome should be written to
summary. The events in the summary are written to the log in
any case. summary is not affected by --paths-testbed.
--timeout-which=seconds
Use a different timeout for operations on or with the testbed.
There are four timeouts affected by four values of which: short:
supposedly short operations like setting up the testbed's apt
and checking the state (default: 100s); install: installation of
packages including dependencies (default: 3ks); test: test runs
(default: 10ks); and build: builds (default: 100ks). The value
must be specified as an integer number of seconds.
--timeout-factor=double
Multiply all of the default timeouts by the specified factor
(see --timeout-which above). Only the defaults are affected;
explicit timeout settings are used exactly as specified.
--debug|-d
Include additional debugging information in the trace log. Each
additional -d increases the debugging level; the current maximum
is -ddd. If you like to see what's going on, -d or -dd is rec-
ommended.
--gnupg-home=dir
Uses dir as the GNUPGHOME for local apt archive signing. The
specified directory should not contain keyrings containing other
unrelated keys, since adt-run does not specify to gpg which keys
to use. The default is $HOME/.autopkgtest. --paths-testbed has
no effect on this option.
--gnupg-home=fresh
Use a fresh temporary directory and generate fresh keys each
run. This can be very slow and depends on the availability of
sufficient quantities of high-quality entropy.
-q | --quiet
Do not send a copy of adt-run's trace logstream to stderr. This
option does not affect the copy sent to logfile, output-dir or
tmpdir. Note that without the trace logstream it can be very
hard to diagnose problems.
--- virt-server virt-server-arg...
Specifies the virtualisation regime server, as a command and
arguments to invoke. All the remaining arguments and options
after --- are passed to the virtualisation server program.
--set-lang=langval
When running commands on the testbed, sets the LANG environment
variable to langval. The default in adt-run is to set it to C.
--leave-lang
Suppresses the setting by adt-run of LANG on the testbed. This
results in tests and builds using the testbed's own normal LANG
value setting.
OUTPUT FORMAT
During a normal test run, one line is printed for each test. This con-
sists of a short string identifying the test, some horizontal whites-
pace, and either PASS or FAIL reason or SKIP reason where the pass/fail
indication is separated by any reason by some horizontal whitespace.
The string to identify the test consists of a short alphanumeric string
invented by adt-run to distinguish different command-line arguments,
the argid, followed by a hyphen and the test name.
Sometimes a SKIP will be reported when the name of the test is not
known or not applicable: for example, when there are no tests in the
package, or a there is a test stanza which contains features not under-
stood by this version of adt-run. In this case * will appear where the
name of the test should be.
If adt-run detects that erroneous package(s) are involved, it will
print the two lines blame: blamed-thing... and badpkg: message. Here
each whitespace-separated blamed-thing is one of arg:argument (repre-
senting a pathname found in a command line argument), dsc:package (a
source package name), deb:package (a binary package name) or possibly
other strings to be determined. This indicates which arguments and/or
packages might have contributed to the problem; the ones which were
processed most recently and which are therefore most likely to be the
cause of a problem are listed last.
EXIT STATUS
0 all tests passed
1 unexpected failure (the python interpreter invents this exit sta-
tus)
2 at least one test skipped
4 at least one test failed
6 at least one test failed and at least one test skipped
8 no tests in this package
12 erroneous package
16 testbed failure
20 other unexpected failures including bad usage
SEE ALSO
adt-virt-chroot(1), adt-virt-xenlvm(1)
BUGS
This tool still lacks some important features and is not very well-
tested.
AUTHORS AND COPYRIGHT
This manpage is part of autopkgtest, a tool for testing Debian binary
packages. autopkgtest is Copyright (C) 2006-2007 Canonical Ltd and
others.
See /usr/share/doc/autopkgtest/CREDITS for the list of contributors and
full copying conditions.
autopkgtest 2007 adt-run(1)
Reply to: