using autopkgtest for functional upgrade testing
I have been experimenting with using autopkgtest to run functional tests on
upgraded systems. This type of automated tests consist of using a testbed on a
previous release (currently wheezy), then upgrading the system to the next
release (currently jessie) after installing the test dependencies, but before
running the tests.
I have been testing this initially with the ruby-defaults source package, using
the following code in the source package + a patched autopkgtest (patches
follow this message):
The adt-run invocation looks like this:
$ adt-run \
--user debci \
--tests-dir debian/upgrade-tests/wheezy \
--upgrade-to unstable \
--built-tree . \
--- schroot debci-wheezy+backports-amd64
I intend to eventually add support for such upgrade tests to debci.
I would like input on a few points:
0) how to declare the existance of such tests in debian/control.
According to the autopkgtest README.package-tests, the `Testsuite` field is
supposed to be a comma-separated of values, which is fine. We just need to
agree on what value to use there.
I thought about something like
In a package that also contains a regular DEP-8 test suite, it could look like
Testsuite: autopkgtest, autopkgtest/upgrade/$codename
1) a new DEP-N, based on DEP-8
I would like to propose a DEP-N spec for this at some point that would refer to
DEP-8 as a basis, but I would like to drive DEP-8 to an acceptable state before
that. Currently DEP-8 reads as just "go read README.packages-tests at the
autopkgtest source tree".
So: would be there any concerns about taking the contents of
README.packages-tests as a draft test for the DEP-9 spec?
I have also written a patch for sadt(1) from devscripts to implent the
--tests-dir, which is simple enough and can be pushed as soon as we agree to
add --tests-dir to autopkgtest:
A summary of the patches that follow this message:
Antonio Terceiro (3):
adt-run: add --tests-dir option for selecting tests directory
adt-run: add --upgrade-to option for upgrade tests
adt-run: set APT_LISTBUGS_FRONTEND=none when installing packages
debian/changelog | 9 +++++++++
lib/testdesc.py | 7 +++----
runner/adt-run | 30 +++++++++++++++++++++++++++---
runner/adt-run.1 | 11 +++++++++++
4 files changed, 50 insertions(+), 7 deletions(-)