using autopkgtest for functional upgrade testing
Hey Antonio,
Antonio Terceiro [2014-06-27 21:17 -0300]:
> I have been experimenting with using autopkgtest to run functional tests on
> upgraded systems. This type of automated tests consist of using a testbed on a
> previous release (currently wheezy), then upgrading the system to the next
> release (currently jessie) after installing the test dependencies, but before
> running the tests.
That sounds interesting! So far we've been doing upgrade testing on a
global scale, not for each package individually. This is mostly
because a lot of upgrade files are not visible for individual
packages, but "emergent" from the combination of packages (package and
file conflicts, apt not being able to calculate an upgrade path,
packages mis-treating config.d/ snippets from other packages and the
like). My gut feeling is that this approach would catch more issues
while having less overhead by having to do smaller upgrades thousands
of times:
* Install the original system, with various scenarios (minimal,
default install, GNOME desktop, KDE desktop, LAMP server, etc.)
* run dist-upgrade, record conffile prompts, orphaned packages and
conffiles, etc.
* if successful, snapshot the resulting upgraded testbed, and run
all autopkgtests of the installed packages on that
But nevertheless this is mostly for distro-wide QA, and from the POV
of a package maintainer it is interesting indeed to test upgrades of
her package(s) in isolation. So I'm happy to teach autopkgtest about
this, I just wonder if we can and should actually run all of them
regularly on our QA machines (that quickly becomes a hw bottleneck).
> $ adt-run \
> --user debci \
> --tests-dir debian/upgrade-tests/wheezy \
> --upgrade-to unstable \
> --built-tree . \
> --- schroot debci-wheezy+backports-amd64
I'm still not clear why you need a separate set of tests after
upgrading and thust the --tests-dir concept, instead of merely running
the package's already existing autopkgtest in debian/tests/control
after the upgrade? Sure, a few tests might need to be adjusted to
check the release they are running on, but that seems easier than
having to maintain another parallel set of tests?
> I would like input on a few points:
Let's discuss them after settling the question above; TBH I'm not
currently convinced that we even need all this; As far as I can see,
adding --upgrade-to like functionality (your second patch) and running
the existing tests ought to suffice?
Thanks for working on this!
Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/autopkgtest-devel/attachments/20140628/6c6fdfbf/attachment.sig>
Reply to: