Hello fellow Debian developers,
First of all, it's great to see that autopkgtests in packages get
adopted so well! We have quite a nice number of tests already, get a
handful of new ones every week, and once we integrate them into
britney we'll get a nice Continuous Integration system out of that. So
thanks to everyone who has added a test already!
But during that I've also seen a big misunderstanding which I see
repeated over and over again (probably from people copy&paste'ing
other people's tests?):
Restrictions: needs-build
debian/tests/upstream:
make check
# or:
python setup.py test
The purpose of an autopkgtest is *not* to run the upstream test suite
against the upstream build tree. That's something which belongs into
the package build and scales much better with the number of buildds
that we have.
The purpose of an autopkgtest is to test *packages*, i. e. whether
your "foo-app" or "python-foo" package has all the necessary files at
the right places, sufficient dependencies, and so on. So instead of
repeating the upstream test suite (which we already know that it
passed [1]), it's much better, and also much faster, to instead have a
"smoke test" which exercises a basic operation of your package using
the commands from $PATH / the installed Python module / the installed
header files and libraries / etc.
E. g. for a Python module you might do something like:
Depends: @
debian/tests/import:
#!/usr/bin/python3
import foo
c = foo.Controller()
assert c.initialized()
For a C library you might do a simple compile/link/run test such as
http://anonscm.debian.org/viewvc/pkg-gnome/desktop/unstable/pango1.0/debian/tests/build?view=markup
For a command line tool you might try and call it with a few
operations in $ADTTMP (e. g. for "git" you could create a repository
and commit a single file into it).
*If* your package's upstream test suite supports running against the
installed project ("make installcheck" or similar), then you are of
course welcome to use that if it makes sense. But otherwise it's often
a lot of work to patch upstream tests to run against the package
build; in that case, please rather just write a simple smoke test.
Thank you for your attention!
Martin
[1] Of course there are still non-zero benefits from that: Some
changes in your build depends could render your packages FTBFS, or due
to a subtle behaviour change break one of your tests. But we currently
don't have the infrastructure to do this kind of "rebuild all reverse
dependencies on each library change" testing, so this just doesn't
scale at the moment.
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
Attachment:
signature.asc
Description: Digital signature