Re: RFS: mwic 0.7.4-1
On Tue, 20 Mar 2018 at 12:18:39 +0100, Gregor Riepl wrote:
> > In case I've misunderstood you, and you're referring to unit tests
> > shipped debian/tests/*, than yes, I agree. :)
>
> As far as I understand, these tests are executed by the package builder after
> the upstream build script has finished. They're meant as a sort of integration
> test, i.e. "does this package run on Debian".
>
> There's even a Lintian tag for them:
> https://lintian.debian.org/tags/testsuite-autopkgtest-missing.html (which, I
> think, is a bit overzealous)
I think you're slightly confusing build-time tests with as-installed
autopkgtests, but you have the right idea.
Build-time tests (dh_auto_test or similar) run the upstream "make check"
or equivalent, in the same environment as the build itself (typically an
autobuilder). For simple libraries, build-time tests are enough. However,
for more complex packages, the autobuilder environment is too artificial
or too restricted for build-time tests to give full coverage:
- can't run tests as root (system-level packages like dbus often need this
for good coverage)
- can't contact the Internet (even if in practice you usually can, Policy
says you must not, for good reasons)
- can't rely on a reasonable/realistic environment, like system services
running, being in a systemd-logind login session on systems that have
it, or even having a home directory
- can't rely on the packages being "properly installed" so that
hard-coded paths can work, and have to rely on overrides like PYTHONPATH
to make newly-built code visible
autopkgtest (debian/tests/) is a form of as-installed testing, which takes
the packages that were built, installs them in a relatively complete and
realistic environment (typically a lxc container or a qemu/kvm virtual
machine) and runs further tests there. Sometimes these tests just repeat
the build-time test coverage, but often they can make use of the ability
to do things that wouldn't work at build-time, like contacting Internet
services, running things as root, or relying on system services. This
often gives them better test coverage.
https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests is an example
of an upstream project that is doing similar things.
Because the autopkgtest container or VM is thrown away after running one
package's tests in it, the tests can do things that would be unacceptable
in an autobuilder environment, which again increases coverage.
In Debian, autopkgtests are run separately, by different infrastructure
(ci.debian.net), long after the package was built. The same package
will usually be tested multiple times against newer versions of its
dependencies, to look for regressions caused by a dependency change.
The normal package upload/autobuild workflow does not typically run
autopkgtests, although it could. Vectis, which is a personal package
building tool that I'm working on, builds the package in sbuild in
a virtual machine then immediately puts it through autopkgtests and
piuparts testing in separate VMs, so that test failures can be checked
(and either used as a reason not to upload, or ignored, based on the
maintainer's judgement about the severity of the failure and the urgency
of the upload - the same way the maintainer would triage Lintian errors).
smcv
Reply to: