On Thu, 16 Apr 2020 09:56:15 +0100, Simon McVittie wrote:
> > Are all those
> >
> > Testsuite: autopkgtest-pkg-*
> >
> > superficial?
>
> Most of them yes, because if they fail, that's Very Bad, but if they
> succeed, they don't give us a whole lot of confidence that the package
> works as intended.
>
> For example, if you have a package python3-dbus, the most that can be
> done with knowledge of Python but no knowledge of dbus is to "import dbus"
> and see what happens. If that fails, obviously the dbus module is
> unusable (or missing a dependency or something); but if that succeeds,
> that fact doesn't tell us whether it can actually do D-Bus successfully,
> which is its real purpose. If I replaced all the code in python3-dbus
> with print("hello"), obviously it wouldn't be implementing its intended
> API any more, but it would still pass the "import" test. That's why the
> "import" test is considered to be superficial.
>
> If a family of packages have a convention for how to discover and
> run a real test suite with non-trivial coverage (for example GNOME-style
> installed-tests in /usr/share/installed-tests/**/*.test), then an
> autopkgtest-* helper that detected and used *that* convention would usually
> *not* be superficial.
Very similar in the perl case, and autodep8 shows the restrictions:
% autodep8
Test-Command: /usr/share/pkg-perl-autopkgtest/runner build-deps
Depends: @, @builddeps@, pkg-perl-autopkgtest
Restrictions: skippable
Features: test-name=autodep8-perl-build-deps
Test-Command: /usr/share/pkg-perl-autopkgtest/runner runtime-deps
Depends: @, pkg-perl-autopkgtest
Restrictions: skippable, superficial
Features: test-name=autodep8-perl
Test-Command: /usr/share/pkg-perl-autopkgtest/runner runtime-deps-and-recommends
Depends: @, pkg-perl-autopkgtest
Restrictions: needs-recommends, skippable, superficial
Features: test-name=autodep8-perl-recommends
The first one runs "smoke" which executes the upstream test suite, so
not superficial; the second and third ones run use.t and syntax.t
which try to load the module and do a syntax check, respectively, and
they are supercifial, for the reasons Simon explained for the python
case above.
Cheers,
gregor
--
.''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org
: :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06
`. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
`- NP: Peter Ratzenbeck: Bonny Bessie Logan
Attachment:
signature.asc
Description: Digital Signature