On Thu, 16 Apr 2020 09:56:15 +0100, Simon McVittie wrote: > > Are all those > > > > Testsuite: autopkgtest-pkg-* > > > > superficial? > > Most of them yes, because if they fail, that's Very Bad, but if they > succeed, they don't give us a whole lot of confidence that the package > works as intended. > > For example, if you have a package python3-dbus, the most that can be > done with knowledge of Python but no knowledge of dbus is to "import dbus" > and see what happens. If that fails, obviously the dbus module is > unusable (or missing a dependency or something); but if that succeeds, > that fact doesn't tell us whether it can actually do D-Bus successfully, > which is its real purpose. If I replaced all the code in python3-dbus > with print("hello"), obviously it wouldn't be implementing its intended > API any more, but it would still pass the "import" test. That's why the > "import" test is considered to be superficial. > > If a family of packages have a convention for how to discover and > run a real test suite with non-trivial coverage (for example GNOME-style > installed-tests in /usr/share/installed-tests/**/*.test), then an > autopkgtest-* helper that detected and used *that* convention would usually > *not* be superficial. Very similar in the perl case, and autodep8 shows the restrictions: % autodep8 Test-Command: /usr/share/pkg-perl-autopkgtest/runner build-deps Depends: @, @builddeps@, pkg-perl-autopkgtest Restrictions: skippable Features: test-name=autodep8-perl-build-deps Test-Command: /usr/share/pkg-perl-autopkgtest/runner runtime-deps Depends: @, pkg-perl-autopkgtest Restrictions: skippable, superficial Features: test-name=autodep8-perl Test-Command: /usr/share/pkg-perl-autopkgtest/runner runtime-deps-and-recommends Depends: @, pkg-perl-autopkgtest Restrictions: needs-recommends, skippable, superficial Features: test-name=autodep8-perl-recommends The first one runs "smoke" which executes the upstream test suite, so not superficial; the second and third ones run use.t and syntax.t which try to load the module and do a syntax check, respectively, and they are supercifial, for the reasons Simon explained for the python case above. Cheers, gregor -- .''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06 `. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe `- NP: Peter Ratzenbeck: Bonny Bessie Logan
Attachment:
signature.asc
Description: Digital Signature