[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: [UDD] Is there any information about failed autopkgtest in UDD?

On Thu, 16 Apr 2020 09:56:15 +0100, Simon McVittie wrote:

> > Are all those
> > 
> >    Testsuite: autopkgtest-pkg-* 
> > 
> > superficial?
> Most of them yes, because if they fail, that's Very Bad, but if they
> succeed, they don't give us a whole lot of confidence that the package
> works as intended.
> For example, if you have a package python3-dbus, the most that can be
> done with knowledge of Python but no knowledge of dbus is to "import dbus"
> and see what happens. If that fails, obviously the dbus module is
> unusable (or missing a dependency or something); but if that succeeds,
> that fact doesn't tell us whether it can actually do D-Bus successfully,
> which is its real purpose. If I replaced all the code in python3-dbus
> with print("hello"), obviously it wouldn't be implementing its intended
> API any more, but it would still pass the "import" test. That's why the
> "import" test is considered to be superficial.
> If a family of packages have a convention for how to discover and
> run a real test suite with non-trivial coverage (for example GNOME-style
> installed-tests in /usr/share/installed-tests/**/*.test), then an
> autopkgtest-* helper that detected and used *that* convention would usually
> *not* be superficial.

Very similar in the perl case, and autodep8 shows the restrictions:

% autodep8
Test-Command: /usr/share/pkg-perl-autopkgtest/runner build-deps
Depends: @, @builddeps@, pkg-perl-autopkgtest
Restrictions: skippable
Features: test-name=autodep8-perl-build-deps

Test-Command: /usr/share/pkg-perl-autopkgtest/runner runtime-deps
Depends: @, pkg-perl-autopkgtest
Restrictions: skippable, superficial
Features: test-name=autodep8-perl

Test-Command: /usr/share/pkg-perl-autopkgtest/runner runtime-deps-and-recommends
Depends: @, pkg-perl-autopkgtest
Restrictions: needs-recommends, skippable, superficial
Features: test-name=autodep8-perl-recommends

The first one runs "smoke" which executes the upstream test suite, so
not superficial; the second and third ones run use.t and syntax.t
which try to load the module and do a syntax check, respectively, and
they are supercifial, for the reasons Simon explained for the python
case above.


 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   NP: Peter Ratzenbeck: Bonny Bessie Logan

Attachment: signature.asc
Description: Digital Signature

Reply to: