Re: Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
On Sun, 12 Sep 2021 at 12:48:36 -0700, Felix Lechner wrote:
> On Sun, Sep 12, 2021 at 11:09 AM Simon McVittie <smcv@debian.org> wrote:
> >
> > lintian has recently started emitting warnings for packages that
> > have autopkgtests, but only superficial autopkgtests.
>
> The tag was implemented in response to Bug#932870. [1] It was
> originally suggested on OFTC#debci by dkg (who was copied here). I
> never heard back whether my implementation met the original intent.
> The tag originally had the info visibility, but was later upgraded to
> a warning. [2]
To be clear, I think this tag would have been fine as originally
implemented; it's the subsequent increase of severity from pedantic
to warning that I think is counterproductive.
I think a package with correctly-marked superficial autopkgtests,
like libsdl2-mixer 2.0.4+dfsg1-3, is better than a package with no
autopkgtests at all, like libsdl2-mixer 2.0.4+dfsg1-2. As a result,
to avoid giving maintainers wrong incentives, I think the Lintian tags
emitted for 2.0.4+dfsg1-3 should be of equal or lower severity.
> > the meaning of
> > testsuite-autopkgtest-missing might have changed at some point so that it
> > is only emitted for packages that have debian/tests/control, rather than
> > for all packages that lack autopkgtests?
>
> Maybe a word is missing? The tag 'testsuite-autopkgtest-missing' was
> renamed to 'missing-tests-control' and is issued at the info level
> when sources do NOT ship debian/tests/control (or declare a predefined
> test suite in d/control, usually for teams). [3]
If that's the case, I would have expected it to be emitted for packages
that have absolutely no autopkgtest coverage, such as
gnome-shell-extension-caffeine. In the past, I think I saw
testsuite-autopkgtest-missing emitted for that package, but at the moment,
the replacement tag missing-tests-control is not emitted. Should it be?
The missing-tests-control description now says:
The source package declares the generic <code>Testsuite: autopkgtest</code>
field but provides no <code>debian/tests/control</code> file.
but packages like gnome-shell-extension-caffeine that don't have any
tests at all will not usually have a Testsuite field. If the intention
was for this tag to detect packages with zero test coverage, then it
should also be emitted for packages with no Testsuite.
GNOME shell extensions are probably a convenient example to use when
evaluating "you have no tests" tags, because they are unlikely to
get meaningful test coverage: they're essentially impossible to test
without using something like openQA, which would be a lot of effort
and inconveniently likely to fail as a result of changes that are
not actually bugs, and it isn't obvious how it would fit into the
autopkgtest framework even if someone has the time to implement it. I
think if we get test coverage for desktop environments, it would be more
likely to be a separate, parallel GUI-testing infrastructure alongside
autopkgtest.
smcv
Reply to: