Re: Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
On Sun, 12 Sep 2021 at 13:49:35 -0700, Felix Lechner wrote:
> Either way, the project relies here on the fact
> that having a meaningful testsuite may provide a faster migration from
> unstable to testing.
I think there might be some misunderstanding here. Tests that are marked
with "Restrictions: superficial" cannot make migration faster, even if
they pass: they can only block or delay migration, by failing. That's
the purpose of the superficial keyword: it tells the autopkgtest
infrastructure not to assume that this particular test-case has meaningful
coverage.
If every test in the test suite has a "neutral" result, then there is
no migration bonus. Tests that pass but are marked as superficial are
considered to be "neutral" rather than a success, meaning they do not
speed up migration for a package that does not "deserve" it.
(Other neutral results include tests that fail but are marked as "flaky",
and tests that are skipped because they require isolation or other
functionality that the testbed cannot provide - for example, trying to
run flatpak's test suite inside an lxc container is skipped, because the
security restrictions imposed by lxc prevent flatpak from working.)
Adding tests that *are* superficial, but are not *marked* as superficial,
is a serious problem, because it can cause packages to migrate too
soon (and the release team's RC policy since bullseye says this is a
release-critical bug). However, this is not really feasible for Lintian
to detect, so I think the Lintian maintainers should treat detection of
that class of bug as being out-of-scope.
> On Sun, Sep 12, 2021 at 1:17 PM Simon McVittie <smcv@debian.org> wrote:
> > If that's the case, I would have expected it to be emitted for packages
> > that have absolutely no autopkgtest coverage
>
> You are right! The tag is issued when 'Testsuite: autopkgtest' was
> declared in d/control but no d/tests/control is present. [1] It should
> arguably be an error instead of info. [2]
I think there are three separate tags that would potentially be useful
in this general space, in decreasing order of severity:
1. (Testsuite: autopkgtest in d/control) && (no d/tests/control)
2. absence of autopkgtest coverage, declared correctly
(no Testsuite and no d/tests/control)
3. there is autopkgtest coverage but it is all marked as superficial
I think the first of those is the new missing-tests-control, and I would
agree with making it an error. It's a contradiction between related
source files: d/control says there is a test suite, but (the absence of)
d/tests/control says there is not, and only one of those can be true.
I think the second of those is the old testsuite-autopkgtest-missing,
which apparently no longer exists. Saying that testsuite-autopkgtest-missing
is an old name for the new missing-tests-control seems inaccurate?
The third of those is the new superficial-tests.
> Lintian does not currently complain when no test suite is present. As
> you noted, some sources like GNOME shell extensions are essentially
> impossible to test.
I don't think it makes sense for the new superficial-tests to be considered
worse (= higher severity) than the old testsuite-autopkgtest-missing.
If the old testsuite-autopkgtest-missing is not serious enough to justify
even an info-level lintian tag, then neither is superficial-tests.
Conversely, if superficial-tests is serious enough to justify a lintian
tag, then testsuite-autopkgtest-missing should be a lintian tag of equal
or higher severity, because it's a strictly lower level of test coverage
than superficial-tests.
If you think these should not emit maintainer-facing tags, to avoid
"noise" for the maintainers of packages where autopkgtest coverage is not
feasible to implement, then perhaps they should be classification tags?
smcv
Reply to: