Re: Mass bugs filing: autopkgtest should be marked superficial (new list)
On Mon, 21 Sep 2020 at 09:09:51 -0300, Antonio Terceiro wrote:
> Maybe you could include something like this (the wording can be improved):
>
> Note, however, that such superficial tests are still somewhat useful,
> as they will be considered, for example, to block dependencies from
> breaking your package. In other words, please do not react to this bug
> report by dropping tests from your package completely. More extensive
> testing is of course better, but even superficial tests are better for
> the overal quality of Debian than no tests at all.
Perhaps a more positive way to phrase the bug report would be to lead with
this, and then go on to say why these tests should be marked superficial?
That will hopefully reduce the tendency for maintainers to remove tests in
response.
It's also really useful for the template that is discussed on -devel to
mention the usertags that are going to be used (if any), so that it's
easy to search for them in a machine-readable way. It looks as though
the bugs that have already been filed are usertagged to appear in
<https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=sudipm.mukherjee%40gmail.com&tag=superficialtest>
so let's go with that.
In cases like this where the change is so trivial to make and so similar
across multiple packages, it's probably also useful to link to examples
of a maintainer making this change correctly, like [3] and [4] below.
Maybe something like this (I'm making some assumptions here about what the
release team does and doesn't encourage, so don't actually use this wording
until someone from the RT has acknowledged it):
----8<----
Subject: <package>: autopkgtest must be marked superficial
Severity: important
User: sudipm.mukherjee@gmail.com
Usertags: superficialtest
It has been noticed that the autopkgtest in <package> is running a
trivial command that does not provide significant test coverage:
- <command being run, e.g. foo --help>
These superficial tests are a useful way to detect regressions in
dependencies and prevent them from breaking your package, and the
Release Team encourages their inclusion in packages that do not have
a more thorough test suite available.
However, it is important that we are realistic about the level of
test coverage provided by these commands: most regressions cannot be
detected in this way, so it is not appropriate for packages that only
have superficial tests to migrate to the testing distribution with less
opportunity for testing by users than if these tests had not been present.
To support this, the keyword "Restrictions: superficial" has been
defined [1]. Packages where all tests are marked with this keyword are not
considered for the reduced migration age from unstable to testing, and
will not be allowed to migrate automatically in later stages of the
freeze [2].
Please mark all superficial autopkgtests with this keyword, similar to
[3] or [4].
The Release Team has listed this issue in the list of Release Critical
Issues for bullseye [5] and has mentioned that the test must be marked
superficial if it is not testing one of its own installed binary
packages in some way. As a result, the severity of this bug report might
be increased to serious in future.
[1] https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst#defined-restrictions
[2] https://release.debian.org/bullseye/freeze_policy.html
[3] https://salsa.debian.org/utopia-team/dbus/-/commit/a80908df7d119b181eec5eb0542634a30c2ad468
[4] https://salsa.debian.org/apparmor-team/apparmor/-/commit/580667513a097088ebe579884b38ac8d8666d3b3
[5] https://release.debian.org/bullseye/rc_policy.txt
----8<----
Regards,
smcv
Reply to: