Re: Mass bugs filing: autopkgtest should be marked superficial (new list)
On Tue, Sep 22, 2020 at 9:02 AM Holger Levsen <holger@layer-acht.org> wrote:
>
> On Sat, Sep 19, 2020 at 08:39:44PM +0100, Sudip Mukherjee wrote:
> > After discussing with few people, I now intend to file them with
> > "severity: important" and I will also reduce the severity of the
> > previously open similar bugs to 'important'.
>
> thank you, for all your work on this! (which includes these discussions! :)
Thanks to everyone for helping me with the bug text.
And, the final version (unless someone suggests some change):
*************************************************************************
Subject: <package>: autopkgtest must be marked superficial
Severity: important
User: sudipm.mukherjee@gmail.com
Usertags: superficialtest
It has been noticed that the autopkgtest in <package> is running a
trivial command that does not provide significant test coverage:
- <command being run, e.g. foo --help>
Executing that command is considered to be a trivial test, that
which does not provide significant coverage for a package as a whole.
But these tests are a useful way to detect regressions in dependencies
and prevent them from breaking your package.
However, it is important that we are realistic about the level of
test coverage provided by these commands: most regressions cannot be
detected in this way. So it is not appropriate for packages with only
superficial tests to have the reduced migration time to migrate from
unstable to testing as that means less
opportunity for testing by users compared to the package with no tests.
To support this, the keyword "Restrictions: superficial" has been
defined [1]. Packages where all tests are marked with this keyword are not
considered for the reduced migration age from unstable to testing, and
will not be allowed to migrate automatically in later stages of the
freeze [2].
Its always better to have more extensive testing than having
superficial testing, which again is better than having no test.
Please consider i) Adding a non-trivial test, and/or ii) Mark the
trivial test with "Restrictions: superficial", similar to
[3] or [4].
The Release Team has listed this issue in the list of Release Critical
Issues for bullseye [5] and has mentioned that the test must be marked
superficial if it is not testing one of its own installed binary
packages in some way. As a result, the severity of this bug report might
be increased to serious in future.
[1] https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst#defined-restrictions
[2] https://release.debian.org/bullseye/freeze_policy.html
[3] https://salsa.debian.org/utopia-team/dbus/-/commit/a80908df7d119b181eec5eb0542634a30c2ad468
[4] https://salsa.debian.org/apparmor-team/apparmor/-/commit/580667513a097088ebe579884b38ac8d8666d3b3
[5] https://release.debian.org/bullseye/rc_policy.txt
*************************************************************************
If the above sounds good, I will continue with this.
--
Regards
Sudip
Reply to: