Re: Mass bugs filing: autopkgtest should be marked superficial (new list)
On Mon, Sep 21, 2020 at 2:53 PM Simon McVittie <smcv@debian.org> wrote:
>
> On Mon, 21 Sep 2020 at 09:09:51 -0300, Antonio Terceiro wrote:
> > Maybe you could include something like this (the wording can be improved):
> >
> > Note, however, that such superficial tests are still somewhat useful,
> > as they will be considered, for example, to block dependencies from
> > breaking your package. In other words, please do not react to this bug
> > report by dropping tests from your package completely. More extensive
> > testing is of course better, but even superficial tests are better for
> > the overal quality of Debian than no tests at all.
>
> Perhaps a more positive way to phrase the bug report would be to lead with
> this, and then go on to say why these tests should be marked superficial?
> That will hopefully reduce the tendency for maintainers to remove tests in
> response.
>
> It's also really useful for the template that is discussed on -devel to
> mention the usertags that are going to be used (if any), so that it's
> easy to search for them in a machine-readable way. It looks as though
> the bugs that have already been filed are usertagged to appear in
> <https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=sudipm.mukherjee%40gmail.com&tag=superficialtest>
> so let's go with that.
Those were from the first list and there was another MBF mail to debian-devel.
>
> In cases like this where the change is so trivial to make and so similar
> across multiple packages, it's probably also useful to link to examples
> of a maintainer making this change correctly, like [3] and [4] below.
>
> Maybe something like this (I'm making some assumptions here about what the
> release team does and doesn't encourage, so don't actually use this wording
> until someone from the RT has acknowledged it):
I was drafting something like the following:
*********************************************************************************************
Subject: <package>: autopkgtest must be marked superficial
Dear Maintainer,
Your package has an autopkgtest and its executing the following command:
- <list the command>
Since executing that command is considered to be a trivial test, that
which does not provide significant coverage for a package as a whole.
All trivial tests must be marked with "Restrictions: superficial" as
defined in [1].
A package with a non-trivial autopkgtest will enjoy a reduced
migration age from unstable to testing, and also it will be allowed to
migrate in a later stage of the freeze [2].
A package having a trivial (superficial) test will not get the
migration advantage but it will have the advantage of blocking
dependencies from breaking your package.
Its always better to have more extensive testing than having
superficial testing, which again is better than having no test.
Please consider i) Adding a non-trivial test, and/or ii) Mark the
trivial test with "Restrictions: superficial".
[1]. https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst
[2]. https://release.debian.org/bullseye/freeze_policy.html
*********************************************************************************************
But yours is much better than what I drafted.
--
Regards
Sudip
Reply to: