Re: Mass bugs filing: autopkgtest must be marked superficial

[Domenico Andreoli <cavok@debian.org>]

On Sun, Nov 08, 2020 at 10:28:54PM +0000, Sudip Mukherjee wrote:
> Hi All,

Hi,

> This is a new list for the autopkgtest superficial test.
> 
> If the test done in the autopkgtest does not provide significant test
> coverage then it should be marked with "Restrictions: superficial".
> Ref: https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst
> 
> Examples of tests which are not significant includes (its not a complete list):
> 
> 1) Executing the binary to check version
>     Test-Command: foo -v
>     Test-Command: foo -V
>     Test-Command: foo --version
> 
> 2) Executing the binary to check help (foo -h)
>     Test-Command: foo -h
>     Test-Command: foo --help
> 
> 3) A Python or Perl library runs import foo or require Foo; but does
> not attempt to use the library beyond that.
>      Test-Command: python3 -c "import foo"
> 

In dwarves-dfsg's case, a suite to analyzes ELF binaries, I run pahole
on itself. I don't validate the output and certainly do not exaust the
use cases of that or any other tool distributed in the same package.

I consider it a superficial test. Do you agree?

Thanks,
Domenico

-- 
rsa4096: 3B10 0CA1 8674 ACBA B4FE  FCD2 CE5B CF17 9960 DE13
ed25519: FFB4 0CC3 7F2E 091D F7DA  356E CC79 2832 ED38 CB05