Mass bugs filing: autopkgtest must be marked superficial
Hi All,
This is a new list for the autopkgtest superficial test.
If the test done in the autopkgtest does not provide significant test
coverage then it should be marked with "Restrictions: superficial".
Ref: https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst
Examples of tests which are not significant includes (its not a complete list):
1) Executing the binary to check version
Test-Command: foo -v
Test-Command: foo -V
Test-Command: foo --version
2) Executing the binary to check help (foo -h)
Test-Command: foo -h
Test-Command: foo --help
3) A Python or Perl library runs import foo or require Foo; but does
not attempt to use the library beyond that.
Test-Command: python3 -c "import foo"
Similar to the last MBF, the intended bug text is:
*************************************************************************
Subject: <package>: autopkgtest must be marked superficial
Severity: important
User: sudipm.mukherjee@gmail.com
Usertags: superficialtest
It has been noticed that the autopkgtest in <package> is running a
trivial command that does not provide significant test coverage:
- <command being run, e.g. foo --help>
Executing that command is considered to be a trivial test, which does
not provide significant coverage for a package as a whole.
But these tests are a useful way to detect regressions in dependencies
and prevent them from breaking your package.
However, it is important that we are realistic about the level of
test coverage provided by these commands: most regressions cannot be
detected in this way. So it is not appropriate for packages with only
superficial tests to have the reduced migration time to migrate from
unstable to testing as that means less opportunity for testing by users
compared to the package with no tests.
To support this, the keyword "Restrictions: superficial" has been
defined [1]. Packages where all tests are marked with this keyword are not
considered for the reduced migration age from unstable to testing, and
will not be allowed to migrate automatically in later stages of the
freeze [2].
Its always better to have more extensive testing than having
superficial testing, which again is better than having no test.
Please consider i) Adding a non-trivial test, and/or ii) Mark the
trivial test with "Restrictions: superficial", similar to
[3] or [4].
The Release Team has listed this issue in the list of Release Critical
Issues for bullseye [5] and has mentioned that the test must be marked
superficial if it is not testing one of its own installed binary
packages in some way. As a result, the severity of this bug report might
be increased to serious in future.
[1] https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst#defined-restrictions
[2] https://release.debian.org/bullseye/freeze_policy.html
[3] https://salsa.debian.org/utopia-team/dbus/-/commit/a80908df7d119b181eec5eb0542634a30c2ad468
[4] https://salsa.debian.org/apparmor-team/apparmor/-/commit/580667513a097088ebe579884b38ac8d8666d3b3
[5] https://release.debian.org/bullseye/rc_policy.txt
*************************************************************************
Attached is the dd-list.
--
Regards
Sudip
Aaron Zauner <azet@azet.org>
lmod
Adrian Vondendriesch <adrian.vondendriesch@credativ.de>
wtf-peewee (U)
Agustin Henze <tin@debian.org>
crrcsim
Alastair McKinstry <mckinstry@debian.org>
cdo
eckit
libdap
pyferret
pygrib
Albert Huang <albert@csail.mit.edu>
pybluez (U)
Alvin Chen <sonoma001@gmail.com>
cgreen
elementpath
Andreas Moog <andreas.moog@warperbbs.de>
nzbget
Andreas Tille <tille@debian.org>
artemis (U)
Barry Warsaw <barry@debian.org>
enum34 (U)
flufl.password (U)
python-progress (U)
python3-openid (U)
Ben Finney <bignose@debian.org>
dput
Carl Chenet <chaica@ohmytux.com>
python-memcache (U)
Carlos Maddela <e7appew@gmail.com>
dvdisaster (U)
Carsten Schoenert <c.schoenert@t-online.de>
python-ilorest
Chris Lamb <lamby@debian.org>
xtrlock (U)
Christoph Berg <myon@debian.org>
libcharon (U)
Christopher Hoskin <mans0954@debian.org>
case (U)
django-memoize (U)
django-setuptest (U)
pycryptodome (U)
Craig Small <csmall@debian.org>
procps
Debian 3-D Printing Packages <3dprinter-general@lists.alioth.debian.org>
libarcus
libcharon
libsavitar
uranium
Debian Astronomy Maintainers <debian-astro-maintainers@lists.alioth.debian.org>
galpy
orbit-predictor
Debian Bluetooth Maintainers <team+pkg-bluetooth@tracker.debian.org>
pybluez
Debian Electronics Team <pkg-electronics-devel@lists.alioth.debian.org>
fpga-icestorm
Debian Med Packaging Team <debian-med-packaging@lists.alioth.debian.org>
artemis
pyode
Debian Middleware Maintainers <pkg-middleware-maintainers@lists.alioth.debian.org>
qpid-proton
Debian Multimedia Maintainers <debian-multimedia@lists.debian.org>
csound
Debian OpenStack <team+openstack@tracker.debian.org>
python-croniter
python-debtcollector
python-memcache
python-openstacksdk
python-osprofiler
python-taskflow
python-tosca-parser
Debian Python Modules Team <python-modules-team@lists.alioth.debian.org>
case
django-memoize
django-setuptest
enum34
flake8-docstrings
flufl.i18n
flufl.password
pycryptodome
pymodbus
pyserial-asyncio
python-box
python-colorama (U)
python-iptables
python-irodsclient
python-progress
python-scripttest
python3-openid
pyvisa-py
readlike
reparser
sortedcollections
sphinxcontrib-asyncio
wtf-peewee
zope.deprecation
Debian Python Modules Team <team+python-modules@tracker.debian.org>
gevent-websocket
Debian Python Team <team+python@tracker.debian.org>
python-exif
python-pyramid
python-socksipy
Debian QA Group <packages@qa.debian.org>
aewm
antigrav
gramofile
iroffer
zope.exceptions
Debian Salt Team <pkg-salt-team@lists.alioth.debian.org>
salt-pepper
Debian Science Maintainers <debian-science-maintainers@lists.alioth.debian.org>
arachne-pnr
python-fluids
python-ulmo
qflow
Debian Security Tools <team+pkg-security@tracker.debian.org>
forensics-extra
Debian Tryton Maintainers <team+tryton-team@tracker.debian.org>
goocalendar
python-zeep
tryton-client
tryton-meta
vatnumber
Diane Trout <diane@ghic.org>
readlike (U)
reparser (U)
sortedcollections (U)
sphinxcontrib-asyncio (U)
Federico Ceratto <federico@debian.org>
django-ranged-response (U)
python-socksipychain (U)
Felipe Sateler <fsateler@debian.org>
csound (U)
Filip Pytloun <filip@pytloun.cz>
salt-pepper (U)
Forrest Cahoon <forrest.cahoon@gmail.com>
csound (U)
FreedomBox packaging team <freedombox-pkg-team@lists.alioth.debian.org>
django-ranged-response
python-socksipychain
Gaetano Guerriero <x.guerriero@tin.it>
eyed3
Ghislain Antony Vaillant <ghisvail@gmail.com>
flake8-docstrings (U)
Giovani Augusto Ferreira <giovani@debian.org>
forensics-extra (U)
Gregor Riepl <onitake@gmail.com>
libarcus (U)
libcharon (U)
libsavitar (U)
uranium (U)
Gürkan Myczko <gurkan@phys.ethz.ch>
python-colorama
IOhannes m zmölnig (Debian/GNU) <umlaeute@debian.org>
python-iptables (U)
James Valleroy <jvalleroy@mailbox.org>
django-ranged-response (U)
Jamie Wilkinson <jaq@debian.org>
pyao
pymad
Joao Eriberto Mota Filho <eriberto@debian.org>
forensics-extra (U)
gimp-texturize
Josue Ortega <josue@debian.org>
consonance
galpy (U)
orbit-predictor (U)
kretcheu <kretcheu@gmail.com>
hcxtools (U)
Kurt Kremitzki <kurt@kwk.systems>
python-fluids (U)
python-ulmo (U)
Marcin Owsiany <porridge@debian.org>
apt-forktracer
Marcio de Souza Oliveira <marciosouza@debian.org>
edb-debugger
Martin <debacle@debian.org>
python-socksipy (U)
Mathias Behrle <mathiasb@m9s.biz>
goocalendar (U)
tryton-client (U)
tryton-meta (U)
vatnumber (U)
Mathias Behrle <mbehrle@debian.org>
python-zeep (U)
Matthew Vernon <matthew@debian.org>
xtrlock
Maximiliano Curia <maxy@debian.org>
pymodbus (U)
Michael Hanke <michael.hanke@gmail.com>
pyode (U)
Michal Arbet <michal.arbet@ultimum.io>
python-box (U)
python-openstacksdk (U)
python-taskflow (U)
Mo Zhou <cdluminate@gmail.com>
termbox
Nilesh Patra <npatra974@gmail.com>
pyode (U)
Olivier Sallou <osallou@debian.org>
python-irodsclient (U)
Ondřej Tůma <mcbig@zeropage.cz>
formiko
Optical Media Tools Team <pkg-opt-media-team@lists.alioth.debian.org>
dvdisaster
Otto Kekäläinen <otto@debian.org>
rdiff-backup (U)
Petter Reinholdtsen <pere@debian.org>
python-socksipychain (U)
Pierre Gruet <pgtdebian@free.fr>
artemis (U)
Pierre-Elliott Bécue <peb@debian.org>
flufl.i18n (U)
Python Applications Packaging Team <python-apps-team@lists.alioth.debian.net>
rdiff-backup
Rogério Brito <rbrito@ime.usp.br>
dvdisaster (U)
Ruben Undheim <ruben.undheim@gmail.com>
arachne-pnr (U)
fpga-icestorm (U)
pyvisa-py (U)
qflow (U)
Sunil Mohan Adapa <sunil@medhas.org>
django-ranged-response (U)
python-socksipychain (U)
SZ Lin (林上智) <szlin@debian.org>
cgreen (U)
elementpath (U)
TANIGUCHI Takaki <takaki@debian.org>
dvdisaster (U)
python-exif (U)
python-pyramid (U)
zope.deprecation (U)
Thomas Goirand <zigo@debian.org>
python-croniter (U)
python-debtcollector (U)
python-memcache (U)
python-openstacksdk (U)
python-osprofiler (U)
python-taskflow (U)
python-tosca-parser (U)
qpid-proton (U)
Thomi Richards <thomi.richards@canonical.com>
python3-openid (U)
Ulises Vitulli <dererk@debian.org>
cbatticon
hcxtools
W. Martin Borgert <debacle@debian.org>
pymodbus (U)
pyserial-asyncio (U)
python-exif (U)
William Grzybowski <william@grzy.org>
gevent-websocket (U)
ytree
Yaroslav Halchenko <debian@onerussian.com>
pyode (U)
Reply to: