Bug#1032900: cjs: autopkgtest should be marked as superficial
Source: cjs
Version: 5.6.0-1
Severity: important
Justification: https://release.debian.org/testing/rc_policy.txt 6a
User: debian-ci@lists.debian.org
Usertags: superficial
X-Debbugs-Cc: debian-ci@lists.debian.org
While looking at cjs' test coverage in relation to discussion of mozjs78's
test coverage, I happened to notice that cjs currently has two autopkgtests
enabled:
- "build" is a -dev smoke-test similar to the ones I've been adding to
various packages, which checks that the -dev package isn't completely
broken;
- "acc" seems to be doing some sort of ABI validation with
abi-compliance-checker
These are great to have and I would encourage the maintainers to keep
them, but they don't provide significant test coverage for cjs' C code,
and most functional regressions in cjs or its dependencies wouldn't be
detected by this sort of test.
Please mark these tests with the "superficial" restriction documented
in [0], similar to [1] and [2], so that these tests don't give cjs the
faster migration time that would be used for test suites with significant
coverage.
The Release Team has included this class of issue in their list of
release-critical issues since bullseye[3][4], and has mentioned that the
test must be marked superficial if it is not testing one of the installed
binary packages in some way. As a result, the severity of this bug report
might be increased to serious in future.
If cjs later gains an autopkgtest that *does* have significant test
coverage, for example an equivalent of [5] in gjs (which runs upstream's
"as-installed" test suite as packaged in the gjs-tests), then that one
does not need to be marked as superficial. cjs does have an equivalent
of that test, but it's commented out in debian/tests/control because
there is currently no cjs-tests package.
Please note that [6] (currently also commented out because it fails)
is not considered to be a valid autopkgtest by the release team or by
the autopkgtest spec[0], because it tests a newly-rebuilt copy of cjs,
and not the cjs binaries installed by apt:
note that the tests must test the *installed* version of the package,
as opposed to programs or any other file from the built tree
— [0]
must test at least one of its own *installed* binary packages
— [3], [4]; my emphasis
so upstream's test suite can only be used as an autopkgtest if it is
possible to adapt it to do "as-installed" testing of binaries from apt,
for example via GNOME-style installed-tests[7].
smcv
[0] https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst
[1] https://salsa.debian.org/utopia-team/dbus/-/commit/a80908df7d119b181eec5eb0542634a30c2ad468
[2] https://salsa.debian.org/apparmor-team/apparmor/-/commit/580667513a097088ebe579884b38ac8d8666d3b3,
[3] https://release.debian.org/bullseye/rc_policy.txt
[4] https://release.debian.org/bookworm/rc_policy.txt
[5] https://sources.debian.org/src/gjs/1.74.2-1/debian/tests/installed-tests/
[6] https://sources.debian.org/src/cjs/5.6.0-1/debian/tests/testsuite/
[7] https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests
Reply to: