Bug#901804: autopkgtest: consider using exit status 8 ("no tests found") if every test was ignored
Package: autopkgtest
Version: 5.3.1
Severity: wishlist
Continuing discussion from
<https://salsa.debian.org/ci-team/autopkgtest/merge_requests/19> because
it has become too involved for a closed merged request:
There is a concern that packages should not be given a testing migration
bonus if they were not actually tested. For example, all of bubblewrap's
tests need isolation-machine, because unprivileged LXC containers don't
give processes enough capabilities to exercise CAP_SYS_ADMIN (even though
bwrap is setuid root so it could normally do that), but afaics it still
gets the faster migration.
I run bubblewrap tests in a VM before each upload, so I don't want to
remove the tests altogether; but Debian shouldn't have to take my word
for it when I say I ran the tests :-)
The introduction of skippable and flaky test Restrictions in git master
potentially exacerbates this, by making it more straightforward to
represent tests that can't always give a result (but at the moment the
state of the art is to make these trivially "pass", so providing more
information can't possibly be a negative).
My proposal was to consider making autopkgtest treat "no test has
succeeded or failed" the same as "there are no tests", so exit 8 if
every test was ignored. I'm using "ignored" here to mean: skipped by
Restrictions; skipped by 'skippable' and exiting 77; skipped by 'flaky'
and exiting nonzero or writing non-ignored stderr; or anything similar
that we might add in future.
That would mean the spec would be:
* exit 0: there is at least one test and all succeeded
* same as now
* overall result: strongly positive
* debci interpretation: pass
* britney interpretation: speed up migration
* exit 2: at least one test succeeded, and at least one was ignored
(skipped by a restriction, skippable and exited 77, or flaky and failed)
* almost the same as now, but require at least one success
* overall result: less strongly positive
* debci interpretation: pass (or a new pass-with-caveats?)
* britney interpretation: speed up migration (but less so?)
* exit 8: no test succeeded or failed (either there were no tests,
or there were tests but they were all ignored)
* same as now if there were no tests
* not the same as now if all tests were ignored
* overall result: neutral
* debci interpretation: a new neutral state (doesn't exist at the moment)
* britney interpretation: default behaviour as though there were no tests
* exit 4, 6 or 12: at least one test failed and was not ignored
* same as now for those exit statuses
* debci currently interprets status 8 like this, but doesn't see
packages with no tests, so in practice it doesn't happen
* overall result: negative (consider delaying or blocking migration)
* debci interpretation: fail
* britney interpretation: delay (or eventually block) migration
* exit with another status: testbed failure or other brokennes
* same as now
* debci interpretation: tmpfail
* britney interpretation: (I don't know what britney does for tmpfail)
This will need debci and possibly britney changes, to add a new neutral
test status and report it to britney, which would interpret it as "don't
expedite migration, but don't delay migration either".
smcv
Reply to: