[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#901804: autopkgtest: consider using exit status 8 ("no tests found") if every test was ignored

Package: autopkgtest
Version: 5.3.1
Severity: wishlist

Continuing discussion from
<https://salsa.debian.org/ci-team/autopkgtest/merge_requests/19> because
it has become too involved for a closed merged request:

There is a concern that packages should not be given a testing migration
bonus if they were not actually tested. For example, all of bubblewrap's
tests need isolation-machine, because unprivileged LXC containers don't
give processes enough capabilities to exercise CAP_SYS_ADMIN (even though
bwrap is setuid root so it could normally do that), but afaics it still
gets the faster migration.

I run bubblewrap tests in a VM before each upload, so I don't want to
remove the tests altogether; but Debian shouldn't have to take my word
for it when I say I ran the tests :-)

The introduction of skippable and flaky test Restrictions in git master
potentially exacerbates this, by making it more straightforward to
represent tests that can't always give a result (but at the moment the
state of the art is to make these trivially "pass", so providing more
information can't possibly be a negative).

My proposal was to consider making autopkgtest treat "no test has
succeeded or failed" the same as "there are no tests", so exit 8 if
every test was ignored. I'm using "ignored" here to mean: skipped by
Restrictions; skipped by 'skippable' and exiting 77; skipped by 'flaky'
and exiting nonzero or writing non-ignored stderr; or anything similar
that we might add in future.

That would mean the spec would be:

* exit 0: there is at least one test and all succeeded
    * same as now
    * overall result: strongly positive
    * debci interpretation: pass
    * britney interpretation: speed up migration

* exit 2: at least one test succeeded, and at least one was ignored
  (skipped by a restriction, skippable and exited 77, or flaky and failed)
    * almost the same as now, but require at least one success
    * overall result: less strongly positive
    * debci interpretation: pass (or a new pass-with-caveats?)
    * britney interpretation: speed up migration (but less so?)

* exit 8: no test succeeded or failed (either there were no tests,
  or there were tests but they were all ignored)
    * same as now if there were no tests
    * not the same as now if all tests were ignored
    * overall result: neutral
    * debci interpretation: a new neutral state (doesn't exist at the moment)
    * britney interpretation: default behaviour as though there were no tests

* exit 4, 6 or 12: at least one test failed and was not ignored
    * same as now for those exit statuses
    * debci currently interprets status 8 like this, but doesn't see
      packages with no tests, so in practice it doesn't happen
    * overall result: negative (consider delaying or blocking migration)
    * debci interpretation: fail
    * britney interpretation: delay (or eventually block) migration

* exit with another status: testbed failure or other brokennes
    * same as now
    * debci interpretation: tmpfail
    * britney interpretation: (I don't know what britney does for tmpfail)

This will need debci and possibly britney changes, to add a new neutral
test status and report it to britney, which would interpret it as "don't
expedite migration, but don't delay migration either".


Reply to: