[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#901847: marked as done ([britney] accept debci/autopkgtest neutral state)

Your message dated Mon, 23 Jul 2018 14:32:53 +0200
with message-id <8274eff8-f238-9b71-ddd4-1f89c38640f8@debian.org>
and subject line Re: Pending 901847 [britney] accept debci/autopkgtest neutral state
has caused the Debian Bug report #901847,
regarding [britney] accept debci/autopkgtest neutral state
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
901847: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=901847
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems
--- Begin Message --- 
Package: autopkgtest
Version: 5.3.1
Severity: wishlist

Continuing discussion from
<https://salsa.debian.org/ci-team/autopkgtest/merge_requests/19> because
it has become too involved for a closed merged request:

There is a concern that packages should not be given a testing migration
bonus if they were not actually tested. For example, all of bubblewrap's
tests need isolation-machine, because unprivileged LXC containers don't
give processes enough capabilities to exercise CAP_SYS_ADMIN (even though
bwrap is setuid root so it could normally do that), but afaics it still
gets the faster migration.

I run bubblewrap tests in a VM before each upload, so I don't want to
remove the tests altogether; but Debian shouldn't have to take my word
for it when I say I ran the tests :-)

The introduction of skippable and flaky test Restrictions in git master
potentially exacerbates this, by making it more straightforward to
represent tests that can't always give a result (but at the moment the
state of the art is to make these trivially "pass", so providing more
information can't possibly be a negative).

My proposal was to consider making autopkgtest treat "no test has
succeeded or failed" the same as "there are no tests", so exit 8 if
every test was ignored. I'm using "ignored" here to mean: skipped by
Restrictions; skipped by 'skippable' and exiting 77; skipped by 'flaky'
and exiting nonzero or writing non-ignored stderr; or anything similar
that we might add in future.

That would mean the spec would be:

* exit 0: there is at least one test and all succeeded
    * same as now
    * overall result: strongly positive
    * debci interpretation: pass
    * britney interpretation: speed up migration

* exit 2: at least one test succeeded, and at least one was ignored
  (skipped by a restriction, skippable and exited 77, or flaky and failed)
    * almost the same as now, but require at least one success
    * overall result: less strongly positive
    * debci interpretation: pass (or a new pass-with-caveats?)
    * britney interpretation: speed up migration (but less so?)

* exit 8: no test succeeded or failed (either there were no tests,
  or there were tests but they were all ignored)
    * same as now if there were no tests
    * not the same as now if all tests were ignored
    * overall result: neutral
    * debci interpretation: a new neutral state (doesn't exist at the moment)
    * britney interpretation: default behaviour as though there were no tests

* exit 4, 6 or 12: at least one test failed and was not ignored
    * same as now for those exit statuses
    * debci currently interprets status 8 like this, but doesn't see
      packages with no tests, so in practice it doesn't happen
    * overall result: negative (consider delaying or blocking migration)
    * debci interpretation: fail
    * britney interpretation: delay (or eventually block) migration

* exit with another status: testbed failure or other brokennes
    * same as now
    * debci interpretation: tmpfail
    * britney interpretation: (I don't know what britney does for tmpfail)

This will need debci and possibly britney changes, to add a new neutral
test status and report it to britney, which would interpret it as "don't
expedite migration, but don't delay migration either".

    smcv

--- End Message ---
--- Begin Message --- 
On Sat, 21 Jul 2018 14:52:19 +0200 Paul Gevers <elbrus@debian.org> wrote:
> This has been merged in git.
> 
> https://salsa.debian.org/release-team/britney2/commit/624b185ba60709f1686fbaa2a7623a94c5cb23ef

And deployed. Thanks Niels.

Paul

Attachment: signature.asc
Description: OpenPGP digital signature


--- End Message ---
Reply to: