Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI
Control: forwarded -1 https://salsa.debian.org/ci-team/autopkgtest/merge_requests/19
On Mon, 16 Jan 2017 at 12:34:50 +0100, Martin Pitt wrote:
> Simon McVittie [2017-01-16 9:06 +0000]:
> > Specifically, if a maintainer is unable to fix a particular unreliable
> > or broken test, or the non-RC bug that it exposes, I think it's still
> > correct for that test to continue to run on CI infrastructure, with
> > its result logged but the usual side-effects of that result ignored.
I've implemented this (provisionally named "flaky", but other names are
available for bikeshedding).
If the flaky test passes, it is recorded as PASS as usual, and
autopkgtest's exit status is the same as if a normal test had passed.
If it fails, it is recorded as FLAKY, and autopkgtest's exit status is
the same as if it had been skipped due to Restrictions: isolation-machine
or similar.
smcv
Reply to: