Bug#901804: autopkgtest: consider using exit status 8 ("no tests found") if every test was ignored
Control: clone -1 -2 -3
Control: reassign -2 debci
Control: retitle -2 debci: interpret exit status 8 as neutral/"cannot test"
Control: block -1 by -2
Control: reassign -3 release.debian.org
Control: retitle -3 britney: accept neutral/"cannot test" status from debci
Control: block -2 by -3
Summary for the release team (in Bcc): At the moment, autopkgtest
does not distinguish between "at least one test passed and at least
one was skipped" and "every test was skipped". We would like to change
the autopkgtest -> debci -> britney pipeline so that there is a way to
report that every test was skipped, which should cause britney to apply
the same migration behaviour as if there had been no tests at all.
A concrete example is bubblewrap, which I think currently gets the
migration speed benefit of having autopkgtests even though ci.debian.net
has never actually been able to run those tests. I run them locally
before upload, but you shouldn't have to take my word for it :-)
At some point I might add a way to mark some tests as trivial, with
the intention that migration is never sped up by trivial tests passing,
only slowed down by them failing - but that's a future feature.
On Mon, 18 Jun 2018 at 21:14:13 +0200, Paul Gevers wrote:
> On 18-06-18 17:02, Simon McVittie wrote:
> > * exit 0: there is at least one test and all succeeded
> > * overall result: strongly positive
> > * britney interpretation: speed up migration
> >
> > * exit 2: at least one test succeeded, and at least one was ignored
> > * overall result: less strongly positive
> > * britney interpretation: speed up migration (but less so?)
>
> In the end we want migration gated by autopkgtest. I don't think it is
> worth it to introduce more speeds (but that is up to the RT anyways).
Sure, I just wanted to mention it as a possibility. What I mainly want
here is a way to distinguish between "this is good, we have tests and
at least some of them passed, be happy" and "we have tests but they were
all ignored, so behave as though there were none".
> This being used for "all tests skipped" was introduced to fix bug
> 737242.
...
> I think that original bug is equally satisfied with exit code 8, except
> the error message should not be "no tests in this package".
Great, I'll try implementing that in autopkgtest.
> > * exit 4, 6 or 12: at least one test failed and was not ignored
> > * same as now for those exit statuses
> > * debci currently interprets status 8 like this, but doesn't see
> > packages with no tests, so in practice it doesn't happen
>
> Have you checked that there are no packages without a test, but that
> claim they have one?
I have not checked that. If such packages exist, debci would currently
say they failed, which is arguably a debci bug (although I think it's
also fair to say that declaring a Testsuite but no tests is a bug in
the package). They seem likely to end up in the "always failed" state
(although I don't know whether it's debci or britney that tracks the
"always" part). The proposed changes would move them to debci's new
neutral state, which britney would effectively treat the same as "always
failed" afaics.
> Britneys states are
> - PASS = neutral or speed up
> - ALWAYS FAILED = neutral
> - REGRESSION = slow down / block
> - Running = slow down / block
> - Running (always failed) = slow down / block
OK, so britney needs a new neutral state, CANNOT TEST or something,
which would be functionally equivalent to ALWAYS FAILED but with a more
suitable name. It might well have a suitable state internally already,
because it presumably needs to use that state for packages that have no
Testsuite field.
Is there someone who knows the britney codebase and would be willing
to implement that? We can't deploy the autopkgtest version that has my
proposed change to ci.debian.net until debci interprets status 8 as
neutral, and we can't deploy that version of debci until britney accepts
that neutral state from it.
smcv
Reply to: