Bug#851558: autopkgtest: define Restrictions for tests that aren't suitable for gating CI
Package: autopkgtest
Version: 4.3
Severity: wishlist
As mentioned in recent discussion on debian-devel, I don't think all
autopkgtests are sufficiently reliable, stable or important to be used to
gate CI or block testing migration. (In Debian terms: every test failure
is a bug, but not every bug is release-critical.) However, I don't want
to have to disable unimportant or known-unreliable tests altogether, and
lose the ability to see how their results change over time: I would prefer
to keep running them for the logs, and just ignore the side-effects.
Specifically, if a maintainer is unable to fix a particular unreliable
or broken test, or the non-RC bug that it exposes, I think it's still
correct for that test to continue to run on CI infrastructure, with
its result logged but the usual side-effects of that result ignored.
This can give the maintainer valuable feedback to send to upstream
(for example I can now tell ostree's author that
ostree/test-local-pull.sh.test fails around half the time, but the
other pull tests that sporadically failed have stopped failing since
a thread-safety issue in libsoup was fixed).
With the ability to ignore selected restrictions (#850494), this could
be addressed by a restriction, perhaps something like one of these:
Restrictions: experimental
Restrictions: intermittently-fails
Restrictions: may-fail
Restrictions: unimportant
possibly accompanied by:
Features: known-bug=http://bugs.debian.org/1234
Tests with that restriction could perhaps be run normally, but
skipped when determining whether a batch of changes would break testing.
This is similar in concept to the TODO directive in TAP
<https://testanything.org/tap-specification.html>, which marks a
failure as not invalidating the overall success of the test.
S
Reply to: