Re: Auto reject if autopkgtest of reverse dependencies fail or cause FTBFS
Steve Langasek writes ("Re: Auto reject if autopkgtest of reverse dependencies fail or cause FTBFS"):
> If the failure of the test is not critical, then it should not be used as a
> gate for CI. Which means you, as the package maintainer who knows that this
> test failure is not critical, should fix your autopkgtest to not fail when
> the non-critical test case fails.
You seem to be suggesting that in the case of
* tests which expose non-RC bugs in the main code or its dependencies
* broken tests, but not the most important test cases
the test should be suppressed: ie, that it should not be run (or, it
should be nobbled to exit "successfully" despite actually failing).
I disagree. Information from such tests is useful and should be
properly recorded and handled.
I also disagree with the proposition that the information, that test X
in package Y is caused by a non-RC bug and should not impede
migration, should be recorded inside the source package Y.
We need to be able to adjust the blockness of tests without uploading
new versions of packages. It should be in the bug system.
> Quite to the contrary of the claims in this thread that gating on
> autopkgtests will create a bottleneck in the release team for overriding
> test failures, this will have the effect of holding maintainers accountable
> for the state of their autopkgtest results. CI tests are only useful if you
> have a known good baseline. If your tests are flaky, or otherwise produce
> failures that you think don't matter, then those test results are not useful
> than anyone but yourself. Please help us make the autopkgtests useful for
> the whole project.
CI tests are useful for purposes other than controlling testing
migration.
> The result of the autopkgtest should be whatever you as the maintainer think
> is the appropriate level for gating. Frankly, I think it's sophistry to
> argue both that you care about seeing the results of the tests, and that you
> don't want a failure of those tests to gate because they only apply to
> "special cases".
This is IMO a silly argument. We always release Debian with bugs.
CI failures that represent non-RC bugs are useful information. Such
failures should be brought to the attention of a human so that the
human can decide whether te failure is RC (or take other appropriate
action).
You are getting dangerously close to the notion that in a
well-functioning organisation the test suite will nearly always
completely pass.
> Why would you mark them non-blocking /before/ you know that the tests are
> flaky enough for this to matter? Upstream put them in the test suite for a
> reason. I'd suggest that it's much better to block by default, and if you
> find that a particular test is becoming a problem (for you or for another
> maintainer), you can upload to make that test non-blocking.
I don't think anyone is arguing the reverse.
The question is whether marking a test non-blocking should involve the
release team. I think it should not. It should involve the package
maintainer (unless there is disagreement).
We want to incentivise people to provide tests. If they cannot
control what action is taken (by automation) in response to the tests,
they will remove or disable (or not provide) tests.
Ian.
--
Ian Jackson <ijackson@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Reply to: