Re: Auto reject if autopkgtest of reverse dependencies fail or cause FTBFS
On Mon, Jan 16, 2017 at 05:45:19PM -0800, Russ Allbery wrote:
> > Well, maybe what it's excessively aggressive or questionable is to run
> > the tests at build time and making the package build as a whole to fail
> > when any test fails.
> I'm quite surprised that you would advocate not failing a build if tests
> fail during the package build? I think that would be an awful way to
> proceed. My packages have test suites for a reason. I do not want
> packages to appear to successfully build if their tests are failing. That
> may mean that the resulting binaries are nonfunctional or even dangerous.
Not exactly. I'm not advocating not failing a build if tests fail
as a general rule.
In this context, I refer specifically to flaky tests. What I call
questionable is keeping a flaky test making the build to fail when the
test fails so much that it's clearly a wrongly designed test.
Or, alternatively, if the test fail a lot and it's correctly designed,
it is questionable not to consider the bug as RC.