Re: Auto reject if autopkgtest of reverse dependencies fail or cause FTBFS

[Santiago Vila <sanvila@unex.es>]

On Mon, Jan 16, 2017 at 05:45:19PM -0800, Russ Allbery wrote:

> > Well, maybe what it's excessively aggressive or questionable is to run
> > the tests at build time and making the package build as a whole to fail
> > when any test fails.
> 
> *blink*.
> 
> I'm quite surprised that you would advocate not failing a build if tests
> fail during the package build?  I think that would be an awful way to
> proceed.  My packages have test suites for a reason.  I do not want
> packages to appear to successfully build if their tests are failing.  That
> may mean that the resulting binaries are nonfunctional or even dangerous.

Not exactly. I'm not advocating not failing a build if tests fail
as a general rule.

In this context, I refer specifically to flaky tests. What I call
questionable is keeping a flaky test making the build to fail when the
test fails so much that it's clearly a wrongly designed test.

Or, alternatively, if the test fail a lot and it's correctly designed,
it is questionable not to consider the bug as RC.

Thanks.