[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Auto reject if autopkgtest of reverse dependencies fail or cause FTBFS

On Mon, Jan 16, 2017 at 08:50:57AM +0100, Ole Streicher wrote:
> Sean Whitton <spwhitton@spwhitton.name> writes:
> > I agree with the principle that test failures should be RC by default.
> This is something which seems to have no disagreement here. My concern
> is just that I want to have a simple way to override this, to assign
> this to a different package etc. I want to have the same flexibility
> here as for bugs.

A failing test means there's a bug. It might be in the test itself, or
in the code being tested. It might be a bug in the test environment.

Personally, I'd really rather have unreliable tests fixed. Unreliable
tests are like playing Russian roulette: mostly OK but sometimes you
get a really loud noise that makes your parents and loved ones be
ashamed of you.

Picture this: a cocktail party. Many people mingling around, dressed
up and engaging in smalltalk, sipping colourful drinks. A new couple
arrives and is immediately surrounded by old fiends. "Hi, Jack and
Joan, how are you? How is that lovely offspring of yours?" The couple
look down, and their faces get a careful, blank expression. "It's not
good. We don't know what we did wrong. We're so ashamed. We don't know
how such a thing could happen. We thought we were such good parents."
A shocked silence fall on the group, in the middle of the hubbub of
the greater party. "You see, our child, our child..." Jack sobs and
can't get the words out, so Joan takes a deep breath and speaks. "Our
child wrote a test that fails randomly, and released it." One by one
their friends leave the group, quietly, and without speaking a single
harsh syllable. But for months, they had to wait for an invitation to
a new party.

Apart from social exclusion, unreliable tests waste a lot of time,
effort, and mental energy. When a test fails, you have to find out
why. What caused the fail? Is it because the test it bad, or because
the code it tests is broken? If you let a test fail randomly, you have
to debug that test many times. It also kills confidence in the test
suite: if all tests pass, is that too also just a random fluke? Can
you actually make a release, or should you do some tedious manual
testing just to make sure flaky test success didn't cover up a bug

Until an unreliable test is fixed, in my opinion it'd be better if the
test suite didn't fail because of it. Run the test by all means, to
gather more information for debugging, but don't fail the whole test

I want to build worthwhile things that might last. --joeyh

Attachment: signature.asc
Description: PGP signature

Reply to: