On Mon, Jan 16, 2017 at 08:50:57AM +0100, Ole Streicher wrote: > Sean Whitton <spwhitton@spwhitton.name> writes: > > I agree with the principle that test failures should be RC by default. > > This is something which seems to have no disagreement here. My concern > is just that I want to have a simple way to override this, to assign > this to a different package etc. I want to have the same flexibility > here as for bugs. A failing test means there's a bug. It might be in the test itself, or in the code being tested. It might be a bug in the test environment. Personally, I'd really rather have unreliable tests fixed. Unreliable tests are like playing Russian roulette: mostly OK but sometimes you get a really loud noise that makes your parents and loved ones be ashamed of you. Picture this: a cocktail party. Many people mingling around, dressed up and engaging in smalltalk, sipping colourful drinks. A new couple arrives and is immediately surrounded by old fiends. "Hi, Jack and Joan, how are you? How is that lovely offspring of yours?" The couple look down, and their faces get a careful, blank expression. "It's not good. We don't know what we did wrong. We're so ashamed. We don't know how such a thing could happen. We thought we were such good parents." A shocked silence fall on the group, in the middle of the hubbub of the greater party. "You see, our child, our child..." Jack sobs and can't get the words out, so Joan takes a deep breath and speaks. "Our child wrote a test that fails randomly, and released it." One by one their friends leave the group, quietly, and without speaking a single harsh syllable. But for months, they had to wait for an invitation to a new party. Apart from social exclusion, unreliable tests waste a lot of time, effort, and mental energy. When a test fails, you have to find out why. What caused the fail? Is it because the test it bad, or because the code it tests is broken? If you let a test fail randomly, you have to debug that test many times. It also kills confidence in the test suite: if all tests pass, is that too also just a random fluke? Can you actually make a release, or should you do some tedious manual testing just to make sure flaky test success didn't cover up a bug somewhere? Until an unreliable test is fixed, in my opinion it'd be better if the test suite didn't fail because of it. Run the test by all means, to gather more information for debugging, but don't fail the whole test suite. -- I want to build worthwhile things that might last. --joeyh
Attachment:
signature.asc
Description: PGP signature