Re: continue on a failing test (dh7)
On Wed, Feb 12, 2014 at 09:20:06AM +0800, Paul Wise wrote:
> Please don't do that , tests are there for a reason. It is better to
> fix the problem or fix the test than ignore the test.
To give an example of why this is important.
procps had a problem where it would fail on certain buildds. At first
I cursed the buildds, cursed the architectures and their flakyness.
Nothing I could do would reproduce it, but some buildds would, at
depressingly semi-regular times, complain.
Eventually the problem was the test was working but made assumptions
about the system. These are valid assumptions for a normal setup (you
wouldn't run it like this) however the program shouldn't of crashed
but complained or exited nicely.
I was *that* close to having a rule in the test setup that basically
said "if the system is in this state, don't run test". This would of
masked a real bug in the program; which admittedly not many people will
ever see, but it shouldn't be there in the first place.
The bug is fixed now, the test will need some adjusting to cater for the
error message, but that's the correct way it should respond.
- Craig
--
Craig Small (@smallsees) http://enc.com.au/ csmall at : enc.com.au
Debian GNU/Linux http://www.debian.org/ csmall at : debian.org
GPG fingerprint: 5D2F B320 B825 D939 04D2 0519 3938 F96B DF50 FEA5
Reply to: