[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Automated testing - design and interfaces

[let's get this over to a technical list like it was supposed to be ;)]

On Thu, Nov 17, 2005 at 10:43:34PM +0100, Stefano Zacchiroli wrote:
> On Thu, Nov 17, 2005 at 06:43:32PM +0000, Ian Jackson wrote:
> >   This means execute debian/tests/fred, debian/tests/bill, etc.,
> >   each with no arguments, expecting exit status 0 and no stderr. The

> Having been involved in various unit testing packages I found the above
> expectations too much constraining.  The first thing you will need after
> requiring all tests not to fail is to be able to distinguish: "test that
> need to suceed" vs "test that need to fail". Only the misbehaviour of
> tests wrt their expected result should be reported as test failures. I
> thus propose to add 

> Following your exit status based approach you could add to stanzas
> something like:

>   Expected-Status: 0

> I found the above requirement the very minimum for a test interface.
> What follows is optional (IMHO).

FWIW, I don't see that there's a clear advantage to having the test harness
*expect* non-zero exit values (or non-empty output as you also suggested).
It may make it easier to write tests by putting more of the logic in the
test harness, but in exchange it makes it harder to debug a test failure
because the debugger has to figure out how "correct" and "incorrect" are
defined for each test, instead of just getting into the meat of seeing why
the test returned non-zero.  Likewise, expecting successful tests to be
silent means that you can rely on any output being error output that can be
used for debugging a test failure.

Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
vorlon@debian.org                                   http://www.debian.org/

Attachment: signature.asc
Description: Digital signature

Reply to: