Re: Automated testing - design and interfaces

[Ian Jackson <ian@davenant.greenend.org.uk>]

(Note: sorry about my earlier header mixup.  This thread is on the
wrong list so I'm crossposting this reply to -project and -policy and
have set Reply-To to point to -policy.  I will also quote more of
Stefano's message than would usually be sensible, to give readers in
-policy an easier time.)

Stefano Zacchiroli writes ("Re: Automated testing - design and interfaces"):
> On Thu, Nov 17, 2005 at 06:43:32PM +0000, Ian Jackson wrote:
> >   This means execute debian/tests/fred, debian/tests/bill, etc.,
> >   each with no arguments, expecting exit status 0 and no stderr. The
> 
> Having been involved in various unit testing packages I found the above
> expectations too much constraining.  The first thing you will need after
> requiring all tests not to fail is to be able to distinguish: "test that
> need to suceed" vs "test that need to fail". Only the misbehaviour of
> tests wrt their expected result should be reported as test failures. I
> thus propose to add 
> 
> Following your exit status based approach you could add to stanzas
> something like:
> 
>   Expected-Status: 0

The need for this can be avoided by wrapping the actual test up with
some test-runner script.

I didn't want to make the _interface_ to the tests the kind of rich
interface a test suite framework has, with arrangements for specifying
expected behaviour, matching the output of programs against regexps,
etc.

Instead, if a package needs those facilities it then the test stanza
would declare a dependency on a package which would provide them.  For
convenience, the source package with the test-runner which interprets
these files would probably also produce a .deb with a few helpful
programs in it, but in general I think this problem is not part of the
_interface_.

The interface should be as simple as we can make it while still being
able to do the job.  Remember that in it should be possible, and not
wholly impractical, to reimplement the test runner.

> I can imagine tons of different ways of specifying [expected output]

Exactly :-).

Ian.