Re: Automated testing - design and interfaces

[Ian Jackson <ian@davenant.greenend.org.uk>]

Stefano Zacchiroli writes ("Re: Automated testing - design and interfaces"):
> On Fri, Nov 18, 2005 at 12:23:49PM +0000, Ian Jackson wrote:
> > The interface should be as simple as we can make it while still being
> > able to do the job.  Remember that in it should be possible, and not
> > wholly impractical, to reimplement the test runner.
> 
> Fair enough, le me just add a few observations:
> 1) if you want the testing framework to be used and widespread (as I
>    hope you want to) the test-runner script infrastructure must exists.
>    So please consider developing (and describing) it as well as part of
>    your efforts of involve someone interested in it

Certainly we will develop this.  But we don't need to agree so firmly
on its published interface, because if you don't like the one I come
up with then you can always wrap it or patch it or invent your own.

The interface that the test-runner expects has to be common to all
packages, so we need to get it right.

> 2) if the above mentioned interface is the one responsible for
>    specifying the actual test expectations then testing a single aspect
>    of the test (namely it exit status to be 0) is enough, no need to
>    test that stderr is empty

Requiring that stderr be empty makes it harder to perpetrate
lost-exit-status bugs, and also means that the test runner can
unconditionally copy tests' stderr to its own stderr if it wants to.

> 3) under the same assumptions of the previous point other aspects of
>    your stanzas may be simplified: tests-directory for example can be
>    removed and delegated to the test-runner script infrastructure. Even
>    "Tests:" can be seen as useless, just standardize a name for the only
>    script that will be executed.

We want a standard way of enumerating the tests.  That will let people
and infrastructure have lists of tests that are allowed to fail, which
are separate from the package.

The Tests-Directory is needed to avoid the control file (which will
want to be handwritten) from becoming repetitive.

Ian.