On Thu, Nov 17, 2005 at 06:43:32PM +0000, Ian Jackson wrote:
> This means execute debian/tests/fred, debian/tests/bill, etc.,
> each with no arguments, expecting exit status 0 and no stderr. The
Having been involved in various unit testing packages I found the above
expectations too much constraining. The first thing you will need after
requiring all tests not to fail is to be able to distinguish: "test that
need to suceed" vs "test that need to fail". Only the misbehaviour of
tests wrt their expected result should be reported as test failures. I
thus propose to add
Following your exit status based approach you could add to stanzas
something like:
Expected-Status: 0
I found the above requirement the very minimum for a test interface.
What follows is optional (IMHO).
I don't see the need/usefulness of requiring no stderr. It seems to me
rather ad-hoc, if people want to test output on file descriptors it can
do it via redirections and textual comparison of files. This bring us to
the next point.
In addition you can think about standardizing a way to compare what a
test prints on file descriptors. Often indeed in unit testing frameworks
you require a function to return a specific value. In the debian/ubuntu
world we can think at having files containing the expected output for a
given file descriptor and enrich stanzas with entries like:
Expected-Output: tests/stdout.txt (0), tests/stderr.txt (2)
Since we are in the semantic-thing era textual comparison is no longer
always the appropriate choice, so it may be a wise to think from the
beginning to a way of specify ad-hoc comparison commands (think for
example at the comparison of XML documents).
I can imagine tons of different ways of specifying them:
- triples: <expected output files, file descriptor, comparison>
- triples: <expected output files, file descriptor, format> + per-format
comparison tools in other stanza fields
- ...
Just choose your favorite one.
My 0.02 EUR.
Cheers.
--
Stefano Zacchiroli -*- Computer Science PhD student @ Uny Bologna, Italy
zack@{cs.unibo.it,debian.org,bononia.it} -%- http://www.bononia.it/zack/
If there's any real truth it's that the entire multidimensional infinity
of the Universe is almost certainly being run by a bunch of maniacs. -!-
Attachment:
signature.asc
Description: Digital signature