[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Automated testing - design and interfaces

On Wed, 2005-11-23 at 18:16 +1000, Anthony Towns wrote:
> On Mon, Nov 21, 2005 at 06:22:37PM +0000, Ian Jackson wrote:
> > > Note that it's often better to have a single script run many tests, so
> > > you probably want to allow tests to pass back some summary information,
> > > or include the last ten lines of its output or similar. Something like:
> > >   foo FAIL:
> > >     FAILURE: testcase 231
> > >     FAILURE: testcase 289
> > >     FAILURE: testcase 314
> > >     3/512 test cases failed
> > This is no good because we want the test environment to be able to
> > tell which tests failed, so the test cases have to be enumerated in
> > the test metadata file.

Replying to two messages here ...
I don't think we have to enumerate the tests in advance. Sure the test
runner needs to be able to identify and [possibly] categorise the tests,
but explicit enumeration is quite orthogonal. A number of python
unittest runners will scan directories and classes for their tests, and
the report from users is consistently that this is easier to use.

> Uh, having to have 1000 scripts, or even just 1000 entries in a metadata
> file, just to run 1000 tests is a showstopper imho. Heck, identifying
> testcases by number mightn't be particularly desirable in some instances,
> if a clearer alternative like, say, "test case failed: add 1, add 2,
> del 1, ch 2" is possible.

A very nice feature in the xUnit world is that tests can be identified
by either their path (inside the language namespace) or by a comment
written by the author. At runtime you can choose which to see. I dont
think we need the ability for runtime selection, but having a heuristic
that works and is overridable would be nice.

I.e. by default you might get
tests/layout/documentation_in_usr_share_doc.sh: PASS

But inside that test you could say:
test_name Documentation is installed in /usr/share/doc

and the output becomes
Documentation is installed in /usr/share/doc: PASS

I've written a project that is somewhat related to this:

Its a python implementation of a cross process test running protocol.
This lets a sub process run 0 to many tests, identify them and provide
pass/fail/error status and traceback or other diagnostics. As the driver
is python its not appropriate here, but I think the basic
protocol/concept might be useful.

> > You can't check that the binary works _when the .deb is installed_
> > without installing it.
> That's okay. You can't check that the binary works _on the user's system_
> without installing it on the user's system either. For Debian's purposes,
> being able to run the tests with minimal setup seems crucial.


> It would probably be quite useful to be able to write tests like:
> 	for mta in exim4 smail sendmail qmail; do
> 	   install $mta
> 	   # actual test
> 	   uninstall $mta
> 	done
> too, to ensure that packages that depend on one of a number of packages
> actually work with all of them.

Yes, that would be excellent.


GPG key available at: <http://www.robertcollins.net/keys.txt>.

Attachment: signature.asc
Description: This is a digitally signed message part

Reply to: