[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Automated testing - design and interfaces

On Mon, Nov 21, 2005 at 06:22:37PM +0000, Ian Jackson wrote:
> > Note that it's often better to have a single script run many tests, so
> > you probably want to allow tests to pass back some summary information,
> > or include the last ten lines of its output or similar. Something like:
> >   foo FAIL:
> >     FAILURE: testcase 231
> >     FAILURE: testcase 289
> >     FAILURE: testcase 314
> >     3/512 test cases failed
> This is no good because we want the test environment to be able to
> tell which tests failed, so the test cases have to be enumerated in
> the test metadata file.

Uh, having to have 1000 scripts, or even just 1000 entries in a metadata
file, just to run 1000 tests is a showstopper imho. Heck, identifying
testcases by number mightn't be particularly desirable in some instances,
if a clearer alternative like, say, "test case failed: add 1, add 2,
del 1, ch 2" is possible.

> You can't check that the binary works _when the .deb is installed_
> without installing it.

That's okay. You can't check that the binary works _on the user's system_
without installing it on the user's system either. For Debian's purposes,
being able to run the tests with minimal setup seems crucial.

> Also, a `Restriction' isn't right because if the test has neither of
> those Restrictions then presumably it can do either but how would it
> know which ?

It would have to not care which; which it could do by expecting the
test harness to put the binaries in the PATH, or provide an environment
variable like INSTALL_ROOT=$(pwd)/debian/tmp .

> No, I mean that if the tests live (say) in
> build/foo-1.0/debian/tests/x then build/foo-1.0/debian/tests/control
> could say
>  Depends: bar
> which would mean bar would have to be installed, effectively making it
> an integration test.

Having test case dependencies is fairly useful; in any event the language
"Even integration tests can be represented like this: if one package's
tests Depend on the other's" is wrong if tests depend on other packages,
not on other package's tests. You'll want Conflicts: as well as Depends:
in that case too. 

It would probably be quite useful to be able to write tests like:

	for mta in exim4 smail sendmail qmail; do
	   install $mta
	   # actual test
	   uninstall $mta

too, to ensure that packages that depend on one of a number of packages
actually work with all of them.


Attachment: signature.asc
Description: Digital signature

Reply to: