On Mon, Nov 21, 2005 at 06:22:37PM +0000, Ian Jackson wrote: > > Note that it's often better to have a single script run many tests, so > > you probably want to allow tests to pass back some summary information, > > or include the last ten lines of its output or similar. Something like: > > foo FAIL: > > FAILURE: testcase 231 > > FAILURE: testcase 289 > > FAILURE: testcase 314 > > 3/512 test cases failed > This is no good because we want the test environment to be able to > tell which tests failed, so the test cases have to be enumerated in > the test metadata file. Uh, having to have 1000 scripts, or even just 1000 entries in a metadata file, just to run 1000 tests is a showstopper imho. Heck, identifying testcases by number mightn't be particularly desirable in some instances, if a clearer alternative like, say, "test case failed: add 1, add 2, del 1, ch 2" is possible. > You can't check that the binary works _when the .deb is installed_ > without installing it. That's okay. You can't check that the binary works _on the user's system_ without installing it on the user's system either. For Debian's purposes, being able to run the tests with minimal setup seems crucial. > Also, a `Restriction' isn't right because if the test has neither of > those Restrictions then presumably it can do either but how would it > know which ? It would have to not care which; which it could do by expecting the test harness to put the binaries in the PATH, or provide an environment variable like INSTALL_ROOT=$(pwd)/debian/tmp . > No, I mean that if the tests live (say) in > build/foo-1.0/debian/tests/x then build/foo-1.0/debian/tests/control > could say > Depends: bar > which would mean bar would have to be installed, effectively making it > an integration test. Having test case dependencies is fairly useful; in any event the language "Even integration tests can be represented like this: if one package's tests Depend on the other's" is wrong if tests depend on other packages, not on other package's tests. You'll want Conflicts: as well as Depends: in that case too. It would probably be quite useful to be able to write tests like: for mta in exim4 smail sendmail qmail; do install $mta # actual test uninstall $mta done too, to ensure that packages that depend on one of a number of packages actually work with all of them. Cheers, aj
Attachment:
signature.asc
Description: Digital signature