On Wed, 2005-11-23 at 18:16 +1000, Anthony Towns wrote: > On Mon, Nov 21, 2005 at 06:22:37PM +0000, Ian Jackson wrote: > > > Note that it's often better to have a single script run many tests, so > > > you probably want to allow tests to pass back some summary information, > > > or include the last ten lines of its output or similar. Something like: > > > foo FAIL: > > > FAILURE: testcase 231 > > > FAILURE: testcase 289 > > > FAILURE: testcase 314 > > > 3/512 test cases failed > > This is no good because we want the test environment to be able to > > tell which tests failed, so the test cases have to be enumerated in > > the test metadata file. Replying to two messages here ... I don't think we have to enumerate the tests in advance. Sure the test runner needs to be able to identify and [possibly] categorise the tests, but explicit enumeration is quite orthogonal. A number of python unittest runners will scan directories and classes for their tests, and the report from users is consistently that this is easier to use. > Uh, having to have 1000 scripts, or even just 1000 entries in a metadata > file, just to run 1000 tests is a showstopper imho. Heck, identifying > testcases by number mightn't be particularly desirable in some instances, > if a clearer alternative like, say, "test case failed: add 1, add 2, > del 1, ch 2" is possible. A very nice feature in the xUnit world is that tests can be identified by either their path (inside the language namespace) or by a comment written by the author. At runtime you can choose which to see. I dont think we need the ability for runtime selection, but having a heuristic that works and is overridable would be nice. I.e. by default you might get tests/layout/documentation_in_usr_share_doc.sh: PASS But inside that test you could say: test_name Documentation is installed in /usr/share/doc and the output becomes Documentation is installed in /usr/share/doc: PASS I've written a project that is somewhat related to this: http://www.robertcollins.net/unittest/subunit/ Its a python implementation of a cross process test running protocol. This lets a sub process run 0 to many tests, identify them and provide pass/fail/error status and traceback or other diagnostics. As the driver is python its not appropriate here, but I think the basic protocol/concept might be useful. > > You can't check that the binary works _when the .deb is installed_ > > without installing it. > > That's okay. You can't check that the binary works _on the user's system_ > without installing it on the user's system either. For Debian's purposes, > being able to run the tests with minimal setup seems crucial. Yup > It would probably be quite useful to be able to write tests like: > > for mta in exim4 smail sendmail qmail; do > install $mta > # actual test > uninstall $mta > done > > too, to ensure that packages that depend on one of a number of packages > actually work with all of them. Yes, that would be excellent. Rob -- GPG key available at: <http://www.robertcollins.net/keys.txt>.
Attachment:
signature.asc
Description: This is a digitally signed message part