Bcc'ed to -project; followups to -devel. On Thu, Nov 17, 2005 at 06:43:32PM +0000, Ian Jackson wrote: > Note that the point is to be able to test the _actual package_, _as > installed_ (eg on a testbed system). This is much better than testing > the package from the source treeu during build time because it will > detect packaging mistakes as well as program source problems and as we > know packaging mistakes of one kind or another are one of the main > causes of problems. Mostly it's just different -- testing at build time lets you do unit tests before putting the objects together and stripping them, which gives you the opportunity to catch other bugs. One isn't better than the other; though doing both is better than either or neither. Other useful tests are things like "install all of optional" which will catch unspecified Conflicts: relations, and "do a partial upgrade from stable to this package in unstable" which will tell you if some dependencies aren't strict enough. Looking through Contents-* files will also let you catch unspecified dependencies. Having multiple machines to do tests can be worthwhile too -- if you want to test firewalls or that there aren't any listening ports on default installs, eg. > The source package provides a test metadata file debian/tests/ > control. This is a file containing zero or more RFC822-style > stanzas, along these lines: > Tests: fred bill bongo > Restrictions: needs-root breaks-computer > This means execute debian/tests/fred, debian/tests/bill, etc., Seems like: debian/tests/bar: #!/bin/sh # Restrictions: needs-root trashes-system # Requires: foo foo FAIL: ... bar SKIP: foo failed would make more sense than a separate file describing the tests? Is the "Depends:" line meant to refer to other Debian packages (and thus be a lower level version of Restrictions:) or is it meant to indiciate test interdependencies? If it's meant to be for debian packages, maybe # Restrictions: deb:xvncviewer might be better. Note that it's often better to have a single script run many tests, so you probably want to allow tests to pass back some summary information, or include the last ten lines of its output or similar. Something like: foo FAIL: FAILURE: testcase 231 FAILURE: testcase 289 FAILURE: testcase 314 3/512 test cases failed bar FAIL: (341123 other lines, then:) xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxx Aborted (core dumped) baz SKIP: foo failed quux PASS maybe. > Any unknown thing in Restrictions, or any unknown field in the > RFC822 stanza, causes the tester core to skip the test with a > message like `test environment does not support "blames-canada" > restriction of test "simpsons"'. You mean southpark, surely? > A basic test could be simply running the binary and checking the > result status (or other variants of this). Eventually every > package would to be changed to include at least one test. These sorts of tests are better done as part of debian/rules, I would've thought -- the advantage of that is that the problems get caught even when users rebuild the package themselves, and you don't need to worry about special test infrastructure like you're talking about for the simple case. > Ideally eventually where possible the upstream regression tests > could be massaged so that they test the installed version. Whether > this is possible and how best to achieve it has to be decided on a > per-package basis. Having Restrictions: package-installed and Restrictions: build-tree might be worth thinking about so that it's easy to do both sorts of testing. > Even integration tests can be represented like this: if one > package's tests Depend on the other's, then they are effectively > integration tests. The actual tests can live in whichever package > is most convenient. Going from build/foo-1.0/debian/tests/x to projects/bar-3.14/debian/tests/y seems difficult. Adding "deb:otherpkg" or "deb:libotherpkg-dbg" to the Restrictions: seems more plausible? Anyway, something that can be run with minimal amounts of setup seems most likely to be most useful: so running as part of the build without installing the package, running without anything special installed but the package being tested and a script that parses the control information, stuff that can be run on a user's system without root privs and without trashing the system, etc. If there's going to be a "debian/rules check" command, "debian/tests/*" probably should just be a suggested standard, or vice-versa -- minimising the number of required interfaces would likely make things more flexible. Being able to add upstream tests by nothing more than symlinking them into debian/tests might be a worthwhile goal, perhaps. > From: Ian Jackson <iwj@ubuntu.com> > Ian. > (wearing both my Debian and Ubuntu hats) Heh. Cheers, aj
Attachment:
signature.asc
Description: Digital signature