[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Automated testing - design and interfaces



Bcc'ed to -project; followups to -devel.

On Thu, Nov 17, 2005 at 06:43:32PM +0000, Ian Jackson wrote:
> Note that the point is to be able to test the _actual package_, _as
> installed_ (eg on a testbed system).  This is much better than testing
> the package from the source treeu during build time because it will
> detect packaging mistakes as well as program source problems and as we
> know packaging mistakes of one kind or another are one of the main
> causes of problems.

Mostly it's just different -- testing at build time lets you do unit
tests before putting the objects together and stripping them, which
gives you the opportunity to catch other bugs. One isn't better than the
other; though doing both is better than either or neither.

Other useful tests are things like "install all of optional" which
will catch unspecified Conflicts: relations, and "do a partial upgrade
from stable to this package in unstable" which will tell you if some
dependencies aren't strict enough. Looking through Contents-* files will
also let you catch unspecified dependencies.

Having multiple machines to do tests can be worthwhile too -- if you want
to test firewalls or that there aren't any listening ports on default
installs, eg.

>   The source package provides a test metadata file debian/tests/
>   control. This is a file containing zero or more RFC822-style
>   stanzas, along these lines:
> 	  Tests: fred bill bongo
> 	  Restrictions: needs-root breaks-computer
>   This means execute debian/tests/fred, debian/tests/bill, etc.,

Seems like:

  debian/tests/bar:
    #!/bin/sh
    # Restrictions: needs-root trashes-system
    # Requires: foo

  foo FAIL: ...
  bar SKIP: foo failed

would make more sense than a separate file describing the tests? Is the
"Depends:" line meant to refer to other Debian packages (and thus be
a lower level version of Restrictions:) or is it meant to indiciate
test interdependencies? If it's meant to be for debian packages, maybe

  # Restrictions: deb:xvncviewer

might be better.

Note that it's often better to have a single script run many tests, so
you probably want to allow tests to pass back some summary information,
or include the last ten lines of its output or similar. Something like:

  foo FAIL:
    FAILURE: testcase 231
    FAILURE: testcase 289
    FAILURE: testcase 314
    3/512 test cases failed
  bar FAIL: (341123 other lines, then:)
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxx
    Aborted (core dumped)
  baz SKIP: foo failed
  quux PASS

maybe.

>   Any unknown thing in Restrictions, or any unknown field in the
>   RFC822 stanza, causes the tester core to skip the test with a
>   message like `test environment does not support "blames-canada"
>   restriction of test "simpsons"'.

You mean southpark, surely?

>   A basic test could be simply running the binary and checking the
>   result status (or other variants of this). Eventually every
>   package would to be changed to include at least one test.

These sorts of tests are better done as part of debian/rules, I would've
thought -- the advantage of that is that the problems get caught even
when users rebuild the package themselves, and you don't need to worry
about special test infrastructure like you're talking about for the
simple case.

>   Ideally eventually where possible the upstream regression tests
>   could be massaged so that they test the installed version. Whether
>   this is possible and how best to achieve it has to be decided on a
>   per-package basis.

Having

  Restrictions: package-installed

and

  Restrictions: build-tree

might be worth thinking about so that it's easy to do both sorts of
testing.

>   Even integration tests can be represented like this: if one
>   package's tests Depend on the other's, then they are effectively
>   integration tests. The actual tests can live in whichever package
>   is most convenient.

Going from build/foo-1.0/debian/tests/x to
projects/bar-3.14/debian/tests/y seems difficult.

Adding "deb:otherpkg" or "deb:libotherpkg-dbg" to the Restrictions:
seems more plausible?

Anyway, something that can be run with minimal amounts of setup seems
most likely to be most useful: so running as part of the build without
installing the package, running without anything special installed but the
package being tested and a script that parses the control information,
stuff that can be run on a user's system without root privs and without
trashing the system, etc.

If there's going to be a "debian/rules check" command, "debian/tests/*"
probably should just be a suggested standard, or vice-versa --
minimising the number of required interfaces would likely make things
more flexible. Being able to add upstream tests by nothing more than
symlinking them into debian/tests might be a worthwhile goal, perhaps.

> From: Ian Jackson <iwj@ubuntu.com>

> Ian.
> (wearing both my Debian and Ubuntu hats)

Heh.

Cheers,
aj

Attachment: signature.asc
Description: Digital signature


Reply to: