Automated testing - design and interfaces

[Ian Jackson <iwj@ubuntu.com>]

Note: that this is one of two messages on roughly the same topic.

This message will deal solely with TECHNICAL issues.  If you have some
technical followup then please go ahead and reply here.  If you have a
complaint or comment about my or Ubuntu's approach, please reply in
debian-*project* to my other message, titled `Automated testing -
politics, information, and Ubuntu's plans'.

The first two paragraphs are common to the two messages:


One thing that the Debian universe is lacking is a good way to
automatically test packages.  This makes it hard to spot regressions,
and difficult to be systematic.

Ubuntu are proposing to invent a system to allow us to do automated
regression tests.  The full plan consists of a number of pieces and
can be read here:
  https://wiki.ubuntu.com/AutomatedTesting


If you're interested in this subject I'd appreciate it if you could go
and take a look.  Feel free to make your comments here on -policy, or
any other appropriate channel.

One of the important parts needed here is an interface for packages to
supply tests.  That is, a standard way for a Debian-format package to
enumerate and describe the tests it supplies, and to allow a test
harness to invoke them.

Note that the point is to be able to test the _actual package_, _as
installed_ (eg on a testbed system).  This is much better than testing
the package from the source treeu during build time because it will
detect packaging mistakes as well as program source problems and as we
know packaging mistakes of one kind or another are one of the main
causes of problems.

It seems to me that the right way for packages to offer their tests in
the source tree.  As I say in the wiki page:

  Other possibilities include a special .deb generated by the source
  (which is a bit strange and what happens to this .deb and will make
  it even harder to reuse upstream test suites), or putting them in
  the .deb to be tested (definitely wrong - most people won't want the
  tests and they might be very large) or having them floating about
  separately somewhere (which prevents us from sharing and exchanging
  tests with other parts of the Free Software community).  The source
  package is always available when development is taking place.


To save you looking at the wiki page for the details of the draft
interface spec. I have reproduced it here:

  Tests/metadata
  ...

  The source package provides a test metadata file debian/tests/
  control. This is a file containing zero or more RFC822-style
  stanzas, along these lines:

	  Tests: fred bill bongo
	  Restrictions: needs-root breaks-computer

  This means execute debian/tests/fred, debian/tests/bill, etc.,
  each with no arguments, expecting exit status 0 and no stderr. The
  cwd is guaranteed to be the root of the source package which will
  have been built (but note that the tests must test the installed
  version).

  If the stanza contains:

	  Tests-Directory: path-from-source-root

  then we execute path-from-source-root/fred, path-from-source-root/
  bar, etc. This allows tests to live outside the debian/ metadata
  area, so that they can more palatably be shared with non-Debian
  distributions.

  Any unknown thing in Restrictions, or any unknown field in the
  RFC822 stanza, causes the tester core to skip the test with a
  message like `test environment does not support "blames-canada"
  restriction of test "simpsons"'.

  Additional possibilities:

	  Depends: ...
	  Tests: filenamepattern*
	  Restrictions: modifies-global-data needs-x-display

  etc. - moves complexity from individual packages into central
  tester core.

  A basic test could be simply running the binary and checking the
  result status (or other variants of this). Eventually every
  package would to be changed to include at least one test.

  Ideally eventually where possible the upstream regression tests
  could be massaged so that they test the installed version. Whether
  this is possible and how best to achieve it has to be decided on a
  per-package basis.

  Even integration tests can be represented like this: if one
  package's tests Depend on the other's, then they are effectively
  integration tests. The actual tests can live in whichever package
  is most convenient.


So, what do you think ?


Thanks,
Ian.
(wearing both my Debian and Ubuntu hats)