RFC: A regression test framework for Debian packages?

[Jérôme Marant <jerome@marant.org>]

One year ago, Martin Michlmayr made a speach at the Debian
Conference 1 about Quality Assurance within Debian and the
progress that have been made so far.
He mentioned that we lack regression tests for packages,
but as far as I know, no solution has been found.

What are regression tests?
--------------------------

  Regression tests are tests that are made to ensure that 
  the behaviour of a given program has not changed across
  releases.

Why?
----

  Currently, we have some way to check a package before it
  gets installed. That is what Lintian (and Linda as well ;-)
  is here for: checks are performed on the content of the
  binary and source package.

  However, we have no way to check a package after its
  installation. And in most of the cases, Lintian checks
  cannot prevent software breakages.

  Until now, developers need to test packages manually:
  installation, upgrade, removal, purge, failures and so on.
  This can be very painful and boring when you have different
  ways to configure your packages, especially when they
  have to be configured through debconf.
  Furthermore, some packages may perform some complex
  tasks like database creation/upgrades, web server configuration,
  mail aliases management, etc and user configuration and
  environment may vary, making the success of such tasks
  unpredictable.

What?
-----

  The main idea is to make tests on package installation
  and after their installation is completed.

  Here is a list of possible tests that come to my mind
  (in no special order):

  - tests on maintainer scripts: maintainer scripts perform
  some very simple tasks like: changing ownerships and
  permissions, starting daemons, creating users/groups, etc.
  They are likely to change across releases, and as they
  are mostly written in shell, they are error prone.

  This could be as complex as checking that a database has
  been properly configured, that a LDAP directory has been
  properly populated, that a web server has been correctly
  modified.

  - debconf tests: there are usually many possible answer
  to debconf question and different paths within. Some
  tests can be written to simulate user interaction, providing
  a broader set of configuration answer. Checks would be
  made with respect to some given answers.

  - configuration files tests: some behaviours may vary with
  respect to configuration options. Tests would provide
  a set of configuration files and their matching expected
  behaviour.

  - application-specific tests: sometimes Debian customize
  applications and we may want to see if nothing has been
  broken.

  - ...

How?
----

  In a chrooted environment, we can install deboostrap and
  package dependencies through APT.

  Then, we need some program that would execute some simple
  scripts in a pseudo language (for safety purpose, tests must
  be error prone as much as possible) describing what tests have
  to be performed, and how they would be performed.

  This is only a short summary :-)

Do we need this?
-----------------

  Although this may look boring for small packages, I have
  many reasons to think that this can be usefull for complex
  packages. We can imagine running such test overnight for
  packages maintained via CVS.


Comments are welcome.

Thanks.

Cheers,

PS: Please, no CC on reply, I'm subscribed to the list.

-- 
Jérôme Marant