RFC: A regression test framework for Debian packages?
One year ago, Martin Michlmayr made a speach at the Debian
Conference 1 about Quality Assurance within Debian and the
progress that have been made so far.
He mentioned that we lack regression tests for packages,
but as far as I know, no solution has been found.
What are regression tests?
Regression tests are tests that are made to ensure that
the behaviour of a given program has not changed across
Currently, we have some way to check a package before it
gets installed. That is what Lintian (and Linda as well ;-)
is here for: checks are performed on the content of the
binary and source package.
However, we have no way to check a package after its
installation. And in most of the cases, Lintian checks
cannot prevent software breakages.
Until now, developers need to test packages manually:
installation, upgrade, removal, purge, failures and so on.
This can be very painful and boring when you have different
ways to configure your packages, especially when they
have to be configured through debconf.
Furthermore, some packages may perform some complex
tasks like database creation/upgrades, web server configuration,
mail aliases management, etc and user configuration and
environment may vary, making the success of such tasks
The main idea is to make tests on package installation
and after their installation is completed.
Here is a list of possible tests that come to my mind
(in no special order):
- tests on maintainer scripts: maintainer scripts perform
some very simple tasks like: changing ownerships and
permissions, starting daemons, creating users/groups, etc.
They are likely to change across releases, and as they
are mostly written in shell, they are error prone.
This could be as complex as checking that a database has
been properly configured, that a LDAP directory has been
properly populated, that a web server has been correctly
- debconf tests: there are usually many possible answer
to debconf question and different paths within. Some
tests can be written to simulate user interaction, providing
a broader set of configuration answer. Checks would be
made with respect to some given answers.
- configuration files tests: some behaviours may vary with
respect to configuration options. Tests would provide
a set of configuration files and their matching expected
- application-specific tests: sometimes Debian customize
applications and we may want to see if nothing has been
In a chrooted environment, we can install deboostrap and
package dependencies through APT.
Then, we need some program that would execute some simple
scripts in a pseudo language (for safety purpose, tests must
be error prone as much as possible) describing what tests have
to be performed, and how they would be performed.
This is only a short summary :-)
Do we need this?
Although this may look boring for small packages, I have
many reasons to think that this can be usefull for complex
packages. We can imagine running such test overnight for
packages maintained via CVS.
Comments are welcome.
PS: Please, no CC on reply, I'm subscribed to the list.