Re: package testing, autopkgtest, and all that

[Ian Jackson <ijackson@chiark.greenend.org.uk>]

Stefano Zacchiroli writes ("package testing, autopkgtest, and all that"):
> Regarding this specific point (tests run on packages as if they were
> installed), IIRC Ian Jackson worked a bit on the matter, producing some
> code (autopkgtest---as mentioned elsewhere in this thread) and a
> specification of the interaction among tests and packages.  Ian: would
> you mind summarizing the status of that effort and comment on whether,
> in your opinion, people interested on this topic should continue from
> there or start over?

Sure.  autopkgtest (the codebase) isn't very big but it contains
several interlocking parts.  The key parts are:

 * A specification which allows a source package to declare that it
   contains tests, and how those tests need to be run.  This
   specification was discussed extensively on debian-devel at the
   time and a copy is in the autopkgtest package, but I'll follow up
   this email with a copy of it.

 * Machinery to interpret those declarations, and:
    - build the package (if needed)
    - install the package(s) needed for the runtime tests
    - run the tests (if any) and collect the results

 * Some surrounding ad-hoc shell scripts and crontab code to:
    - select a package to test
    - run the test
    - send the results in a fairly raw form to a webserver host
    - make some notes about how the test went for the benefit of the
       selection algorithm

 * A standardised interface to a virtualisation/snapshot testbed, with
   three implementations: Xen VMs and LVM snapshots; chroot; or
   simply running things on the actual host.

All of this seemed to work reasonably well.  The 1.2.0 in the archive
is essentially identical to my bzr head so all the autopkgtest code is
out there.


The problems are that:

 * Nowhere is actually automatically running these tests on an
   ongoing basis.  I was doing so but the machine I was doing it on
   was causing electrical problems in our house so I stopped.

 * We are lacking code to publish the results in a useful format
   that integrates well with the package qa pages etc.

 * Most packages in the archive declare no tests.  This is less of a
   show-stopper than you might imagine because in the obvious
   configuration even a package which declares no tests gets _built_,
   so we check for buildability.

 * Even when I was automatically running these tests, no-one was
   systematically looking at the results, filing bugs, etc.  I didn't
   have time to do this.  But even publishing results on the qa pages
   would be a good start.

 * Much of this code hasn't been run for a few years and may have
   rotted slightly.


> Having a specification and some code to run the tests early on in the
> Wheezy release cycle would be amazing, as it will enable maintainers to
> add tests to their packages during the expected package updates for
> Wheezy.

Absolutely.

If someone would like to set up a machine running these tests and
perhaps do some of the qa.debian.org integration, I would be
delighted.  

I'm doing a lot of test automation in my day job now so I'm somewhat
less interested in this myself (rather too much of a busman's holiday)
but I'd be very happy to help in any way I can.

Thanks,
Ian.