Re: Automated testing - design and interfaces
Bill Allombert <allomber@math.u-bordeaux.fr> writes:
> Debian is an organisation which can afford a lot of decentralisation,
> but where centralisation is very expensive (debian-admin time, etc.).
> Doing the checks in debian/rules is not perfect, but it happens before
> the package is uploaded, is performed for all architecture and more
> importantly is done with the current infrastructure.
> Going toward a centralised testing facility where packages are checked
> before they go to unstable might be nice but even if the software was
> ready today, it would take ages before we could adapt the infrastructure
> to take advantage of it.
> It might be that Ubuntu is a different organisation with different
> ressources and the Debian way is not the most efficient for Ubuntu.
However, there do appear to me to be some obvious technical areas in which
Debian and Ubuntu could cooperate even if Debian isn't in a position to do
centralized testing for the time being. Standardizing a way to run
post-install tests, for example; a lot of software, particularly software
that doesn't have upstream tests and wasn't designed for testing out of
the build tree, will have to be installed in a chroot before it can be
effectively tested. That may be as simple as agreeing on some rule names
for debian/rules and possibly an output format.
Similarly, a way of enumerating the tests available, standardizing a way
of expressing test status (passed, failed, skipped, possibly various
reasons for skipping), dividing tests into fast and resource-intensive
tests, tagging tests that require an Internet connection... I agree that
many, if not most, Debian packages will not be in a position to *use* this
infrastructure immediately, but provided that Ubuntu has some cases in
mind so that these protocols can be designed in contact with the real
world and not just as a theoretical exercise, hashing out the protocol
details is still useful. That way, as people slowly add tests, all the
tests work together and one can automate testing at scale.
This is fairly standard stuff, really; gcc, Perl, and many other large
upstream projects have gone through this exercise to standardize test
formats so that testing can be automated.
> If Ubuntu want to improve the testing coverage, you could start by
> submitting patches to packages that don't run test-suite in
> debian/rules. That would profit both Debian and Ubuntu and there are
> lot of work to do there.
Upstream test suites are sometimes problematic, though; in particular,
many upstreams routinely ship releases with failing tests and you have to
be familiar with upstream development to know if that's a real problem or
not. That being said, certainly if one can run the upstream test suite
without causing such problems, it's a good idea.
--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>
Reply to: