QA data repository
Hi all,
Today I've been working on a set of tools that will allow the results of
all sorts of automated tests on the Debian archive to be collected at
one central place, and be made available for Developers, testers, and
anyone else interested via one clear interface.
I'm thinking here of: lintian results, piuparts results, the various
efforts by people on automatically rebuilding the whole archive, etc.
But also, infrastructure to collect simpler types of data about
packages, like changelogs and copyright files, the section/priority of a
.deb (not generally available otherwise), and the explosed symbols of
shared libraries, explosed shlibs, and much more. This would for
example make it ultimately possible to do cross-archive symbol conflict
testing, and to implement archive-wide ABI regressions, and do all sort
of fancy testing that would otherwise involve having all packages
installed or something similarly.
This is (obviously) still work in progress, details can still change,
and not everything is implemented yet. As I see it, there are multiple
parts:
- A data repository. Currently in /org/qa.debian.org/rebuildtesting/{db,pool}
on merkel. This repository should get some smarter indices, but
basicly, the pool structure is most flexible and efficient, at the
cost of a level of indirection if you want "all $foo from unstable".
- A way to submit data into the repository. This works via email to
incoming@qa.debian.org, in a sepecific format. I've based my inject
script on the buildd.d.o code, because it's quite similar to that
actually. An example email would look like:
Subject: Result for lintian libc6_2.3.6-4_i386 1.23.6 ok 1E9W0I
<body with lintian results>
There isn't yet any sort of real authentication, but that will be
implemented. The testname (Result for $foo) needs to be configured
first, people who want to start submitting their buildlogs or whatnot,
can email me to get their tests added.
- A web interface to actually query in some convenient way the database.
To be implemented, though I did export pool and db via my public_html
on merkel (http://merkel.debian.org/~jeroen/{pool,db}). There are
various things around that might be worth looking at and taking code
from, suggestions and help is warmly welcomed. Requirements have some
overlap with the PTS, and DARTS (darts.alioth.debian.org) also has
some code to show similar results.
- A command-line interface to query the database, or at least, some
tools to help people quicly do one-off checks over the whole archive,
the type of work that people nowadays typically use the lintian lab
for. Probably will come into existance 'for free' with the web
interface.
- A way to actually distribute work. With this design all kinds of tests
can be done distributed, and it would be very good to have some
generic framework in which work can be divided to the slaves that
actually do the expensive tests. Currently, bin/genlist can maintain a
'todo' file that a script on lintian.d.o understands and can perform
lintian checks on to be submitted. It's intentially started off a bit
limited like this, but can reasonably easily be generalized. Different
type of tests have different requirements though, so this takes some
more work. Again, various pieces of code already exist, there's
piuparts, darts, the 'old' (May 2005) rebuildtesting code,
wanna-build/quinn-diff, to name just 4 I know of out of the top of
my head.
Unlike the previous iteration, the rebuildtesting code, it is now not
required for any given test to actually use this code distribution,
autonomous submission of results works just fine too.
- Something else I forgot about yet?
Comments welcome. I'll keep you posted, and once it's a bit more ready
(that is, with actually a web interface), this can be annouced more
widely.
--Jeroen
--
Jeroen van Wolffelaar
Jeroen@wolffelaar.nl (also for Jabber & MSN; ICQ: 33944357)
http://Jeroen.A-Eskwadraat.nl
Reply to: