Thoughts on the new Lintian test suite
I love the new test suite, and as you can probably see from the commit
notifications, I've been trying it out and adding a bunch of additional
test cases. After working with it for a while, I've developed a bit of a
wishlist. Here are my ideas; please send any comments (particularly
Frank, since it's building on your work):
* The t/tests directory is getting quite large. One easy thing we could
do that would help this, and which I think would also make sense given
the rest of the layout, is to move the *.desc file for a test from
t/tests into the test directory (calling it desc or something) and then
adding Sequence as a required field.
This requires runtests to do a bit more work to find all the test cases
and order them, but it shouldn't be too bad. It would parse
t/tests/*/desc and then sort by $desc->{sequence} followed by
$desc->{testname}. For debian/rules onlyrun, you'd have to omit the
sequence number, but I would prefer that anyway.
* I think we're going to need more types of tests than package builds for
a long-term goal of coverage of as much of Lintian as possible. Other
test case ideas that I've already started wanting:
- Normal Perl tests, using Test::More or the like, run under
Test::Harness (probably via prove). This would be a better framework
for testing our utility libraries, things like Dep or Lintian::Data.
We could also use Test::Pod to syntax-check the POD documentation in
our libraries.
- A directory for a Debian package including a DEBIAN directory,
suitable for running dpkg -b on and then testing with Lintian. This
would let us test certain pathological cases that are hard to shove
through dpkg-buildpackage, such as missing debian/rules or a missing
required field in debian/control like Maintainer. The purpose here is
less to test the tags (although that would be nice to do; it is
possible for people to build and upload packages without using
dpkg-buildpackage, although generally ftp-master would catch them)
than to ensure that Lintian doesn't crash or misbehave on pathological
packages. I'm imagining a directory of <test> directories and
<test>.tags files. (Hopefully we won't need sed for these.)
- A directory of *.changes files to test some of the things in Lintian's
parsing that are hard to trigger via straight dpkg-buildpackage. I'm
thinking of a test case for #315538, for example (double-signed
*.changes files). I'm imagining a directory of *.changes and *.tags
files.
It would be fairly easy to add the logic to t/runtests to handle all
those test types. If we do this, the current t/tests is too generic of
a name and should probably be renamed to t/packages.
* I don't think we need to maintain both test harnesses until we've had a
chance to break the legacy test cases apart into separate,
better-documented test cases. I propose moving all the testset test
cases into the new test suite as legacy-* test cases with a 6600
sequence number so that they run last.
I should also note, as I mentioned in the documentation, that I've not
been sticking to one test per tag, since if we do that we're going to end
up with a truly amazing number of tests. Instead, when a set of tests
seem closely related, I've been combining them into one test case,
sometimes one that generates multiple binary packages if that's needed to
test a variety of things. Examples are control-file-general, which tests
a variety of syntax errors and problems in debian/control, or
copyright-file-general, which generates a bunch of binary packages with
different copyright problems. I find this structure a bit simpler than
one test case per tag since it makes it a little easy to add, say, a new
copyright file by just creating a new file in the debian/ directory and
adding a new stanza to debian/control.in.
Another example where I think this makes sense is a files-general test
that installs a bunch of junk into a test package and tests all the tags
for "weird thing in package" at once, since they're all fairly obviously
independent.
--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>
Reply to: