Proposal for a new test suite
Hi.
It is my opinion that the current test suite doesn't scale very well and
I would like to supplement (and in the long run replace) it with
something better.
Here is a first proposal on how to do that. Comments welcome.
The old test suite has some disadvantages:
* It mixes several tests in one test package,
making it ever more likely that the tests
are affected by each other.
* This also makes it ever more difficult to find
out which parts of a test package were created
for which tag to test.
* It does not denote whether a tag is only issued
as a side effect or it was caused deliberatly
* It does not allow to specify whether a test package is
specifically designed to *not* issue a tag, making it
possible that this intent gets lost in time.
* The tests include some code duplication.
* Many test packages issue many tags only as side effect
since they are very basic.
* .orig.tar.gz files have to be included in the VCS.
The new test suite tries to address these issues:
* It creates the test packages on-the-fly, making
it possible to create many test packages with
a smaller scope with very little duplication.
* It uses a debhelper 7 based default skeleton,
which makes the skeleton very flexible and achieves
a good standards compatibility (i.e. low overhead of
unneccessary tags).
* It includes a control file for each test which can be
be used to include meta data about the intentions of
the test.
* It supports building .orig.tar.gz files on-the-fly,
making it easier to create non-native test packages.
Status of this code:
* It builds a list of test, creates the packages for them,
builds them and runs lintian.
* It also support the literal comparison method (i.e. cmp -s)
between two tag files as done by the old test suite.
* It does not yet use the description files to refine this
comparison.
---
.gitignore | 1 +
debian/rules | 8 +-
t/runtests | 395 ++++++++++++++++++++++++++++++
t/templates/skel/debian/changelog | 9 +
t/templates/skel/debian/changelog.in | 6 +
t/templates/skel/debian/compat | 1 +
t/templates/skel/debian/control.in | 13 +
t/templates/skel/debian/copyright | 21 ++
t/templates/skel/debian/rules | 3 +
t/tests/0000_basic.desc | 2 +
t/tests/0001_basic-non-native.desc | 3 +
t/tests/basic-non-native/upstream/README | 1 +
t/tests/basic/debian/README | 1 +
13 files changed, 461 insertions(+), 3 deletions(-)
create mode 100755 t/runtests
create mode 100644 t/templates/skel.upstream/.dummy
create mode 100644 t/templates/skel/debian/changelog
create mode 100644 t/templates/skel/debian/changelog.in
create mode 100644 t/templates/skel/debian/compat
create mode 100644 t/templates/skel/debian/control.in
create mode 100644 t/templates/skel/debian/copyright
create mode 100755 t/templates/skel/debian/rules
create mode 100644 t/tests/0000_basic.desc
create mode 100644 t/tests/0001_basic-non-native.desc
create mode 100644 t/tests/basic-non-native/tags
create mode 100644 t/tests/basic-non-native/upstream/README
create mode 100644 t/tests/basic/debian/README
create mode 100644 t/tests/basic/tags
diff --git a/.gitignore b/.gitignore
index 1d3d3b9..428ea69 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,3 +1,4 @@
debian/tests
runtests
!testset/runtests
+!t/runtests
diff --git a/debian/rules b/debian/rules
index ae67ee5..3a6fd2c 100755
--- a/debian/rules
+++ b/debian/rules
@@ -6,13 +6,15 @@ usl := $(tmp)/usr/share/lintian
neededfiles := debian/rules frontend/lintian
allchecks := $(wildcard checks/*)
allcollect := $(wildcard collection/*)
-tagfiles := $(wildcard testset/tags.*)
+tagfiles := $(wildcard testset/tags.* t/*/tags)
+testfiles := $(wildcard t/tests/*.desc)
onlyrun =
-runtests: $(neededfiles) $(allchecks) $(allcollect) $(tagfiles)
+runtests: $(neededfiles) $(allchecks) $(allcollect) $(tagfiles) $(testfiles)
@echo .... running tests ....
[ -d debian/tests ] || mkdir debian/tests
LINTIAN_ROOT="" /usr/bin/perl testset/runtests -k testset debian/tests $(onlyrun)
+ LINTIAN_ROOT="" /usr/bin/perl t/runtests -k t debian/tests $(onlyrun)
touch $@
build: $(neededfiles)
diff --git a/t/runtests b/t/runtests
new file mode 100755
index 0000000..1ac48c9
--- /dev/null
+++ b/t/runtests
@@ -0,0 +1,395 @@
+#!/usr/bin/perl
+
+# Copyright © 1998 Richard Braakman
+# Copyright © 2008 Frank Lichtenheld
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, you can find it on the World Wide
+# Web at http://www.gnu.org/copyleft/gpl.html, or write to the Free
+# Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston,
+# MA 02110-1301, USA.
+
+use strict;
+use warnings;
+
+sub usage {
+ print <<END;
+Usage: $0 [-k] [-v] [-d] testset-directory testing-directory [test]
+
+The -k option means do not stop after one failed test, but try
+them all and report all errors.
+
+The -v option will also display those tests that have a description, but are
+not tested in any testset-package.
+
+The -d option will display debugging information.
+
+The optional 3rd parameter causes runtests to only run that particular test.
+END
+ exit 2;
+}
+
+# for debugging:
+my $debug = 0;
+
+# Tests layout:
+# XXX FIXME XXX
+
+# The build output is directed to build.pkgname in the testing-directory.
+
+# Exit codes:
+# 0 - success
+# 1 - one or more tests failed
+# 2 - an error prevented proper running of the tests
+
+# Turns out I might as well have written this in bash. Oh well.
+
+my $run_all_tests = 0;
+my $verbose = 0;
+
+# --- Parse options, such as they are.
+while ($#ARGV >= 0 && $ARGV[0] =~ m/^-/) {
+ if ($ARGV[0] eq '-k') {
+ $run_all_tests = 1;
+ } elsif ($ARGV[0] eq '-v') {
+ $verbose = 1;
+ } elsif ($ARGV[0] eq '-d') {
+ $debug = 1;
+ } else {
+ usage;
+ }
+ shift;
+}
+
+# --- Parse directory arguments
+if ($#ARGV < 1 || $#ARGV > 2) {
+ usage;
+}
+
+my $testset = shift;
+my $rundir = shift;
+my $singletest;
+if ($#ARGV == 0) {
+ $singletest = shift;
+}
+
+# --- Set and unset environment variables that lintian is sensitive to
+BEGIN {
+ my $LINTIAN_ROOT = $ENV{'LINTIAN_ROOT'};
+ if (not $LINTIAN_ROOT) {
+ use Cwd ();
+ $ENV{'LINTIAN_ROOT'} = $LINTIAN_ROOT = Cwd::cwd();
+ }
+ delete $ENV{'LINTIAN_CFG'};
+ delete $ENV{'LINTIAN_LAB'};
+ delete $ENV{'LINTIAN_DIST'};
+ delete $ENV{'LINTIAN_UNPACK_LEVEL'};
+ $ENV{'LC_COLLATE'} = 'C';
+
+ # Set standard umask because many of the test packages rely on this
+ # when creating files from the debian/rules script.
+ umask(022);
+}
+
+my $LINTIAN_ROOT = $ENV{'LINTIAN_ROOT'};
+
+use lib "$ENV{'LINTIAN_ROOT'}/lib";
+use Util;
+use Tags;
+
+# --- Set the ways to call lintian and dpkg-buildpackage
+my $lintian_options = '-I -E';
+my $lintian_info_options = '-I -E -i';
+my $dpkg_buildpackage_options = '-rfakeroot -us -uc -d -iNEVER_MATCH_ANYTHING'
+ . ' -INEVER_MATCH_ANYTHING';
+my $lintian_path = $LINTIAN_ROOT . "/frontend/lintian";
+
+my $testok = 0;
+my %tags;
+my %types = ( 'E' => 'error', 'W' => 'warning', 'I' => 'info', 'X' => 'experimental' );
+
+# --- Display output immediately
+$| = 1;
+
+# --- Let's play.
+
+-d $rundir
+ or fail("test directory $rundir does not exist\n");
+
+# does every tag have an info section?
+print "Checking for missing info tags ... ";
+
+$testok = 1;
+for my $desc_file (<$LINTIAN_ROOT/checks/*.desc>) {
+ for my $i (read_dpkg_control($desc_file)) {
+ $desc_file =~ s#.*/##;
+ if (exists $i->{'tag'}) {
+ if ($i->{'tag'} !~ /^[\w0-9.+-]+$/) {
+ print "E: test-tag-has-invalid-characters $i->{'tag'}"
+ . " in $desc_file\n";
+ }
+ if (not exists $i->{'info'}) {
+ print "E: test-has-no-info $i->{'tag'} in $desc_file\n";
+ $testok = 0;
+ }
+
+ # Check the tag info for unescaped <> or for unknown tags (which
+ # probably indicate the same thing).
+ my $info = $i->{'info'};
+ my @tags;
+ while ($info =~ s,<([^\s>]+)(?:\s+href=\"[^\"]+\")?>.*?</\1>,,s) {
+ push (@tags, $1);
+ }
+ my %known = map { $_ => 1 } qw(a em i tt);
+ my %seen;
+ @tags = grep { !$known{$_} && !$seen{$_}++ } @tags;
+ if (@tags) {
+ print "E: test-info-has-unknown-html-tags $i->{'tag'} @tags"
+ . " in $desc_file\n";
+ }
+ if ($info =~ /[<>]/) {
+ print "E: test-info-has-stray-angle-brackets $i->{'tag'}"
+ . " in $desc_file\n";
+ }
+
+ if (!exists($i->{'type'}) && !exists($i->{'severity'})) {
+ use Data::Dumper;
+ print Dumper $i;
+ print "E: test-has-no-type $i->{'tag'} in $desc_file\n";
+ $testok = 0;
+ next;
+ }
+
+ $tags{$i->{'tag'}}{'desc_file'} = $desc_file;
+ if (exists $i->{'experimental'}) {
+ $tags{$i->{'tag'}}{'desc_type'} = "experimental";
+ } else {
+ $tags{$i->{'tag'}}{'desc_type'} = $i->{'type'} ||
+ $Tags::sev_to_type[$i->{'severity'}];
+ }
+ }
+ }
+}
+
+if ($testok) {
+ print "done.\n";
+} else {
+ print "FAILED!\n";
+ exit 1 unless $run_all_tests;
+}
+
+# can I make a lab?
+print "Running static lab test ... create ... ";
+$testok = runsystem_ok("$lintian_path --lab $rundir/test_lab --setup-lab");
+# can I renew a lab?
+print " renew ... ";
+$testok = runsystem_ok("$lintian_path --lab $rundir/test_lab --setup-lab")
+ if $testok;
+# can I remove a lab?
+print " remove ...";
+$testok = runsystem_ok("$lintian_path --lab $rundir/test_lab --remove-lab")
+ if $testok;
+# should be empty now
+print " rmdir ...";
+$testok = runsystem_ok("rmdir $rundir/test_lab")
+ if $testok;
+# cleanup
+runsystem("rm -r $rundir/test_lab") if -d "$rundir/test_lab";
+if ($testok) {
+ print "done.\n";
+} else {
+ print "FAILED!\n";
+ exit 1 unless $run_all_tests;
+}
+
+# ok, I can make a static lab, now let's test the package checks
+# in temporary labs
+my @tests;
+if ($singletest) {
+ @tests = map { s/\.desc$// } ( $singletest );
+} else {
+ -d $testset
+ or fail("cannot find $testset: $!\n");
+
+ @tests = map { s#^\Q$testset/tests/\E## ;s/\.desc$//; $_ } sort(<$testset/tests/*.desc>);
+}
+
+print "found the following tests: @tests\n" if $debug;
+for (@tests) {
+ my $testdesc = "$testset/tests/$_.desc";
+
+ print "process $testdesc...\n" if $debug;
+ my $testdata = (read_dpkg_control($testdesc))[0];
+
+ check_test_is_sane($testset, $testdata);
+ print "Running test $testdata->{testname} $testdata->{version}... ";
+
+ my $pkg = $testdata->{srcpkg};
+ my $pkgdir = "$pkg-$testdata->{version}";
+ my $origdir = "$testset/tests/$testdata->{testname}";
+ my $targetdir = "$rundir/$pkgdir";
+ my $tmpldir = "$testset/templates";
+
+ my $is_native = ($testdata->{type} eq 'native');
+
+ print "Cleaning up and repopulating $targetdir...\n" if $debug;
+ runsystem_ok("rm", "-rf", $targetdir);
+ if ($is_native) {
+ runsystem("cp", "-rp", "$tmpldir/skel", $targetdir);
+ runsystem("rm", "-f", "$targetdir/debian/changelog");
+ runsystem("rsync", "-rp", "$origdir/debian/", "$targetdir/")
+ if -d "$origdir/debian/";
+ } else {
+ runsystem("cp", "-rp", "$tmpldir/skel.upstream", $targetdir);
+ runsystem("rm", "-f", "$targetdir/.dummy");
+ runsystem("rsync", "-rp", "$origdir/upstream/", "$targetdir/");
+ runsystem("cd $rundir && ".
+ "tar czf ${pkg}_$testdata->{version}.orig.tar.gz $pkgdir");
+ runsystem("rsync", "-rp", "--exclude=debian/changelog",
+ "$tmpldir/skel/", "$targetdir/");
+ runsystem("rsync", "-rp", "$origdir/debian/", "$targetdir/")
+ if -d "$origdir/debian/";
+ }
+
+ unless (-e "$targetdir/debian/changelog") {
+ fill_in_tmpl("$targetdir/debian/changelog", $testdata);
+ }
+ unless (-e "$targetdir/debian/control") {
+ fill_in_tmpl("$targetdir/debian/control", $testdata);
+ }
+ unless ($is_native || -e "$targetdir/debian/watch") {
+ runsystem("echo >$targetdir/debian/watch");
+ }
+ if (-x "$origdir/pre_build") {
+ print "running pre_build hook...\n";
+ runsystem("$origdir/pre_build", $targetdir);
+ }
+
+ print "building... ";
+ runsystem("cd $rundir/$pkgdir && dpkg-buildpackage $dpkg_buildpackage_options >../build.$pkg 2>&1");
+
+ print "testing... ";
+ runsystem_ok("$lintian_path $lintian_options $rundir/$pkg\_$testdata->{version}*.changes".
+ " 2>&1 | sort > $rundir/tags.$pkg");
+
+ # Run a sed-script if it exists, for tests that have slightly variable
+ # output
+ runsystem_ok("sed -i -f $origdir/post_test $rundir/tags.$pkg")
+ if -e "$origdir/post_test";
+
+ $testok = runsystem_ok("cmp", "-s", "$rundir/tags.$pkg", "$origdir/tags");
+ if ($testok) {
+ print "done.\n";
+ } else {
+ print "FAILED:\n";
+ runsystem_ok("diff", "-u", "$origdir/tags", "$rundir/tags.$pkg");
+ exit 1 unless $run_all_tests;
+ next;
+ }
+
+ open TAGS, "$rundir/tags.$pkg" or fail("Cannot open $rundir/tags.$pkg");
+ while (<TAGS>) {
+ next if m/^N: /;
+ if (not /^(.): (\S+)(?: (?:source|udeb))?: (\S+)/) {
+ print "E: Invalid line:\n$_";
+ next;
+ }
+ $tags{$3}{'tested_type'} = $types{$1};
+ $tags{$3}{'tested_package'} = $2;
+ }
+ close TAGS;
+}
+
+print "Checking whether all tags are tested and tags have description ... \n";
+$testok = 1;
+for (keys %tags) {
+ my $values = $tags{$_};
+ if (not defined $values->{'desc_type'}) {
+ print "E: tag-has-no-description $_ in $values->{'tested_package'}\n";
+ $testok = 0;
+ } elsif (not defined $values->{'tested_type'}) {
+ print "I: tag-is-not-tested $_ in $values->{'desc_file'}\n"
+ if $verbose;
+ } elsif ($values->{'desc_type'} ne $values->{'tested_type'}) {
+ print "E: tag-has-inconsistent-type $_ $values->{'tested_type'} vs ".
+ "$values->{'desc_type'}\n";
+ $testok = 0;
+ }
+}
+
+if ($testok) {
+ print "done.\n";
+} else {
+ print "FAILED\n";
+ exit 1 unless $run_all_tests;
+}
+
+# --------------
+sub runsystem {
+ print "runsystem(@_)\n" if $debug;
+ system(@_) == 0
+ or fail("failed: @_\n");
+}
+
+sub runsystem_ok {
+ print "runsystem_ok(@_)\n" if $debug;
+ my $errcode = system(@_);
+ $errcode == 0 or $errcode == (1 << 8)
+ or fail("failed: @_\n");
+ return $errcode == 0;
+}
+
+use Text::Template;
+sub fill_in_tmpl {
+ my ($file, $data) = @_;
+ my $tmpl = "$file.in";
+
+ my $template = Text::Template->new(TYPE => 'FILE', SOURCE => $tmpl);
+ open my $out, '>', $file
+ or fail("cannot open $file: $!");
+
+ unless ($template->fill_in(OUTPUT => $out, HASH => $data)) {
+ fail("cannout create $file");
+ }
+ close $out;
+}
+
+use Data::Dumper;
+sub check_test_is_sane {
+ my ($dir, $data) = @_;
+
+ if ($debug) {
+ print "check_test_is_sane <= ".Dumper($data);
+ }
+
+ unless ($data->{testname} && $data->{version}) {
+ fail("Name or Version missing");
+ }
+
+ $data->{srcpkg} ||= $data->{testname};
+ $data->{type} ||= 'native';
+ $data->{date} ||= `date -R`; chomp $data->{date};
+ $data->{description} ||= 'No Description Available';
+ $data->{author} ||= 'Debian Lintian Maintainers <lintian-maint@debian.org>';
+ $data->{architecture} ||= 'all';
+
+ if ($debug) {
+ print "check_test_is_sane => ".Dumper($data);
+ }
+}
+
+# Local Variables:
+# indent-tabs-mode: t
+# cperl-indent-level: 4
+# End:
+# vim: ts=8 sw=4
diff --git a/t/templates/skel.upstream/.dummy b/t/templates/skel.upstream/.dummy
new file mode 100644
index 0000000..e69de29
diff --git a/t/templates/skel/debian/changelog b/t/templates/skel/debian/changelog
new file mode 100644
index 0000000..66eca6b
--- /dev/null
+++ b/t/templates/skel/debian/changelog
@@ -0,0 +1,9 @@
+skel (1.0) unstable; urgency=low
+
+ * Initial design.
+ Please see changelog.in for the actual changelog
+ that will be used unless it is overwritten in the
+ individual test.
+
+ -- Frank Lichtenheld <djpig@debian.org> Sun, 03 Aug 2008 16:46:13 +0200
+
diff --git a/t/templates/skel/debian/changelog.in b/t/templates/skel/debian/changelog.in
new file mode 100644
index 0000000..240c83a
--- /dev/null
+++ b/t/templates/skel/debian/changelog.in
@@ -0,0 +1,6 @@
+{$srcpkg} ({$version}) unstable; urgency=low
+
+ * Lintian Test Suite.
+ * Test: {$testname}
+
+ -- {$author} {$date}
diff --git a/t/templates/skel/debian/compat b/t/templates/skel/debian/compat
new file mode 100644
index 0000000..7f8f011
--- /dev/null
+++ b/t/templates/skel/debian/compat
@@ -0,0 +1 @@
+7
diff --git a/t/templates/skel/debian/control.in b/t/templates/skel/debian/control.in
new file mode 100644
index 0000000..bc8f6c0
--- /dev/null
+++ b/t/templates/skel/debian/control.in
@@ -0,0 +1,13 @@
+Source: {$srcpkg}
+Priority: extra
+Section: devel
+Maintainer: {$author}
+Standards-Version: 3.8.0
+Build-Depends: debhelper (>= 7)
+
+Package: {$srcpkg}
+Architecture: {$architecture}
+Depends: $\{shlib:Depends\}, $\{misc:Depends\}
+Description: {$description}
+ .
+ Part of the lintian test suite.
diff --git a/t/templates/skel/debian/copyright b/t/templates/skel/debian/copyright
new file mode 100644
index 0000000..b780d82
--- /dev/null
+++ b/t/templates/skel/debian/copyright
@@ -0,0 +1,21 @@
+This is part of the testsuite of lintian. See the file debian/copyright
+in the lintian source directory for more details.
+
+So far as it is copyrightable at all, this template is
+ Copyright © 2008 Frank Lichtenheld <djpig@debian.org>
+
+This program is free software; you may redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2, or (at your option)
+any later version.
+
+This is distributed in the hope that it will be useful, but without
+any warranty; without even the implied warranty of merchantability or
+fitness for a particular purpose. See the GNU General Public License
+for more details.
+
+A copy of the GNU General Public License version 2 is available as
+/usr/share/common-licenses/GPL-2 in the Debian GNU/Linux distribution
+or at http://www.gnu.org/licenses/old-licenses/gpl-2.0.html.
+You can also obtain it by writing to the Free Software Foundation, Inc.,
+51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA.
diff --git a/t/templates/skel/debian/rules b/t/templates/skel/debian/rules
new file mode 100755
index 0000000..cbe925d
--- /dev/null
+++ b/t/templates/skel/debian/rules
@@ -0,0 +1,3 @@
+#!/usr/bin/make -f
+%:
+ dh $@
diff --git a/t/tests/0000_basic.desc b/t/tests/0000_basic.desc
new file mode 100644
index 0000000..96d9f45
--- /dev/null
+++ b/t/tests/0000_basic.desc
@@ -0,0 +1,2 @@
+Testname: basic
+Version: 1.0
diff --git a/t/tests/0001_basic-non-native.desc b/t/tests/0001_basic-non-native.desc
new file mode 100644
index 0000000..e17a229
--- /dev/null
+++ b/t/tests/0001_basic-non-native.desc
@@ -0,0 +1,3 @@
+Testname: basic-non-native
+Type: non-native
+Version: 1.0
diff --git a/t/tests/basic-non-native/tags b/t/tests/basic-non-native/tags
new file mode 100644
index 0000000..e69de29
diff --git a/t/tests/basic-non-native/upstream/README b/t/tests/basic-non-native/upstream/README
new file mode 100644
index 0000000..e845566
--- /dev/null
+++ b/t/tests/basic-non-native/upstream/README
@@ -0,0 +1 @@
+README
diff --git a/t/tests/basic/debian/README b/t/tests/basic/debian/README
new file mode 100644
index 0000000..e845566
--- /dev/null
+++ b/t/tests/basic/debian/README
@@ -0,0 +1 @@
+README
diff --git a/t/tests/basic/tags b/t/tests/basic/tags
new file mode 100644
index 0000000..e69de29
--
1.5.6.3
--
Frank Lichtenheld <djpig@debian.org>
www: http://www.djpig.de/
Reply to: