Re: Proto-type britney test suite
On 15/10/2011 12:20, Niels Thykier wrote:
>
> The primary issue with this suite is that you have to hand-craft *all*
> the data needed, which is a pain. I strongly suspect that the long
> term implications of this is that tests will end up being very hard to
> understand and that people will generally avoid writing tests.
>
I guess that's why you added "description" files for (almost) each test?
I think it is enough to not forget why we added $some test.
> The "second" problem is that the test-suite only checks the heidi
> result, so it is cannot tell the difference between "pkg with rc bugs"
> and "migrating pkgX would break pkgY".
>
But we do have (thanks to you) minimal tests for those two situations. So,
somehow, they are distinguishable. But, I agree it could be a problem with
more complicated tests.
>
> For now, feel free to give the test suite a spin. Either clone it from
> [1] or run it from franck [2]. If you clone it, have a look at the
> README (there are no "-h" or "--help").
>
I didn't find time to play with it enough yet but I find this project very
nice. It helps (some of) us to understand better what britney does in some
specific situations, serves as a regression test base and tells new briney
implementations what they should implement (at least).
I have some minor patches:
#1) fixes a typo.
#2) makes run-single-test exit with a non zero value when test fails
(the change is not pretty but kinda wfm™) Ideally, run-single-test should
also tell us why the test failed (maybe by `cat diff`?).
#) a bug in britney fixed by Adam recently (that requires disabling
compatible mode). But the test passes using the buggy britney… so maybe I
did something wrong.
Kind regards,
--
Mehdi Dogguy مهدي الدڤي
http://dogguy.org/
>From 85adbdee956c8cd966c8570c502b8f03200d82d0 Mon Sep 17 00:00:00 2001
From: Mehdi Dogguy <mehdi@debian.org>
Date: Tue, 18 Oct 2011 11:50:59 +0200
Subject: [PATCH 1/3] Fix typo
---
bin/runtests.pl | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/bin/runtests.pl b/bin/runtests.pl
index c71e838..4c6b48b 100755
--- a/bin/runtests.pl
+++ b/bin/runtests.pl
@@ -64,7 +64,7 @@ foreach my $t (@tests) {
print "$res\n";
}
-print "\nSummery:\n";
+print "\nSummary:\n";
print 'Ran ' . scalar (@tests) . " tests\n";
print "Failed tests: $failed\n";
exit $failed ? 1 : 0;
--
1.7.2.5
>From 9f9adfbf200f8bca7152af4f7fce79da73b09f22 Mon Sep 17 00:00:00 2001
From: Mehdi Dogguy <mehdi@debian.org>
Date: Tue, 18 Oct 2011 14:05:34 +0200
Subject: [PATCH 2/3] Use exit code returned by method 'run'
---
bin/run-single-test.pl | 5 ++---
1 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/bin/run-single-test.pl b/bin/run-single-test.pl
index 832ff60..154e615 100755
--- a/bin/run-single-test.pl
+++ b/bin/run-single-test.pl
@@ -16,7 +16,6 @@ unless ( -d $RUNDIR ) {
my $bt = BritneyTest->new ({}, "$RUNDIR/$test", "$TESTSET/$test");
$bt->setup;
-$bt->run ($britney);
-
-exit 0;
+my $result = $bt->run ($britney);
+exit (1 - $result);
--
1.7.2.5
>From 114a7058cb1aa7ad1643400af332b31bdddcb1d5 Mon Sep 17 00:00:00 2001
From: Mehdi Dogguy <mehdi@debian.org>
Date: Tue, 18 Oct 2011 11:47:27 +0200
Subject: [PATCH 3/3] Add regression test basic-removed-binary-package
---
t/basic-removed-binary-package/description | 10 +++++++++
t/basic-removed-binary-package/expected | 3 ++
.../var/data/testing/Dates | 1 +
.../var/data/testing/Hints | 1 +
.../var/data/testing/Packages_i386 | 21 ++++++++++++++++++++
.../var/data/testing/Sources | 6 +++++
.../var/data/testing/Urgency | 1 +
.../var/data/unstable/Hints | 1 +
.../var/data/unstable/Packages_i386 | 13 ++++++++++++
.../var/data/unstable/Sources | 6 +++++
10 files changed, 63 insertions(+), 0 deletions(-)
create mode 100644 t/basic-removed-binary-package/description
create mode 100644 t/basic-removed-binary-package/expected
create mode 100644 t/basic-removed-binary-package/hints/test-hints
create mode 100644 t/basic-removed-binary-package/input/FauxPackages
create mode 100644 t/basic-removed-binary-package/var/data/testing-proposed-updates/BugsV
create mode 100644 t/basic-removed-binary-package/var/data/testing-proposed-updates/Packages_i386
create mode 100644 t/basic-removed-binary-package/var/data/testing-proposed-updates/Sources
create mode 100644 t/basic-removed-binary-package/var/data/testing/BugsV
create mode 100644 t/basic-removed-binary-package/var/data/testing/Dates
create mode 120000 t/basic-removed-binary-package/var/data/testing/Hints
create mode 100644 t/basic-removed-binary-package/var/data/testing/Packages_i386
create mode 100644 t/basic-removed-binary-package/var/data/testing/Sources
create mode 100644 t/basic-removed-binary-package/var/data/testing/Urgency
create mode 100644 t/basic-removed-binary-package/var/data/unstable/BugsV
create mode 120000 t/basic-removed-binary-package/var/data/unstable/Hints
create mode 100644 t/basic-removed-binary-package/var/data/unstable/Packages_i386
create mode 100644 t/basic-removed-binary-package/var/data/unstable/Sources
diff --git a/t/basic-removed-binary-package/description b/t/basic-removed-binary-package/description
new file mode 100644
index 0000000..59e9cdb
--- /dev/null
+++ b/t/basic-removed-binary-package/description
@@ -0,0 +1,10 @@
+Regression test (seen with fbreader)
+
+This is a regression test. The situation has been experienced with fbreader:
+src:fbreader dropped one of its binary packages in sid (say bin:A) but the
+other binary package bin:fbreader had a dependency on bin:A in testing only.
+This led britney2 to think that fbreader could not be removed without increasing
+the uninstallability count, which is false since the newer bin:fbreader dropped
+its dependency on bin:A in sid.
+
+This bug has been fixed in britney2.git:94071b16491e90d78605791a316141b58cc3c6fc
diff --git a/t/basic-removed-binary-package/expected b/t/basic-removed-binary-package/expected
new file mode 100644
index 0000000..9b0ddc1
--- /dev/null
+++ b/t/basic-removed-binary-package/expected
@@ -0,0 +1,3 @@
+fbreader 0.10.7dfsg-5 i386 lib
+libzltext0.10 0.10.7dfsg-5 i386 lib
+fbreader 0.10.7dfsg-5 source lib
diff --git a/t/basic-removed-binary-package/hints/test-hints b/t/basic-removed-binary-package/hints/test-hints
new file mode 100644
index 0000000..e69de29
diff --git a/t/basic-removed-binary-package/input/FauxPackages b/t/basic-removed-binary-package/input/FauxPackages
new file mode 100644
index 0000000..e69de29
diff --git a/t/basic-removed-binary-package/var/data/testing-proposed-updates/BugsV b/t/basic-removed-binary-package/var/data/testing-proposed-updates/BugsV
new file mode 100644
index 0000000..e69de29
diff --git a/t/basic-removed-binary-package/var/data/testing-proposed-updates/Packages_i386 b/t/basic-removed-binary-package/var/data/testing-proposed-updates/Packages_i386
new file mode 100644
index 0000000..e69de29
diff --git a/t/basic-removed-binary-package/var/data/testing-proposed-updates/Sources b/t/basic-removed-binary-package/var/data/testing-proposed-updates/Sources
new file mode 100644
index 0000000..e69de29
diff --git a/t/basic-removed-binary-package/var/data/testing/BugsV b/t/basic-removed-binary-package/var/data/testing/BugsV
new file mode 100644
index 0000000..e69de29
diff --git a/t/basic-removed-binary-package/var/data/testing/Dates b/t/basic-removed-binary-package/var/data/testing/Dates
new file mode 100644
index 0000000..da362e6
--- /dev/null
+++ b/t/basic-removed-binary-package/var/data/testing/Dates
@@ -0,0 +1 @@
+fbreader 0.10.7dfsg-5 15232
diff --git a/t/basic-removed-binary-package/var/data/testing/Hints b/t/basic-removed-binary-package/var/data/testing/Hints
new file mode 120000
index 0000000..c641a30
--- /dev/null
+++ b/t/basic-removed-binary-package/var/data/testing/Hints
@@ -0,0 +1 @@
+../../../hints/
\ No newline at end of file
diff --git a/t/basic-removed-binary-package/var/data/testing/Packages_i386 b/t/basic-removed-binary-package/var/data/testing/Packages_i386
new file mode 100644
index 0000000..4ca3cd0
--- /dev/null
+++ b/t/basic-removed-binary-package/var/data/testing/Packages_i386
@@ -0,0 +1,21 @@
+Package: fbreader
+Architecture: i386
+Depends: libzltext0.10 (>= 0.10.7dfsg-4), libzlui-maemo (>= 0.10.7dfsg-4)
+Priority: optional
+Section: lib
+Version: 0.10.7dfsg-4
+
+Package: libzlui-maemo
+Architecture: i386
+Depends: libzltext0.10 (= 0.10.7dfsg-4)
+Priority: optional
+Section: lib
+Source: fbreader
+Version: 0.10.7dfsg-4
+
+Package: libzltext0.10
+Architecture: i386
+Priority: optional
+Section: lib
+Source: fbreader
+Version: 0.10.7dfsg-4
diff --git a/t/basic-removed-binary-package/var/data/testing/Sources b/t/basic-removed-binary-package/var/data/testing/Sources
new file mode 100644
index 0000000..331fb80
--- /dev/null
+++ b/t/basic-removed-binary-package/var/data/testing/Sources
@@ -0,0 +1,6 @@
+Package: fbreader
+Binary: fbreader, libzlui-maemo, libzltext0.10
+Maintainer: The R-Team <debian-release@lists.debian.org>
+Standards-Version: 3.9.2
+Section: lib
+Version: 0.10.7dfsg-4
diff --git a/t/basic-removed-binary-package/var/data/testing/Urgency b/t/basic-removed-binary-package/var/data/testing/Urgency
new file mode 100644
index 0000000..bb2ff17
--- /dev/null
+++ b/t/basic-removed-binary-package/var/data/testing/Urgency
@@ -0,0 +1 @@
+fbreader 0.10.7dfsg-4 low
diff --git a/t/basic-removed-binary-package/var/data/unstable/BugsV b/t/basic-removed-binary-package/var/data/unstable/BugsV
new file mode 100644
index 0000000..e69de29
diff --git a/t/basic-removed-binary-package/var/data/unstable/Hints b/t/basic-removed-binary-package/var/data/unstable/Hints
new file mode 120000
index 0000000..c641a30
--- /dev/null
+++ b/t/basic-removed-binary-package/var/data/unstable/Hints
@@ -0,0 +1 @@
+../../../hints/
\ No newline at end of file
diff --git a/t/basic-removed-binary-package/var/data/unstable/Packages_i386 b/t/basic-removed-binary-package/var/data/unstable/Packages_i386
new file mode 100644
index 0000000..905173e
--- /dev/null
+++ b/t/basic-removed-binary-package/var/data/unstable/Packages_i386
@@ -0,0 +1,13 @@
+Package: fbreader
+Architecture: i386
+Depends: libzltext0.10 (>= 0.10.7dfsg-4)
+Priority: optional
+Section: lib
+Version: 0.10.7dfsg-5
+
+Package: libzltext0.10
+Architecture: i386
+Priority: optional
+Section: lib
+Source: fbreader
+Version: 0.10.7dfsg-5
diff --git a/t/basic-removed-binary-package/var/data/unstable/Sources b/t/basic-removed-binary-package/var/data/unstable/Sources
new file mode 100644
index 0000000..e97ec04
--- /dev/null
+++ b/t/basic-removed-binary-package/var/data/unstable/Sources
@@ -0,0 +1,6 @@
+Package: fbreader
+Binary: fbreader, libzltext0.10
+Maintainer: The R-Team <debian-release@lists.debian.org>
+Standards-Version: 3.9.2
+Section: lib
+Version: 0.10.7dfsg-5
--
1.7.2.5
Reply to: