Re: RFC: Ideas around pkg-perl-autopkgtest
On Fri, Aug 29, 2014 at 01:11:18PM +0200, Axel Beckert wrote:
> I wondered how we could keep track properly which package uses which
> test in pkg-perl-autopkgtest and one (still rough) idea is to use one
> virtual package per included test.
The problem here seems to be that all packages will need updates to their
test control files when tests in pkg-perl-autopkgtest change or new tests
are added. This clearly doesn't scale for the 3000 pkg-perl packages, and
adding ways to track which ones need changes is a workaround at best IMO.
While we're shortcutting the scaling problem altogether for now with
Antonio setting up ci.debian.net for all the pkg-perl packages in one go,
I think we should keep a longer term approach in mind too.
It seems to me that generic tests can be grouped by their dependencies:
there are tests that need to have only the minimal runtime or build
dependencies installed for the tests to actually be effective, and then
there are tests relying on external machinery like adequate to work.
I propose something like this:
Package: libfoo-bar-perl # this is what needs to go to all the packages
debian/tests/control:
Tests: pkg-perl-with-build-deps
Depends: @, @builddeps@, pkg-perl-autopkgtest
Test-Command: /usr/share/pkg-perl-autopkgtest/runner build-deps
Tests: pkg-perl-with-runtime-deps
Depends: @, pkg-perl-autopkgtest
Test-Command: /usr/share/pkg-perl-autopkgtest/runner runtime-deps
Tests: pkg-perl-with-heavy-deps
Depends: @, pkg-perl-autopkgtest, pkg-perl-autopkgtest-heavy
Test-Command: /usr/share/pkg-perl-autopkgtest/runner heavy-deps
Package: pkg-perl-autopkgtest
Depends: perl # and not much else
/usr/share/pkg-perl-autopkgtest/runner:
# looks at the argument and runs corresponding tests; simplified:
for t in /usr/share/pkg-perl-autopkgtest/tests/$1.d; do $t; done
/usr/share/pkg-perl-autopkgtest/tests/build-deps.d/smoke:
# the script we have now: copy t/ somewhere else and run prove -v on that
/usr/share/pkg-perl-autopkgtest/tests/runtime-deps.d/compile
# the script we have now: run perl -c on all the .pm files of this package
# more tests can be added later
Package: pkg-perl-autopkgtest-heavy
Depends: adequate, checker1, checker2
/usr/share/pkg-perl-autopkgtest/tests/heavy-deps.d/adequate:
# the script we have now: run adequate on the package
# more tests can be added later
The idea here is that the test control files in all the packages are
generic enough to "never" (heh) need changing. Even if that turns out
to be too optimistic, we're no worse off than we are now if we start
hardcoding just the existing tests and their dependencies in all packages.
I can see problems if we come up with a generic test that needs
for instance some restrictions. A really generic test control file
would have those as well, but this leads to a combinatoric explosion.
It's unfortunate that there's no 'include' functionality in the test
control file. It seems like having something like that would solve the
scaling problem quite neatly. Not sure if it's too late to add that to
the spec.
--
Niko Tyni ntyni@debian.org
Reply to: