Re: package testing, autopkgtest, and all that
Hi Ian,
First a brief question:
> The source package provides a test metadata file
> debian/tests/control. This is a file containing zero or more
> RFC822-style stanzas, along these lines:
Do you still have somewhere that awk package demo package which had
debian/tests/ ? Currently our archive does not carry any package having
debian/tests/ (unfortunately).
Just a brief summary for such a ignorant DDs as myself who did not know about
those additional projects in Ubuntuland before they got mentioned in this
thread:
Ubuntu "Testing Automation" project has been going on for a while:
https://wiki.ubuntu.com/Testing/Automation , with active (i.e. actually being
developed/used) components such as
- checkbox (package in Ubuntu): user-land tool (packaged and available
in Ubuntu) to provide primarily hardware testing, with some basic software
tests available; reporting back to launchpad.net with gathered information on
hardware.
- mago (package in Ubuntu): built on top of LDTP (package in Debian), provides
testing of GNOME GUI.
- qa-regression-testing: collection of unit and regression tests for various
components (from kernel to xine and apache) of the systems.
Is not designed for distribution to the users (yet?)
All of the above approaches seems to separate testing "materials" from the
actual packages. Both mago and checkbox come with user interfaces, thus
enabling users to test/validate their own systems without requiring setting up
any virtual environment. And they ship their tests along (there seems to be
some discovery mechanisms but I haven't checked in detail yet).
On the other hand, Ian's autopkgtest aims to provide a unified testing
framework built around packages, so that it becomes possible for us,
maintainers, to equip packages with necessary tests batteries; which could be
reused by others (e.g. QA teams). So it seems to go closer toward the goals of
qa-regression-testing project above, but tries to scale via making tests
materials provided by the corresponding maintainers in the source
packages. (As is now at least) it requires relatively complex setup of the
system to start crunching the tests batteries, thus not user-oriented.
For our purposes of testing scientific applications, as it was previously
mentioned and exposed in the "case studies" on
http://neuro.debian.net/proj_debtest.html page, we want the cocktail of all the
above solutions ;-) with less accent on GUI testing but with convenient
user-interface not requiring root access (besides possibly installation
of the tests batteries). And it seems that the autopkgtest basis, maybe with
some functionality from checkbox and mago (e.g. for collecting relevant
software/hardware statistics and running basic system tests) could unroll into
the ultimate solution for our goals.
Unfortunately the core aspect of the current autopkgtest - relying on tests in
source packages -- imho to be not the ideal solution to target both sides
of the userbase (i.e. maintainers/QA vs mortals).
https://wiki.ubuntu.com/AutomatedTesting associated with autopkgtest
addresses the FAQ on why source packages:
,---
| Q. Why put the tests in the source package ?
|
| A. Other possibilities include a special .deb generated by the source
| (which is a bit strange and what happens to this .deb and will make it even
| harder to reuse upstream test suites), or putting them in the .deb to be tested
| (definitely wrong - most people won't want the tests and they might be very
| large) or having them floating about separately somewhere (which prevents us
| from sharing and exchanging tests with other parts of the Free Software
| community). The source package is always available when development is taking
| place.
`---
Ian -- could you please unroll your arguments a bit? I still do not see why
source packages are beneficial besides build-time testing (which we often do
already without any additional framework)
In our view ideal solution from the user's (or maintainer/QA) point of
view should involve exactly that -- interaction with binary packages since that
is what everyone knows how to deal with, e.g.:
sudo apt-get install apache2-tests
adt-run apache2
or even, having installed X tests batteries
adt-run --all # to run all "installed" tests batteries
We could also complement it with
sudo adt-install --all-relevant
which would install test batteries complementing installed software
packages, thus providing tailored coverage for a particular user needs.
To accomplish above with tests only in source packages, would require apt-src
like infrastructure (and do version/dependencies tracking on them), and
moreover why should I care to keep sources on my (user's) system at all! What
is relevant for testing seems to be: testing materials and already installed
applications. And that is what other (checkbox/mago/qa-regression-testing)
frameworks rely upon.
And sure thing, for maintainers/QA it could get more evolved indeed requiring
adt-run --all-available --- virt-server [virt-server-arg...]
so that testing could be performed on a distant or virtualized environment.
So, where do we start/continue sharing the thoughts on a tentative DEP? ;)
--
=------------------------------------------------------------------=
Keep in touch www.onerussian.com
Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
Reply to: