Re: package testing, autopkgtest, and all that
Yaroslav Halchenko writes ("Re: package testing, autopkgtest, and all that"):
> First a brief question:
> > The source package provides a test metadata file
> > debian/tests/control. This is a file containing zero or more
> > RFC822-style stanzas, along these lines:
>
> Do you still have somewhere that awk package demo package which had
> debian/tests/ ? Currently our archive does not carry any package having
> debian/tests/ (unfortunately).
Probably. I'll see if I can dig it out.
> All of the above approaches seems to separate testing "materials"
> from the actual packages. Both mago and checkbox come with user
> interfaces, thus enabling users to test/validate their own systems
> without requiring setting up any virtual environment. And they ship
> their tests along (there seems to be some discovery mechanisms but I
> haven't checked in detail yet).
These tools are mostly orthogonal to and even complementary to
autopkgtest. There are a whole bunch of other automatic testing
utilities like xnee, etc., which solve other parts of the automatic
testing problem and can be usefully combined with autopkgtest.
I wasn't aware of "qa-regression-testing":
> - qa-regression-testing: collection of unit and regression tests for
> various components (from kernel to xine and apache) of the
> systems. Is not designed for distribution to the users (yet?)
That seems to want to collect the tests in one central place. Debian
(and to an extent our derivatives) have some difficulty with
centralised collections of information about multiple different pieces
of software. Our social and technical structures are based around
more-or-less freestanding packages.
So it makes more sense in our context to locate tests with the
package.
> On the other hand, Ian's autopkgtest aims to provide a unified testing
> framework built around packages, so that it becomes possible for us,
> maintainers, to equip packages with necessary tests batteries; which could be
> reused by others (e.g. QA teams).
Yes.
> Unfortunately the core aspect of the current autopkgtest - relying
> on tests in source packages -- imho to be not the ideal solution to
> target both sides of the userbase (i.e. maintainers/QA vs mortals).
I'm not sure I know what you mean.
> ,---
> | Q. Why put the tests in the source package ?
> |
> | A. Other possibilities include a special .deb generated by the
> | source (which is a bit strange and what happens to this .deb and
> | will make it even harder to reuse upstream test suites), or
> | putting them in the .deb to be tested (definitely wrong - most
> | people won't want the tests and they might be very large) or
> | having them floating about separately somewhere (which prevents us
> | from sharing and exchanging tests with other parts of the Free
> | Software community). The source package is always available when
> | development is taking place.
> `---
>
> Ian -- could you please unroll your arguments a bit? I still do not
> see why source packages are beneficial besides build-time testing
> (which we often do already without any additional framework)
You're asking why the tests should be in source packages rather than
binary packages.
The underlying point you seem to be missing is that a major aim of
autopkgtest is that it should be straightforward to provide
autopkgtest bindings for existing upstream test suites.
So, firstly, binary packages have a much bigger overhead, and more
complicated restrictions, compared to adding new material to source
packages:
- Binary packages get entries in a large number of databases both
in our central infrastructure and on users' systems
- Binary packages are highly visible in ways that test infrastructure
doesn't need to be
- They are relatively complicated to produce
- They can only be installed in particular locations
- Only one version can be present on a system at once (unless even
more complicated things are done)
The additional complexity and effort doesn't buy us anything, and the
restrictions are pointless.
Secondly: very often the tests will want to use other material from
the source package. Upstream packages often come with tests, or
strange example programs, or other material, which is useful for
testing but not useful for users, and which the upstream package does
not install.
So if the tests were in binary packages, often we'd have to construct
a weird binary package which contained all or part of the built source
tree. This would be very ugly and also bulky.
And additional violence would have to be done to the upstream test
suite to try to make it work when "installed", rather than when run
out of the source tree as it expects. This is probably going to be
much harder than simply arranging for the tests to test the installed
rather than built copy (if they don't already).
Thirdly: having the tests in a source package makes it easier to run
one version of the tests against a different version of the package.
This is because we typically have different distribution channels and
configuration for the sources of source packages, but also because
there is a good chance that the dependencies are likely to be more
relaxed.
It would be quite reasonable to run the tests from unstable on the
code in testing. That gets you quicker updates to the tests, and is
something I definitely wanted to support.
Fourthly: doing it this way makes the whole automated testing system
more easily shareable with upstream and with non-Debian-derived
distros. A different distro can use different metadata, but we can
share the underlying tests. This is less true if our tests involve
messing with "make install" and extra binary packages.
So overall, it's easier this way.
To counter your objections:
> [with tests in binary packages one could do]
> sudo apt-get install apache2-tests
> adt-run apache2
We already have a good architecture and good machinery for installing
sources.
> To accomplish above with tests only in source packages, would
> require apt-src like infrastructure (and do version/dependencies
> tracking on them),
We already have all of that infrastructure. I didn't have to write
any substantial amount of new dependency-handling code.
> and moreover why should I care to keep sources on my (user's) system
> at all!
I think that users who want to do testing can reasonably be asked to
install the source code.
Ian.
Reply to: