Re: package testing, autopkgtest, and all that
Hi Ian,
thanks for the input! It does make motivation behind source
packages-based testing clearer. And Simon's example is a good one ;-)
As a summary: source packages-based testing often provides more
convenient and upstream-friendly approach, thus it must not be
"excluded", echoing Stefano's comment that ideally we might want having
both (source and binary packages testing).
Moreover we might need a simple registry/specification on how
already existing packages could expose included test batteries (so
echoing back to debian/README.test): e.g. many python modules have
tests included and could easily be invoked with 'nosetests module' or
'python -c "import module; module.test()"', so we just need a
specification installed on how to run the tests (and possibly common
output formats so our backend could pick them up, e.g. nosetests,
py.test, ctest, etc).
As such, those packages do not need separate binary package, nor
source package for testing. or alternatively -- they are already
'testing' binary packages without clear specification on how to invoke
the tests and collect the results.
> > Unfortunately the core aspect of the current autopkgtest - relying
> > on tests in source packages -- imho to be not the ideal solution to
> > target both sides of the userbase (i.e. maintainers/QA vs mortals).
> I'm not sure I know what you mean.
in the core -- users usually do not deal with source packages; many of
them do not even have deb-src lines for apt. They do not care how
things are built, but if we want them to contribute by testing their
systems, the simplest approach is exposing test batteries as binary
packages. Of cause, user-friendly front-end might overcome those
difficulties even if tests are provided in source packages.
Once again, many arguments behind source packages-based testing are
sound to me, but I must disagree with many statements in the "bigger
overhead" argument:
> - Binary packages get entries in a large number of databases both
> in our central infrastructure and on users' systems
and imho it is ok -- we already have -dbg packages which are also of
marginal importance to users, unless they need them, so they get
installed them explicitly
> - Binary packages are highly visible in ways that test infrastructure
> doesn't need to be
we (at least me and Michael) do want making them visible -- that is the
point. Otherwise they would not be exercised, and be left unknown (e.g.
like qa-regression-testing ;-) )
> - They are relatively complicated to produce
yes -- unless the opposite ;) i.e. for many packages, indeed, source
packages testing seems to be more adequate.
> - They can only be installed in particular locations
I do not see it much as a disadvantage
> - Only one version can be present on a system at once (unless even
> more complicated things are done)
for the goal of testing current system setup, installing the single,
most recent battery, sounds sufficient. To complement there are
snapshot.debian.org and backports.debian.org, so any previous or
backported version could be made available
--
=------------------------------------------------------------------=
Keep in touch www.onerussian.com
Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
Reply to: