Re: package testing, autopkgtest, and all that
On Wed, Feb 02, 2011 at 04:13:37PM +0000, Ian Jackson wrote:
> Yaroslav Halchenko writes ("Re: package testing, autopkgtest, and all that"):
> > in the core -- users usually do not deal with source packages; many of
> > them do not even have deb-src lines for apt. They do not care how
> > things are built, but if we want them to contribute by testing their
> > systems, the simplest approach is exposing test batteries as binary
> > packages. Of cause, user-friendly front-end might overcome those
> > difficulties even if tests are provided in source packages.
>
> I don't understand what your usage model is. Are you expecting random
> users to execute the tests ? Why would they do that ? What kind of
> useful outcome is this likely to produce ?
We expect any person that is interested or asked to run test to do it
and be able to do it. They would do that because they need to be sure
that things work as intended and already do that in many cases. A common
use case are scientific analysis toolkits: We have seen them break due
to a change in a shared lib in Debian -- a change that upstream doesn't
see, because they link statically against ancient and "approved"
versions.
There are various labs that are very interested in verifying that
"random" library updates don't break their systems, which can happen
with any update.
If "random user" clicks the TestMyDebianSystem "button" the outcome
might be a passed test and everybody is happy, or a failure with a log.
> I think that someone who doesn't have a "deb-src" line in their
> sources.list, and has no knowledge of the existence of source
> packages, is very unlikely to produce a useful bug report which leads
> to an improvement to the software.
Think about a dashboard that collects failures and passes. Even if a
single user cannot produce a good report, you'll know that something is
wrong if suddenly many machines report trouble.
But to get many machines, we need to make it dead-simple to participate
in this type of croud-testing. We can have GUI frontends to let people
do specific tests, or offer "backfill" job configurations for compute
clusters.
> Making test suites highly end-user-visible is simply likely to result
> in a lot of noise.
Higher noise, but more samples -- I'm not sure what offers the best
estimate of the actual status.
I see the point in having less by better-quality input to package
maintainers, but again, the test results do not have to go one-by-one to
a human to inspect.
> > for the goal of testing current system setup, installing the single,
> > most recent battery, sounds sufficient. To complement there are
> > snapshot.debian.org and backports.debian.org, so any previous or
> > backported version could be made available
>
> Our mechanisms for downloading and installing specific binary packages
> from different source are not very well-developed.
Maybe that is something we should work on in this context.
Michael
--
Michael Hanke
http://mih.voxindeserto.de
Reply to: