Re: package testing, autopkgtest, and all that

[Ian Jackson <ijackson@chiark.greenend.org.uk>]

Michael Hanke writes ("Re: package testing, autopkgtest, and all that"):
> But to get many machines, we need to make it dead-simple to participate
> in this type of croud-testing. We can have GUI frontends to let people
> do specific tests, or offer "backfill" job configurations for compute
> clusters.
> 
> > Making test suites highly end-user-visible is simply likely to result
> > in a lot of noise.
> 
> Higher noise, but more samples -- I'm not sure what offers the best
> estimate of the actual status.

Maintainers don't want large amounts of low-quality information.  That
is very difficult to use for fixing bugs.

> I see the point in having less by better-quality input to package
> maintainers, but again, the test results do not have to go one-by-one to
> a human to inspect.

We don't have any infrastructure for dealing with this kind of
information in bulk.  I think that what you are proposing is a
different kind of thing to autopkgtest and it would be best for us to
deploy autopkgtest now as it already exists.

> There are various labs that are very interested in verifying that
> "random" library updates don't break their systems, which can happen
> with any update.

This is easily done with autopkgtest; the only difference from your
proposal is that the source package needs to be downloaded.  Doing so
is not difficult or troublesome, and can be done automatically.

Ian.