On Sat, 11 Oct 2014 15:30:27 +0200, intrigeri wrote:
> > 2. people uploading a package with a Testsuite header should try to
> > verify it works. This probably means setting up a chroot for adt-run
> > to do it properly; there are some pointers for this in autopkgtest.html.
> I'm a little bit concerned about this one. I'm pretty sure I can live
> with it, but it's still probably worth raising this point: this
> requirement adds manual steps to the "updating a package" workflow.
Correct.
> I'd like to question why humans need to do it by hand, while an army
> of robots, somewhere, are going to do it anyway.
Well, we also run lintian manually, or check if changes are needed
before updating Standards-Version :)
> I'm aware that the archive is not gated by the autopkgtest results
> (yet), which is probably a part of the answer to my question: we want
> to avoid uploading broken stuff to the archive, so for the time being,
> we have to run these tests ourselves before uploading.
And once these guards are in effect, we should be confident that
packages which declare to be autopkgtest-able actually are.
> OTOH, my understanding is that ci.d.n runs autopkgtests on every
> reverse-dependency whenever a given package is updated. This tests not
> only the updated package itself, but also how it impacts the rest of
> the archive. How I see it, I think it's, by far, the most important
> part of the entire exercise as far as our packages are concerned
Ack.
> (because I dare hoping that as long as the build-time test suite
> passes, in the vast majority of the cases, autopkgtests failure will
> indicate either bugs in the autopkgtests machinery, or changes in the
> test suite that require adapting that machinery, rather than actual
> bugs in the code).
That was my hope as well before we started to work on it :)
In practice it's not so easy, and quite fascinating to see in how
many ways autopkgtests can fail.
Niko has tuned pkg-perl-autopkgtest already in all kinds of fancy
ways to catch common traps, still what I see in the last few weeks
are all kinds of interesting failures:
- for smoke tests: tests requiring some files from the source
package; or tests failing under autopktest but not during build
- syntax.t and use.t have already caught missing runtime
dependencies, and also code errors (which are in code paths not
exercised by the buildtime tests)
> If my understanding is correct, running adt-run on
> an individual package before uploading is *not* testing this.
You're right that running them manually doesn't help to catch the
later breakage due to changed rdeps: but it guarantees that they at
least work initially :)
Of course the question is valid if we want to make sure the tests
work before adding them to d/control, or if we just fix them if we
get failure reports ...
> My secondary question is: have we got the glue (e.g. pbuilder hooks or
> similar) anywhere to *automatically* run these tests after building
> a package? (OK, it looks trivial to write once one has the right
> chroots ready, but still, might be suboptimal if half of us each write
> their own :)
I've added some lines to check-build, the post-build script carnil
and me (at least) are running after each build, also available (as an
example) in pkg-perl-tools. But I haven't pushed the changes yet
since carnil wanted to test them before :)
Anyway, here's the relevant code block, which boils down to running
one command (and some prompting around it):
#v+
if grep -q 'Testsuite: autopkgtest' debian/control && [ -x /usr/bin/schroot ] && schroot -l | grep -q default ; then
read -n 1 -p "adt-run? y/N " ADT
if [ "$ADT" = "y" ]; then
adt-run --changes $CHANGES --- schroot default
fi
fi
echo
#v-
Cheers,
gregor
--
.''`. Homepage: http://info.comodo.priv.at/ - OpenPGP key 0xBB3A68018649AA06
: :' : Debian GNU/Linux user, admin, and developer - http://www.debian.org/
`. `' Member of VIBE!AT & SPI, fellow of the Free Software Foundation Europe
`- NP: Pogues: London Girl
Attachment:
signature.asc
Description: Digital Signature