On Sat, 11 Oct 2014 15:30:27 +0200, intrigeri wrote: > > 2. people uploading a package with a Testsuite header should try to > > verify it works. This probably means setting up a chroot for adt-run > > to do it properly; there are some pointers for this in autopkgtest.html. > I'm a little bit concerned about this one. I'm pretty sure I can live > with it, but it's still probably worth raising this point: this > requirement adds manual steps to the "updating a package" workflow. Correct. > I'd like to question why humans need to do it by hand, while an army > of robots, somewhere, are going to do it anyway. Well, we also run lintian manually, or check if changes are needed before updating Standards-Version :) > I'm aware that the archive is not gated by the autopkgtest results > (yet), which is probably a part of the answer to my question: we want > to avoid uploading broken stuff to the archive, so for the time being, > we have to run these tests ourselves before uploading. And once these guards are in effect, we should be confident that packages which declare to be autopkgtest-able actually are. > OTOH, my understanding is that ci.d.n runs autopkgtests on every > reverse-dependency whenever a given package is updated. This tests not > only the updated package itself, but also how it impacts the rest of > the archive. How I see it, I think it's, by far, the most important > part of the entire exercise as far as our packages are concerned Ack. > (because I dare hoping that as long as the build-time test suite > passes, in the vast majority of the cases, autopkgtests failure will > indicate either bugs in the autopkgtests machinery, or changes in the > test suite that require adapting that machinery, rather than actual > bugs in the code). That was my hope as well before we started to work on it :) In practice it's not so easy, and quite fascinating to see in how many ways autopkgtests can fail. Niko has tuned pkg-perl-autopkgtest already in all kinds of fancy ways to catch common traps, still what I see in the last few weeks are all kinds of interesting failures: - for smoke tests: tests requiring some files from the source package; or tests failing under autopktest but not during build - syntax.t and use.t have already caught missing runtime dependencies, and also code errors (which are in code paths not exercised by the buildtime tests) > If my understanding is correct, running adt-run on > an individual package before uploading is *not* testing this. You're right that running them manually doesn't help to catch the later breakage due to changed rdeps: but it guarantees that they at least work initially :) Of course the question is valid if we want to make sure the tests work before adding them to d/control, or if we just fix them if we get failure reports ... > My secondary question is: have we got the glue (e.g. pbuilder hooks or > similar) anywhere to *automatically* run these tests after building > a package? (OK, it looks trivial to write once one has the right > chroots ready, but still, might be suboptimal if half of us each write > their own :) I've added some lines to check-build, the post-build script carnil and me (at least) are running after each build, also available (as an example) in pkg-perl-tools. But I haven't pushed the changes yet since carnil wanted to test them before :) Anyway, here's the relevant code block, which boils down to running one command (and some prompting around it): #v+ if grep -q 'Testsuite: autopkgtest' debian/control && [ -x /usr/bin/schroot ] && schroot -l | grep -q default ; then read -n 1 -p "adt-run? y/N " ADT if [ "$ADT" = "y" ]; then adt-run --changes $CHANGES --- schroot default fi fi echo #v- Cheers, gregor -- .''`. Homepage: http://info.comodo.priv.at/ - OpenPGP key 0xBB3A68018649AA06 : :' : Debian GNU/Linux user, admin, and developer - http://www.debian.org/ `. `' Member of VIBE!AT & SPI, fellow of the Free Software Foundation Europe `- NP: Pogues: London Girl
Attachment:
signature.asc
Description: Digital Signature