On Jan 16, 2017, at 05:45 PM, Russ Allbery wrote: >autopkgtest is useful for adding additional tests of the built binaries, >but I don't believe it's intended as a replacement for build-time testing. >Maybe I've missed something? No, I think you're exactly right. If an upstream provides unit tests, those are totally appropriate to run at build time -and to fail the build if they fail- but may not be appropriate to run in autopkgtest. autopkgtests should be reserved for larger suitability tests on the built and installed package. An example might be a Python library's test suite. It makes sense to run these at build time because that's usually when upstream will run them (i.e. during development of the package). But since the test suite usually isn't run on a built package, it shouldn't be autopkgtested. The environment for build tests and autopkgtests are importantly different, e.g. the former does not/should not allow access to the internet while the latter can and sometimes must. A good example of an autopkgtest would be an import test for a Python module, i.e. once the package is built and installed, does it import? In fact, autodep8 will automatically add import tests for Python modules in a safe way (by cd'ing to a temporary directory first). There are occasionally good reasons why an upstream's test suite can't be run at build-time, and in those few cases I will run them in an autopkgtest. But generally, I think the two are there to test different aspects or lifecycles of the package. Cheers, -Barry
Attachment:
pgpCiUh1SX7NP.pgp
Description: OpenPGP digital signature