On Mar 01, 2016, at 09:05 AM, Martin Pitt wrote: >Very nice! There's precedent for Perl, Ruby and DKMS packages which >have a fairly standard way to run the upstream test suite. For Python >there are some conventions, like "./setup.py test" or running >nosetests, maybe it's worth experimenting with those after getting the >initial "import works" smoketest in place. > >This situation (testing many related packages in a similar way) is >what autodep8 is for. When run in a Debian source package tree, it >checks if it knows something about this type of package, and if so, >synthesizes a debian/tests/control. Have a look at I didn't know about autodep8, thanks for the link! I gather that it's supposed to be used by the maintainer in a source package ($vcs repo) to jumpstart a debian/tests directory? But after that, would you use autodep8 again? I'll try to clear some time to look at adding Python support; shouldn't be too difficult I think. I'm not sure how useful things like "setup.py test" are for DEP-8. In general I think a package's test suite are best exercised during package build[*]. For the majority of Python packages, I view DEP-8 as ensuring that the user experience is what they expect, and that means at a minimum that the packages import and Do Something Simple. A generic proof of non-broken import can mean printing the package's version or the module path, and that's something that autopkg8 could templatize. Anything more than that requires some more detailed knowledge about what the package does. I've tended to taking a pretty dumb approach since I like using Test-Command, but Ben's smoketests in lockfile are certainly more thorough. For an autodep8 plugin, I'd probably fall somewhere in between. >Not sure what you mean with "shortcoming", you can put arbitrary shell >code into Test-Command:. However, the spirit is that these should be >very short, usually one line. I suggest that they should mostly just >call something like "/usr/share/python-something/run-autopkgtest $py $pkgname" >and you maintain the actual test code centrally in e. g. >python-defaults. Oh, that's a good idea. Then when we come up with better generic tests, we only need to update one package, not everything. >That seems like an excellent starting point indeed. Making sure that >all modules are importable will catch missing builds for particular >Python versions, regressions in newer Python versions, missing >dependencies, packaging errors, etc. +1 >As said above, maybe in the future we want to try and run upstream test >suites, but this will need a lot more heuristics (starting with additional >test deps). I'm skeptical about running the upstream test suite in DEP-8, but baby steps! Cheers, -Barry [*] Although it's true for a small handful of packages, it's not feasible to run their test suite during package build for various reasons. In those cases I've sometimes added DEP-8 tests for running the test suite, but I've also found those to be rather fragile, so I'm on the fence about gating e.g. package promotion based on their passing their own test suite.
Attachment:
pgpVCJLD2CnJ7.pgp
Description: OpenPGP digital signature