On 2022-07-27 16 h 32, Julian Gilbey wrote:
On Wed, Jul 27, 2022 at 07:45:12PM +0100, Julian Gilbey wrote:On Wed, Jul 27, 2022 at 10:26:33AM -0400, Louis-Philippe Véronneau wrote:The way I see it: 1. We should have a Lintian tag for packages not using the new pybuild-autodep8 autopkgtest. It would be even better if this tag would be a pointed hint that identified 'manually' written unit test autopkgtests that could be replaced. This way, you get something like: python-foo source: not-using-pybuil-autodep8 [debian/tests/unittests] for python packages that have old 'manually' written unit test autopkgtests and: python-foo source: not-using-pybuild-autodep8 [no-autopkgtest] for python packages without any autopkgtest. 2. lintian-brush (or something else, but I think lintian-brush is the right tool) would go over these packages to: 2.1 Add the new autodep8 autopkgtests and build the package to see if they pass 2.2 Remove the "manual" unit test autopkgtests if 2.1 succeeds 2.3 Open a bug report if 2.1 failsI'd be wary about 2.2 and 2.3. I have several packages where I know that an automated test will fail; there are all sorts of weird cases where I've had to write tests manually. I would also be quite cross if manually crafted tests were automatically removed, especially in cases such as Simon mentioned where they do things that that automatically generated test does not do. Another thing I could imagine happening is that the automated test succeeds in a trivial way - it succeeds but doesn't actually test much because of the nature of the package. On the other hand, a bug report saying something like the following seems much more reasonable: "We've tested this package using the automated autopkgtest system and it seems to work by adding the line 'Testsuite: autopkgtest-pkg-pybuild'; please check that the automated tests cover all of the tests of your manually written debian/tests/* and if so, then please remove them. The autopkgtest-pkg-pybuild logs are attached." This would give the maintainer the chance to decide how best to proceed.Here's another alternative to steps 2.1-2.3 based on this: For packages which currently have manually-written autopkgtests: 2.A Try removing debian/tests and adding Testsuite: autopkgtest-pkg-pybuild to debian/control, then building the package and running autopkgtest. 2.B If this works, then submit a bug report to the BTS as I suggested above. 2.C If this does not work, don't do anything more; trust that the maintainer knew what they were doing when they wrote the manual autopkgtests. For packages which don't currently have manually-written autopkgtests: 2.A' Try adding Testsuite: autopkgtest-pkg-pybuild to debian/control, then building the package and running autopkgtest. 2.B' If this works, then either Janitor adds this line to debian/control or submit a bug report to the BTS to recommend this. (But we would not expect Janitor to do step 2.1', so this might have to be a different setup, or maybe the script doing 2.A' could leave a list of packages for Janitor, or something like that.) 2.C' If this does not work, submit a wishlist bug to the BTS to recommend that the maintainer adds some autopkgtest tests, either by making the autodep8 system work or by writing some manual tests. I'd be wary about adding lintian tags for this, though: with so many packages not being able to use the autodep8 system (I vaguely recall someone suggesting that a third of Python packages would not be able to use the system), that's a lot of packages getting false positives. In particular, for the suggested first version of not-using-pybuild-autodep8 tag (which would probably be better named manual-autopkgtest-could-be-pybuild-autodep8), how would lintian go about identifying which packages fall into category 2.B and which into 2.C? The second version of the tag (better named something like no-autopkgtest-could-use-pybuild-autodep8, but that's still not very good) is less problematic.
This all makes a lot of sense to me and I think this is the way forward, once the MR is merged. Thanks for the discussion.
-- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Louis-Philippe Véronneau ⢿⡄⠘⠷⠚⠋ pollo@debian.org / veronneau.org ⠈⠳⣄
Attachment:
OpenPGP_0xE1E5457C8BAD4113.asc
Description: OpenPGP public key
Attachment:
OpenPGP_signature
Description: OpenPGP digital signature