Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF)

[Julian Gilbey <julian@d-and-j.net>]

On Wed, Jul 27, 2022 at 07:45:12PM +0100, Julian Gilbey wrote:
> On Wed, Jul 27, 2022 at 10:26:33AM -0400, Louis-Philippe Véronneau wrote:
> > The way I see it:
> > 
> > 1. We should have a Lintian tag for packages not using the new
> > pybuild-autodep8 autopkgtest. It would be even better if this tag would be a
> > pointed hint that identified 'manually' written unit test autopkgtests that
> > could be replaced.
> > 
> > This way, you get something like:
> > 
> > python-foo source: not-using-pybuil-autodep8 [debian/tests/unittests]
> > 
> > for python packages that have old 'manually' written unit test autopkgtests
> > and:
> > 
> > python-foo source: not-using-pybuild-autodep8 [no-autopkgtest]
> > 
> > for python packages without any autopkgtest.
> > 
> > 2. lintian-brush (or something else, but I think lintian-brush is the right
> > tool) would go over these packages to:
> > 
> > 2.1 Add the new autodep8 autopkgtests and build the package to see if they
> > pass
> > 2.2 Remove the "manual" unit test autopkgtests if 2.1 succeeds
> > 2.3 Open a bug report if 2.1 fails
> 
> I'd be wary about 2.2 and 2.3.  I have several packages where I know
> that an automated test will fail; there are all sorts of weird cases
> where I've had to write tests manually.  I would also be quite cross
> if manually crafted tests were automatically removed, especially in
> cases such as Simon mentioned where they do things that that
> automatically generated test does not do.  Another thing I could
> imagine happening is that the automated test succeeds in a trivial way
> - it succeeds but doesn't actually test much because of the nature of
> the package.
> 
> On the other hand, a bug report saying something like the following
> seems much more reasonable: "We've tested this package using the
> automated autopkgtest system and it seems to work by adding the line
> 'Testsuite: autopkgtest-pkg-pybuild'; please check that the automated
> tests cover all of the tests of your manually written debian/tests/*
> and if so, then please remove them.  The autopkgtest-pkg-pybuild logs
> are attached."  This would give the maintainer the chance to decide
> how best to proceed.

Here's another alternative to steps 2.1-2.3 based on this:

 For packages which currently have manually-written autopkgtests:

 2.A Try removing debian/tests and adding Testsuite:
     autopkgtest-pkg-pybuild to debian/control, then building the
     package and running autopkgtest.

 2.B If this works, then submit a bug report to the BTS as I suggested
     above.

 2.C If this does not work, don't do anything more; trust that the
     maintainer knew what they were doing when they wrote the manual
     autopkgtests.

 For packages which don't currently have manually-written
 autopkgtests:

 2.A' Try adding Testsuite: autopkgtest-pkg-pybuild to debian/control,
      then building the package and running autopkgtest.

 2.B' If this works, then either Janitor adds this line to
      debian/control or submit a bug report to the BTS to recommend
      this.  (But we would not expect Janitor to do step 2.1', so this
      might have to be a different setup, or maybe the script doing
      2.A' could leave a list of packages for Janitor, or something
      like that.)

 2.C' If this does not work, submit a wishlist bug to the BTS to
      recommend that the maintainer adds some autopkgtest tests,
      either by making the autodep8 system work or by writing some
      manual tests.

I'd be wary about adding lintian tags for this, though: with so many
packages not being able to use the autodep8 system (I vaguely recall
someone suggesting that a third of Python packages would not be able
to use the system), that's a lot of packages getting false positives.
In particular, for the suggested first version of
not-using-pybuild-autodep8 tag (which would probably be better named
manual-autopkgtest-could-be-pybuild-autodep8), how would lintian go
about identifying which packages fall into category 2.B and which into
2.C?  The second version of the tag (better named something like
no-autopkgtest-could-use-pybuild-autodep8, but that's still not very
good) is less problematic.

Best wishes,

   Julian