Re: Package testing with Salsa CI for new packages
On Friday, 18 August 2023 18:21:00 CEST Andrey Rakhmatullin wrote:
> On Thu, Aug 17, 2023 at 05:10:08PM +0200, Paul Boddie wrote:
> > Here, it would seem that the most prudent approach is to use the Salsa CI
> > service instead of trying to get the test suite to run during the package
> > build process.
>
> You should do both if possible, assuming that by "Salsa CI service" you
> mean autopkgtests which you can, and IMO should, also run locally.
I'm not really clear on what autopkgtest really is, other than a tool that
uses some kind of test suite description that may reside in debian/test. I'm
also not completely clear on what is supposed to invoke it, other than either
the Salsa CI mechanism or dh_auto_test. In the Debian Wiki documentation...
https://wiki.debian.org/Python/LibraryStyleGuide
...it mentions a field in debian/control:
Testsuite: autopkgtest-pkg-python
My impression is that this calls autodep8 to generate some kind of test suite
description which is then invoked by dh_auto_test. It doesn't help that there
is an alternative to this that resembles it but behaves differently:
Testsuite: autopkgtest-pkg-pybuild
What I have previously managed to do with the Salsa CI mechanism is to run
tests specified in debian/test, disabling these tests at build time using the
following in debian/rules:
export PYBUILD_DISABLE=test
For another package which has limited dependencies, I do this because the
tests take a long time to run, and I even need to increase the timeout for the
Salsa CI jobs. For this package, the tests don't take too long to run (those
that do run), but some tests fail for reasons previously described, and this
is due to some configuration issue that is unpleasant to debug.
Success with testing using Salsa CI steers me away from using the Testsuite
field in the normal build environment because the resulting test suite
invocation failures suggest that some customisation is needed or that the
environment is inappropriate in some way (even if I specify numerous build
dependencies).
> > One motivation for doing so involves not having to specify
> > build dependencies for packages only needed for test suite execution,
> > which itself requires the invocation of gbp buildpackage with
> > --extra-package arguments since some packages are completely new to
> > Debian.
>
> This is fine. Not to mention that the same problem exists for
> autopkgtests, as you say below.
>
> > I have also found it difficult to persuade the tests to run successfully
> > during the build process. A few of these attempt to invoke the moin
> > program, but this cannot be located since it is not installed in the
> > build environment.
>
> This should also be fine, unless it's completely impossible to run it
> without installing into /.
The moin program is made available in setup.py using an entry point. Maybe if
there were some kind of script instead, it would work.
> > However, one conclusion is that testing a system, as some of the
> > test cases appear to do, and as opposed to testing library functionality,
> > is not necessarily appropriate when directed by something like
> > dh_auto_test.
>
> If there are tests that can't be run at build time you can skip those. You
> can even ask the upstream to provide tool arguments to simplify that.
I may well discuss such matters with them. One challenge that is relevant in
this situation is that upstream have been working in their own virtualenv (or
venv, or whatever it is now) world for a few years, using plenty of
dependencies that were not packaged in Debian. That hasn't been entirely
conducive to adoption of Moin 2 which has in turn isolated its developers
further.
Fortunately, the gulf between that world and Debian can be bridged, and some
of the more tedious packaging seems to have been done for many of the
dependencies involved. It just needs someone to shepherd this effort to an
acceptable conclusion, which is of concern to Debian itself since Moin runs
the Debian Wiki.
Thanks for the encouragement and advice.
Paul
Reply to: