Re: Automated backports based on Janitor work?

[Christian Kastner <ckk@debian.org>]

On 26.08.21 11:39, Lucas Nussbaum wrote:
> Additionally, one could imagine a DSL to:
> - make minor changes to the source package before building (adjust
>   dependencies, apply an additional patch, etc.)
> - tell sbuild that some build-dependencies must be pulled from backported
>   packages
> 
> Jelmer, did you already think about that? Is there a way one could help
> you?

I worked on something like this last fall, which I tentatively named
'apt-derive'.

Given a source package in some origin distribution, apt-derive can
automatically create a derivative of the source package, and of any
necessary dependencies, for a target distribution.

The popular derivation would be the backport type, of course. But other
possibilities exist. For example, I was interested in optimized build
flavors for certain target environments.

Here's sample code for a numpy buster backport, with the following
output reflecting the rebuild chain needed on amd64 circa September
2020, when I ran this:

  | from apt_derive import RebuildResolver
  |
  | # Used to identify Releases in python-apt
  | bullseye   = dict(origin='Debian',           codename='bullseye')
  | buster_bpo = dict(origin='Debian Backports', codename='buster backports')
  | buster     = dict(origin='Debian',           codename='buster')
  |
  | # origin: bullseye
  | # target: buster-backports
  | # Packages from buster can be used to satisfy build dependencies
  | rr = RebuildResolver(wanted_dist=bullseye,
  |                      target=buster_bpo,
  |                      added_dists=[buster])
  |
  | # Figure out what needs to be rebuilt
  | tree = rr.resolve('numpy')
  | if tree:
  |     print(tree.pformat())

  python3-numpy_1:1.19.2-2
    └─cython_0.29.21-1
    └─dvisvgm_2.10.1-1
      └─freetype_2.10.2+dfsg-3
    └─python-hypothesis_5.32.1-1
      └─sphinx_3.2.1-1
      └─python-lark_0.9.0-1
    └─sphinx_3.2.1-1

So back then, 7 unique packages needed to be backported (numpy and 6
others for newer build dependencies).

That's the resolving part of apt-derive. There's more to it of course,
for example there is a "blocked" list of packages where automatic
derivation would be refused (eg: for libc).

The rebuild part of apt-derive would create backported sources (with
changelog entries, and any other configured extensions) for the
packages.


My plan was to provide such a automated backports service under the name
"autobpo", with package suffix ~autobpo11, and suites
  * bullseye-autobpo-dangerous for the immediate build results
  * bullseye-autobpo for results that passed CI and rebuilt
    reproducibly

As to which packages to automatically backport, my plan was two-fold:
  * By popular request
  * Maintainers indicating so with X-AutoBPO-Policy: <something>
    somewhere in debian/control

What eventually stalled my effort was that I was lacking a reasonably
simple build environment for multiple architectures. I wrote sbuild-qemu
to get local access to many architectures, but in the end, I didn't see
a way around wanna-build etc. for a multi-node build farm. I found
wanna-build overly complicated, and then other stuff came up in $LIFE.

Raphael initiated debusine in the meantime, which seems to be ideal
solution to my build problem.

However, If anyone with wanna-build experience is interested in
collaborating to get this launched sooner, please ping me.

I'll try to finalize the first draft of apt-derive as soon as possible
in the hopes that something can come of it.

Best,
Christian