Seeking feedback on a meta package builder

[Helmut Grohne <helmut@subdivi.de>]

Hi fellow developers,

Building Debian packages is a job that is not only part of our daily
development practice, but also a task regularly performed by tools.
These include:
 * QA tools frequently rebuild packages to find bugs
   * Archive rebuilds (e.g. Lucas Nussbaum, Matthias Klose, Santiago
     Vila, et al)
   * reproducible.debian.net
   * crossqa.debian.net
   * ratt
 * Tools whose task is larger than building packages, but happens to
   include doing so
   * https://wiki.debian.org/HelmutGrohne/rebootstrap
   * libdvd-pkg
   * apt-src

I suspect that this list is incomplete. My primary motivation for
looking into this came from rebootstrap though.

The task of building Debian packages has been solved numerous times.
Here are some known solutions:
 * sbuild
 * pbuilder
 * cowbuilder
 * qemubuilder
 * docker-based tools:
   + debocker
   + debdocker
   + docker-buildpackage
   + whalebuilder
 * systemd-nspawn-based tools
   + conbuilder
   + debspawn
 * manually without chroot
 * Create a chroot for each build using mmdebstrap

Again, the list is incomplete and if you want to improve it, please edit
https://wiki.debian.org/SystemBuildTools instead.

When someone writes a tool, that happens to build packages as part of
its job (the earlier list), typically one of the build tools (the second
list) is chosen and fixed. It's not like you can mix and match them.
Here are a number of lock-ins I happen to know:

 * reproducible.debian.net -> pbuilder
 * crossqa.debian.net -> sbuild
 * ratt -> sbuild
 * libdvd-pkg -> manually
 * rebootstrap -> manually

I see this tight coupling as a problem for mainly two reasons.
 * When someone else works on your tool, they have to set up your
   preferred builder. As a developer, you cannot just choose between
   pbuilder and sbuild. You effectively get to use both, because some
   tools use one and others use the other.
 * As tasks grow, there is a desire to distribute builds to multiple
   machines. As such, the various tools typically grow their own
   strategy for moving build tasks between machines.

Lengthy problem description, I know. If you've read some of my bug
reports, you may have noticed that only few come without solutions.

So I set out to solve this. More precisely, I looked for an API that
could abstract both sbuild and pbuilder (because they're the most common
ones) and allow executing builds on remote machines. A tool performing
builds should not have to care which one is in use and whether the build
is performed locally or not. Since describing the settings of a build
can be complex, I chose json for representation. Some aspects that you
may want to set for a build are:
 * build architecture / host architecture
 * arch-only / indep-only / full
 * DEB_BUILD_PROFILES
 * DEB_BUILD_OPTIONS
 * whether network access is permissible
 * forcing a build path

I called it "Meta Dpkg-BuildPackage" or simply mdbp for now and you can
find it at:
    https://git.subdivi.de/?p=~helmut/mdbp.git;a=blob;f=README.md
    git://git.subdivi.de/~helmut/mdbp.git

So with this mail I seek multiple sorts of replies:
 * What other tools perform builds as part of their job, but don't
   exactly care how they're being performed?
 * What build tools am I missing?
 * Does the problem scope I've chosen make sense?
 * Feedback on the proposed API
 * What feature is missing for your tool to use mdbp?
 * Which build tools should mdbp support to make it useful to you?

Thank you for considering. If you want to discuss on irc, I think both
#debian-bootstrap and #debian-qa would be appropriate.

Helmut