Re: Seeking feedback on a meta package builder
Hi Raphaël,
On Wed, Jun 02, 2021 at 09:12:24AM +0200, Raphael Hertzog wrote:
> On Mon, 31 May 2021, Helmut Grohne wrote:
> > I see this tight coupling as a problem for mainly two reasons.
> [...]
> > * As tasks grow, there is a desire to distribute builds to multiple
> > machines. As such, the various tools typically grow their own
> > strategy for moving build tasks between machines.
>
> We already exchanged a while ago about "debusine", it ought to solve this
> part a least.
>
> https://freexian-team.pages.debian.net/debusine/
Yes, we did.
> We're just starting to work on the project with the goal of handling
> package builds as a first step:
> https://salsa.debian.org/freexian-team/debusine/-/milestones/1
>
> I invite you to "watch" the project on
> https://salsa.debian.org/freexian-team/debusine so that you get
> notification of comments on issues as that's where we are discussing
> the design.
>
> And feel free to file feature requests.
I think debusine does more and less than what I want at the same time:
* debusine thinks of a build more generically than building Debian
packages. Building a Debian package just happens to be a special case
that is very relevant to debusine, but it shall be able to build
other kinds of things as well.
* Beyond building, debusine also handles scheduling of builds and
storing of artifacts.
* debusine's build milestone implements a lock-in on sbuild. The
abstraction that I'm seeking doesn't happen here.
Possibly debusine could replace its sbuild task with a mdbp task? It
would automatically work on pbuilder and mmdebstrap for free. It also
doesn't have to implement another method for issuing builds remotely and
can concentrate on the scheduling part.
> In the current design
> (https://salsa.debian.org/freexian-team/debusine/-/issues/7) for the
> current iteration, I don't have that level of details but it should
> not be too hard to add this on top. Only the differentiated handling of
> build/host architecture might be non-trivial as it will impact the scheduling
> (identification of an appopriate worker).
Please get the architecture part right from the start. It is not that
hard actually. Your issue already mentions an unspecific "architecture"
field, so you already get to choose suitable workers. It is only the
build architecture that affects worker choice. The host architecture is
only a setting to pass down and debusine does not have to care about it
at all.
> > * Does the problem scope I've chosen make sense?
>
> "Package Build as a Service" :-)
Kinda. That would be awesome, yes. Indeed, the abstraction I am thinking
of should allow you to swap out a local builder for PBaaS as a simple
configuration change.
Maybe we can use some AWS credits to provide PBaaS to developers at some
point. Or even archive rebuilds as a service.
> When thinking of debusine, I thought something of something similar, I
> have set out to use sbuild in the first iteration because that what I
> (and Debian) uses for official package builds but given the number of
> different ways to build packages, I was wondering whether I should
> consider to have a tool-agnostic "PackageBuild" task that can be scheduled
> and that could then be performed by tool-specific implementations.
While Debian uses sbuild in official buildds, reproducible.d.n is fully
built on pbuilder instead. It isn't that homogeneous in the end. In
order to make debusine easier to set up, I think it should not tightly
couple with sbuild.
Consider using mdbp to implement your tool-agnostic Debian package
building task. What is missing on the mdbp side to make this work for
you?
Helmut
Reply to: