Re: Seeking feedback on a meta package builder
Hi Raphaël,
Reading your reply makes me think there must be a misunderstanding. Much
of what you write only makes sense to me that way. I hope my reply is
not too repetitive and clears up some of that.
On Sun, Jun 06, 2021 at 06:00:28PM +0200, Raphael Hertzog wrote:
> I don't want to sound too negative but you can't put at the same level
> "mdbp-ssh" and "mdbp-sbuild". Your abstraction is going too far by trying
> to include remote building IMO.
The abstraction says precisely nothing about where a build is being
performed. The use of ssh for a remote middleware is a detail
(hopefully) invisible to the user.
> And while abstractions are nice, for debusine I need to know how each
> implementation work to be able to answer questions like "can I build for
> that distribution on this worker" (i.e. do I have the required chroots) ?
> And here your current abstraction is hurting more than it helps.
My abstraction says nothing about workers or how to select them. The
idea behind that was that a user should not have to care. Yes, you
(debusine) do need a mapping between workers and available distributions
somewhere. You can implement scheduler as a middleware (or router) in
the abstraction I'm proposing to do this selection. What I don't see is
where this hurts. The worker selection is something that you have to do
anyway regardless of whether you use mdbp for the build abstraction.
I'm actually interested in figuring this out as the fewer competing
abstractions we end up with, the better they will be supported.
> So I'm afraid that mdbp is just not suited for the needs of debusine at
> this point.
I don't fully follow your reasoning yet and hope that we can get this
sorted out.
> Also given that you want to include support of building across multiple
> machines, I don't really understand why you select a command line API
> as the interface to drive this all. It's just not a good fit. Having one
> process drive all the machines via ssh is just fragile and not really
> scalable. And it requires all the machines to be directly adressable. In
> debusine, only the server needs a public IP, the workers connect to it.
Good questions. A command line API gives freedom to the implementer of
the API. It allows easy wrapping ("middleware") of implementations and
choice of implementation language. While my implementations have been
done in Python, I don't think Python is a requirement here. It just
happens to be convenient to me. Actually having competing
implementations can be a nice thing as long as their interface is
compatible.
I'm not sure where you got this "one process drive all machines". The
proposed abstraction calls for one process per build. One invocation
will certainly only deal with one machine at a time. The thing that
might not scale is the number of processes.
You mention the need for machines to be directly addressable. That is
true for mdbp-ssh, but it is only one implementation of the abstraction.
You are free to implement other remote middlewares that lack this
requirement.
Your paragraph gives good reasons for why the mdbp-ssh middleware is a
bad fit for debusine. What I don't see is good reasons for the
abstraction not being a good fit.
I note that my abstraction kinda has two levels. For one thing there is
a build object (represented as json, but that really doesn't matter). It
describes the action that is requested by the user (what to build,
architectures, options, etc.). The other is a command line interface
that receives a build object and transmits (via files) the relevant
artifacts. Maybe the build object level better addresses the needs of
debusine while leaving some aspects unanswered (e.g. artifact
transmission).
Helmut
Reply to: