Re: Debian-based product build system

[Manuel Traut <manut@mecka.net>]

On 17:53 Thu 31 Mar     , Baurzhan Ismagulov wrote:
> On Wed, Mar 30, 2016 at 09:17:49PM +0100, Wookey wrote:
> > And there is ELBE:
> > https://lists.debian.org/debian-embedded/2016/01/msg00000.html
> > 
> > There does seem to be interest in collaborating here. I hope you can
> > find some common ground.
> 
> Yes, I've actually mentioned ELBE and this link in the original announcement.
> 
> Our colleagues have looked seriously at ELBE. As already mentioned, it produces
> the right result and has many features, which is important for us. That said, I
> see a number of issues with its current implementation. I don't know it well,
> so please correct me if anything is wrong.
> 
> * Structure, scalability and variability: ELBE follows a "one image - one
>   config file" policy. It's effectively a central tool with a single metadata
>   file. This precludes effective component sharing, product and vendor
>   variations.

That's right. The intend was, to build sothing that is easy to use and
understand. The posibilities in e.g. yocto are great, but often it is hard
to understand which version of a bb file was used, or what was modified by
bbappend files and so on. It's hard to explain this to somebody who just wants
to build its application on top of a GNU/Linux system.

>   There is a proposal to use some XML include mechanism. Splitting an existing
>   XML file top-down into three files for two products (one common and two
>   product-specific parts) solves the problem only to a small extent, for one
>   particular use case. Besides, split commits are very unreadable, especially
>   if they contain some adaptations to keep the splitted version functional.
>   Including specific files instead of depending on provided features makes
>   multiple vendor layering impossible.
>   Isar follows Yocto's bottom-up approach, every package having its own recipe.
>   In this way, any kind of mix can be defined through dependencies. Recipe
>   groups ("layers") can be organized e.g. by vendor, product. This facilitates
>   effective component sharing and variability.

I don't like the include mechanism and don't use it. If prefer debian
meta-packages and one minimal config file per target containing the image
format and the needed meta-packages.

> * Application development: If I have a product-specific app in an external
>   repo, can I dpkg-buildpackage it with ELBE, or do I have to build it myself
>   and commit the binary package into an apt repo, to be used later by ELBE?

In elbe 1.0 we don't care about building from source. In elbe 2.0 will be a
feature to specify VCS-URIs containing debianized source-code. The packages
will be generated by a pbuilder inside the elbe initvm and hosted in a project
specific Debian repository inside the initvm.

The produced packages can be used like any other debian package during image
generation.

The commit implementing this is here [0], but it's still work in progress.

> * Flexibility and extensibility: The elbe tool and the XML file format
>   prescribe what one can do with the build system. Implementing anything not
>   foreseen by the policy requires either updating the tool (which blocks
>   development), or creating wrappers around the tool (which I don't want if I
>   have a central tool like elbe).

You can specify finetuning tags in the XML file. Some tags are used e.g. in
this example [1]. They apply to an image and are not package specific.

If you need package specific customizations we recommend to store the source
and debianization in a VCS. Product/machine specifics can be separated in a
VCS branch and the target recipe needs to point to the correct branch/commit.

This depends on [0] and is not really feature complete. But the description
above describe what we like to implement.

>   Isar performs every step (build a package, create a rootfs) in a separate
>   recipe. To add new functionality, a new recipe can be created. The execution
>   order is defined by recipe dependencies. Alternatively, an existing recipe
>   can be extended, without actually modifying it. This is important if one
>   extends an upstream recipe. Upgrading to a newer upstream version doesn't
>   result in a tree with conflicts that have to be fixed first.

That's definetly a good point. We also thought about using yocto to build
Debian repositories that then can be used by elbe to build the image. But
debootstrap depends on the 'essential' tags, but they are missing in the
packages generated from poky. Maybe the 'meta-debian' can resolve this problem.

> * Maintainability: The XML file format with blobs isn't very readable (lower
>   SNR compared to e.g. BitBake recipes). Neither are patches to it.

The binary blobs shouldn't be stored in VCS. Store the RFS overlay parallel
to the XML file and use elbe chg_archive before building the XML.

The combination is just done to have it in a single file to ease reproducability.

>   There is an idea of adding a higher-level language translating to XML. That
>   would add complexity and make debugging more difficult, so I'm not a fan of
>   this approach.

I don't care much of the input format, if someone presents another format
we could maybe support differnt input formats.

>   Isar is a set of individual recipes. Recipes are very terse (for Debian
>   packages, almost one-liners without code) and readable (only variable
>   assignments and shell / Python code if required). The best way to maintain
>   package sources is a repo outside of the project one (compared to committing
>   tarballs into the project repo). In this way, blobs are avoided and patches
>   are clean.
>
> We would be glad to see if we could find some common ground for collaboration.
> Manuel, what do you think?

elbe generates a pbuilder setup for building packages from source, maybe that
could be interesting for you. It ensures that each package is build in a clean
environment and the Build-depends are specified correctly.

Another thing is the image generation code. It is able to generate a lot of
formats. I think about splitting it out of elbe and maybe share some
python-module with vmdebootstrap who have nearly the same code. Maybe this
could also be interesting for yocto.

  Manuel

[0] https://github.com/Linutronix/elbe/commit/4847eefd3e9de844c40e063fd9bf2f24cb3fba6a
[1] https://github.com/Linutronix/elbe/blob/devel/elbe-2.0/examples/x86_64-pc-hdimg-gnome3.xml