Hello,
we're developing an integration system for building Debian-based products, isar
[1]. It is essentially a set of commands for building product packages and
installing them into a Debian root filesystem to produce a complete working
image. It aims at supporting the whole product development lifecycle:
1. Create a base system:
* Define a list of binary Debian packages as a base system.
* Define base system customizations, such as inittab, fstab, etc.
2. Use a clean environment (chroot / VM) for building the following components:
* Product's modified Debian packages.
* Product's own driver, library, and application packages.
3. Support multiple diverse targets in one project:
* Debug / release builds.
* Images for different products.
* Images for different hardware platforms.
4. Good support for typical project use cases, such as:
* Create complete, bootable images automatically, with one command.
* Do that reproducibly, for any older version.
= Prior Art =
The task is trivial enough that everyone develops his own scripts for that [2].
After maintaining such scripts since 2004, we'd like to have something better.
Reasons:
* The scripts have become a mess and are difficult to maintain.
* They don't scale well for the following use cases:
* Maintaining several products (sharing or not sharing components) leads to
exploding complexity.
* The same applies for several levels of development (upstream software,
Debian packages, hardware vendors' BSP components, software vendors'
libraries, product's own components).
* They don't offer structure, and developers happen to solve every next problem
in a different way.
* "Every tool is tailored for the developer's use case and personal taste" [3].
A build system should Just Work(tm), be easy to use, offer structure, and be
flexible. The best would be to combine individual efforts to avoid solving the
same problems many times and constantly improve a single toolset that is
suitable for various use cases.
The closest to that vision is ELBE [4]. It solves the right problem and has
many features OOTB -- very important if you want to focus on your end product
and not develop another build system. However, its current implementation
doesn't scale well for our use cases, and we expect maintainability issues in
the projects using it.
= How Isar Works =
Isar follows the one job, one tool approach. Integration is achieved by using
BitBake, a build tool similar to make. It is the tool behind the source-based
Yocto embedded Linux distribution. With BitBake, one can define files with
build targets (recipes like coreutils, busybox), their dependencies (other
recipes like libc), and rules how to build them (shell or python statements).
All recipes together are collectively called "metadata" as opposed to the
source code itself, which may contain configure, Makefile.am, debian/, etc.
Isar is implemented as a set of BitBake recipes. There is one recipe per source
package. Build target is the Debian package. Since download ("fetch") and build
rules ("tasks") in Debian are uniform thanks to apt and dpkg, they are defined
once in a BitBake class and don't have to be duplicated in every recipe. The
source code of individual packages may be a part of an isar-based project, or
it may be in a different (public or private) repo. The recipes check out the
software packages from their repos and call dpkg-buildpackage on them. The
packages are later installed into the root filesystem using multistrap. There
are separate recipes for creating the build chroot and the final rootfs.
Conceptually, the whole build system can be seen as a result of splitting one
monolithic script into chunks within a predefined structure. Recipes, in turn,
are grouped in directories ("layers") by distro, vendor, product, or target
platform.
The structure is achieved as follows:
* On file level, by BitBake recipes. Different packages are built in different
recipes. Recipe execution order results from dependencies specified in the
recipes.
* On directory level, layers:
* meta: Base system recipes (e.g., coreutils).
* meta-VENDOR-bsp: Hardware vendor's recipes (e.g., drivers, libs).
* meta-PRODUCT: Product recipes and customizations (e.g., product apps and
libs, partitioning, splash screen).
Layer directories may be a part of a project repo, or be external to it. In
this way, anyone can fork an upstream repo, add his own layers (possibly
extending or overriding the core recipes), and pass that to his downstream
without having to fork and modify the core recipes (thus reducing maintenance
effort). The desired layer mix is chosen in the build config file by the end
developer.
* Different target platforms are chosen by specifying the MACHINE variable
in the BitBake configuration file in the local build directory.
* Component sharing between products is achieved by dependencies specified in
BitBake recipes. If several products share 90% of components, their top-level
recipes depend on the recipe of a common base system, similar to Debian
metapackages.
Component variation (kernel, U-Boot) is achieved by the Provides: mechanism,
like in Debian.
Depending on the extent of differences, product variation may be achieved by
different build targets or co-existing meta-PRODUCT1, meta-PRODUCT2 layers.
With added structure, flexibility of "raw" scripts is also preserved: BitBake
recipes contain procedural shell or Python code. New tasks and their order can
be defined in any recipe; the development is not blocked if the build system
doesn't provide an appropriate hook.
An important issue is readable SCM history. With isar, editing one big build
script or mixing changes to code and blobs (e.g., tarballs) can be avoided.
Patches tend to be very local and very readable.
= Workflow =
The workflow is similar to that of Yocto.
To create a project based on isar, its repository is cloned to become the
project repository. Then, recipes for project-specific packages are added.
Debian package recipes are almost one-liners with no procedural code necessary
by default.
To build the images, a build directory must be created and configured. This is
done with a helper script: isar-init-build-env <dir>. The config files in
<dir>/conf/ can be customized.
System images are built with one command: bitbake <image name>. A default image
recipe is supplied in the isar repo. Packages are built natively using a chroot
with qemu (cross-building is also possible). Root filesystem customizations go
into multistrap scripts. Repeatability is achieved by versioning apt repos and
referencing the tags in project recipes.
= Advantages of Isar =
* Modularity, flexibility and scalability through using BitBake.
* Proven-in-use structure and workflows of the Yocto project.
* Re-use of familiar, mature tools like BitBake, multistrap, dpkg, apt, qemu.
* Re-use of pre-built binary Debian packages rather than building them from
scratch every time.
= What's next? =
Siemens is using a predecessor of this system and would like to develop it
further. What we are looking for is a discussion. Do you see a need for uniting
forces to develop such a system as a community? Would you use it for your next
project? Do you think this is the optimal approach? Feedback and questions
welcome. We'd like to combine efforts under a shared vision rather than having
everyone developing his own tool.
References:
1. Proof-of-concept implementation: https://github.com/ilbers/isar
2. http://annex.debconf.org/debconf-share/debconf15/slides/creating-bootable-debian-images/#/
3. http://annex.debconf.org/debconf-share/debconf15/slides/creating-bootable-debian-images/#/24
4. https://lists.debian.org/debian-embedded/2016/01/msg00000.html
With kind regards,