Re: Debian-based product build system
On Wed, 30 Mar 2016 19:03:59 +0200
Baurzhan Ismagulov <ibr@radix50.net> wrote:
>Hello,
>
>we're developing an integration system for building Debian-based
>products, isar [1]. It is essentially a set of commands for building
>product packages and installing them into a Debian root filesystem to
>produce a complete working image. It aims at supporting the whole
>product development lifecycle:
>
>1. Create a base system:
> * Define a list of binary Debian packages as a base system.
> * Define base system customizations, such as inittab, fstab, etc.
>
>2. Use a clean environment (chroot / VM) for building the following
>components:
> * Product's modified Debian packages.
> * Product's own driver, library, and application packages.
>
>3. Support multiple diverse targets in one project:
> * Debug / release builds.
> * Images for different products.
> * Images for different hardware platforms.
>
>4. Good support for typical project use cases, such as:
> * Create complete, bootable images automatically, with one command.
> * Do that reproducibly, for any older version.
>
>
>= Prior Art =
>
>The task is trivial enough that everyone develops his own scripts for
>that [2]. After maintaining such scripts since 2004, we'd like to have
>something better. Reasons:
>
>* The scripts have become a mess and are difficult to maintain.
>
>* They don't scale well for the following use cases:
>
> * Maintaining several products (sharing or not sharing components)
> leads to exploding complexity.
>
> * The same applies for several levels of development (upstream
> software, Debian packages, hardware vendors' BSP components, software
> vendors' libraries, product's own components).
>
>* They don't offer structure, and developers happen to solve every
>next problem
> in a different way.
>
>* "Every tool is tailored for the developer's use case and personal
>taste" [3].
>
>A build system should Just Work(tm), be easy to use, offer structure,
>and be flexible. The best would be to combine individual efforts to
>avoid solving the same problems many times and constantly improve a
>single toolset that is suitable for various use cases.
>
>The closest to that vision is ELBE [4]. It solves the right problem
>and has many features OOTB -- very important if you want to focus on
>your end product and not develop another build system. However, its
>current implementation doesn't scale well for our use cases, and we
>expect maintainability issues in the projects using it.
>
>
>= How Isar Works =
>
>Isar follows the one job, one tool approach. Integration is achieved
>by using BitBake, a build tool similar to make. It is the tool behind
>the source-based Yocto embedded Linux distribution. With BitBake, one
>can define files with build targets (recipes like coreutils, busybox),
>their dependencies (other recipes like libc), and rules how to build
>them (shell or python statements). All recipes together are
>collectively called "metadata" as opposed to the source code itself,
>which may contain configure, Makefile.am, debian/, etc.
>
>Isar is implemented as a set of BitBake recipes. There is one recipe
>per source package. Build target is the Debian package. Since download
>("fetch") and build rules ("tasks") in Debian are uniform thanks to
>apt and dpkg, they are defined once in a BitBake class and don't have
>to be duplicated in every recipe. The source code of individual
>packages may be a part of an isar-based project, or it may be in a
>different (public or private) repo. The recipes check out the software
>packages from their repos and call dpkg-buildpackage on them. The
>packages are later installed into the root filesystem using
>multistrap. There are separate recipes for creating the build chroot
>and the final rootfs.
>
>Conceptually, the whole build system can be seen as a result of
>splitting one monolithic script into chunks within a predefined
>structure. Recipes, in turn, are grouped in directories ("layers") by
>distro, vendor, product, or target platform.
>
>The structure is achieved as follows:
>
>* On file level, by BitBake recipes. Different packages are built in
>different
> recipes. Recipe execution order results from dependencies specified
> in the recipes.
>
>* On directory level, layers:
>
> * meta: Base system recipes (e.g., coreutils).
>
> * meta-VENDOR-bsp: Hardware vendor's recipes (e.g., drivers, libs).
>
> * meta-PRODUCT: Product recipes and customizations (e.g., product
> apps and libs, partitioning, splash screen).
>
> Layer directories may be a part of a project repo, or be external to
> it. In this way, anyone can fork an upstream repo, add his own layers
> (possibly extending or overriding the core recipes), and pass that to
> his downstream without having to fork and modify the core recipes
> (thus reducing maintenance effort). The desired layer mix is chosen
> in the build config file by the end developer.
>
>* Different target platforms are chosen by specifying the MACHINE
>variable
> in the BitBake configuration file in the local build directory.
>
>* Component sharing between products is achieved by dependencies
>specified in
> BitBake recipes. If several products share 90% of components, their
> top-level recipes depend on the recipe of a common base system,
> similar to Debian metapackages.
>
> Component variation (kernel, U-Boot) is achieved by the Provides:
> mechanism, like in Debian.
>
> Depending on the extent of differences, product variation may be
> achieved by different build targets or co-existing meta-PRODUCT1,
> meta-PRODUCT2 layers.
>
>With added structure, flexibility of "raw" scripts is also preserved:
>BitBake recipes contain procedural shell or Python code. New tasks and
>their order can be defined in any recipe; the development is not
>blocked if the build system doesn't provide an appropriate hook.
>
>An important issue is readable SCM history. With isar, editing one big
>build script or mixing changes to code and blobs (e.g., tarballs) can
>be avoided. Patches tend to be very local and very readable.
>
>
>= Workflow =
>
>The workflow is similar to that of Yocto.
>
>To create a project based on isar, its repository is cloned to become
>the project repository. Then, recipes for project-specific packages
>are added. Debian package recipes are almost one-liners with no
>procedural code necessary by default.
>
>To build the images, a build directory must be created and configured.
>This is done with a helper script: isar-init-build-env <dir>. The
>config files in <dir>/conf/ can be customized.
>
>System images are built with one command: bitbake <image name>. A
>default image recipe is supplied in the isar repo. Packages are built
>natively using a chroot with qemu (cross-building is also possible).
>Root filesystem customizations go into multistrap scripts.
>Repeatability is achieved by versioning apt repos and referencing the
>tags in project recipes.
>
>
>= Advantages of Isar =
>
>* Modularity, flexibility and scalability through using BitBake.
>
>* Proven-in-use structure and workflows of the Yocto project.
>
>* Re-use of familiar, mature tools like BitBake, multistrap, dpkg,
>apt, qemu.
>
>* Re-use of pre-built binary Debian packages rather than building them
>from
> scratch every time.
>
>
>= What's next? =
>
>Siemens is using a predecessor of this system and would like to
>develop it further. What we are looking for is a discussion. Do you
>see a need for uniting forces to develop such a system as a community?
>Would you use it for your next project? Do you think this is the
>optimal approach? Feedback and questions welcome. We'd like to combine
>efforts under a shared vision rather than having everyone developing
>his own tool.
>
>
>References:
>
>1. Proof-of-concept implementation: https://github.com/ilbers/isar
>2.
>http://annex.debconf.org/debconf-share/debconf15/slides/creating-bootable-debian-images/#/
>3.
>http://annex.debconf.org/debconf-share/debconf15/slides/creating-bootable-debian-images/#/24
>4. https://lists.debian.org/debian-embedded/2016/01/msg00000.html
>
>
>With kind regards,
Looks very interesting. Per your TODO, it looks like you're
already thinking about what I had questions about, e.g. dependencies
(would there be specification duplication here?)
Curious, was there a reason you did not submodule bitbake in your repo?
Seems like you should keep bb current, but also be able to reproduce an
image at the originally recorded release-time bb commitish too if
desired/required.
--
Regards,
Christopher
Reply to: