On Wed, 30 Mar 2016 19:03:59 +0200
Baurzhan Ismagulov <firstname.lastname@example.org> wrote:
>we're developing an integration system for building Debian-based
>products, isar . It is essentially a set of commands for building
>product packages and installing them into a Debian root filesystem to
>produce a complete working image. It aims at supporting the whole
>product development lifecycle:
>1. Create a base system:
> * Define a list of binary Debian packages as a base system.
> * Define base system customizations, such as inittab, fstab, etc.
>2. Use a clean environment (chroot / VM) for building the following
> * Product's modified Debian packages.
> * Product's own driver, library, and application packages.
>3. Support multiple diverse targets in one project:
> * Debug / release builds.
> * Images for different products.
> * Images for different hardware platforms.
>4. Good support for typical project use cases, such as:
> * Create complete, bootable images automatically, with one command.
> * Do that reproducibly, for any older version.
>= Prior Art =
>The task is trivial enough that everyone develops his own scripts for
>that . After maintaining such scripts since 2004, we'd like to have
>something better. Reasons:
>* The scripts have become a mess and are difficult to maintain.
>* They don't scale well for the following use cases:
> * Maintaining several products (sharing or not sharing components)
> leads to exploding complexity.
> * The same applies for several levels of development (upstream
> software, Debian packages, hardware vendors' BSP components, software
> vendors' libraries, product's own components).
>* They don't offer structure, and developers happen to solve every
> in a different way.
>* "Every tool is tailored for the developer's use case and personal
>A build system should Just Work(tm), be easy to use, offer structure,
>and be flexible. The best would be to combine individual efforts to
>avoid solving the same problems many times and constantly improve a
>single toolset that is suitable for various use cases.
>The closest to that vision is ELBE . It solves the right problem
>and has many features OOTB -- very important if you want to focus on
>your end product and not develop another build system. However, its
>current implementation doesn't scale well for our use cases, and we
>expect maintainability issues in the projects using it.
>= How Isar Works =
>Isar follows the one job, one tool approach. Integration is achieved
>by using BitBake, a build tool similar to make. It is the tool behind
>the source-based Yocto embedded Linux distribution. With BitBake, one
>can define files with build targets (recipes like coreutils, busybox),
>their dependencies (other recipes like libc), and rules how to build
>them (shell or python statements). All recipes together are
>collectively called "metadata" as opposed to the source code itself,
>which may contain configure, Makefile.am, debian/, etc.
>Isar is implemented as a set of BitBake recipes. There is one recipe
>per source package. Build target is the Debian package. Since download
>("fetch") and build rules ("tasks") in Debian are uniform thanks to
>apt and dpkg, they are defined once in a BitBake class and don't have
>to be duplicated in every recipe. The source code of individual
>packages may be a part of an isar-based project, or it may be in a
>different (public or private) repo. The recipes check out the software
>packages from their repos and call dpkg-buildpackage on them. The
>packages are later installed into the root filesystem using
>multistrap. There are separate recipes for creating the build chroot
>and the final rootfs.
>Conceptually, the whole build system can be seen as a result of
>splitting one monolithic script into chunks within a predefined
>structure. Recipes, in turn, are grouped in directories ("layers") by
>distro, vendor, product, or target platform.
>The structure is achieved as follows:
>* On file level, by BitBake recipes. Different packages are built in
> recipes. Recipe execution order results from dependencies specified
> in the recipes.
>* On directory level, layers:
> * meta: Base system recipes (e.g., coreutils).
> * meta-VENDOR-bsp: Hardware vendor's recipes (e.g., drivers, libs).
> * meta-PRODUCT: Product recipes and customizations (e.g., product
> apps and libs, partitioning, splash screen).
> Layer directories may be a part of a project repo, or be external to
> it. In this way, anyone can fork an upstream repo, add his own layers
> (possibly extending or overriding the core recipes), and pass that to
> his downstream without having to fork and modify the core recipes
> (thus reducing maintenance effort). The desired layer mix is chosen
> in the build config file by the end developer.
>* Different target platforms are chosen by specifying the MACHINE
> in the BitBake configuration file in the local build directory.
>* Component sharing between products is achieved by dependencies
> BitBake recipes. If several products share 90% of components, their
> top-level recipes depend on the recipe of a common base system,
> similar to Debian metapackages.
> Component variation (kernel, U-Boot) is achieved by the Provides:
> mechanism, like in Debian.
> Depending on the extent of differences, product variation may be
> achieved by different build targets or co-existing meta-PRODUCT1,
> meta-PRODUCT2 layers.
>With added structure, flexibility of "raw" scripts is also preserved:
>BitBake recipes contain procedural shell or Python code. New tasks and
>their order can be defined in any recipe; the development is not
>blocked if the build system doesn't provide an appropriate hook.
>An important issue is readable SCM history. With isar, editing one big
>build script or mixing changes to code and blobs (e.g., tarballs) can
>be avoided. Patches tend to be very local and very readable.
>= Workflow =
>The workflow is similar to that of Yocto.
>To create a project based on isar, its repository is cloned to become
>the project repository. Then, recipes for project-specific packages
>are added. Debian package recipes are almost one-liners with no
>procedural code necessary by default.
>To build the images, a build directory must be created and configured.
>This is done with a helper script: isar-init-build-env <dir>. The
>config files in <dir>/conf/ can be customized.
>System images are built with one command: bitbake <image name>. A
>default image recipe is supplied in the isar repo. Packages are built
>natively using a chroot with qemu (cross-building is also possible).
>Root filesystem customizations go into multistrap scripts.
>Repeatability is achieved by versioning apt repos and referencing the
>tags in project recipes.
>= Advantages of Isar =
>* Modularity, flexibility and scalability through using BitBake.
>* Proven-in-use structure and workflows of the Yocto project.
>* Re-use of familiar, mature tools like BitBake, multistrap, dpkg,
>* Re-use of pre-built binary Debian packages rather than building them
> scratch every time.
>= What's next? =
>Siemens is using a predecessor of this system and would like to
>develop it further. What we are looking for is a discussion. Do you
>see a need for uniting forces to develop such a system as a community?
>Would you use it for your next project?
Do you think this is the
>optimal approach? Feedback and questions welcome. We'd like to combine
>efforts under a shared vision rather than having everyone developing
>his own tool.
Looks very interesting. Per your TODO, it looks like you're>
>1. Proof-of-concept implementation: https://github.com/ilbers/isar
>With kind regards,
already thinking about what I had questions about, e.g. dependencies
(would there be specification duplication here?)
Curious, was there a reason you did not submodule bitbake in your repo?
Seems like you should keep bb current, but also be able to reproduce an
image at the originally recorded release-time bb commitish too if