[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: State of emdebian / Future plans

(Please CC me in replies.)

On Sun, Mar 12, 2017 at 07:00:28PM +0100, Baurzhan Ismagulov wrote:
> Wookey provides the toolchains. @Wookey: Does "as long as jessie is supported"
> mean that stretch provides cross-compiler sources or binaries out of the box?

Yes, stretch includes cross compilers maintained by gcc maintainer
Matthias Klose targeting a fair number of architectures.

> Neil has pushed the effort on making packages cross-buildable out of the box
> [2, 3]. He released Emdebian distributions [4]. Grip provided stripped Debian
> binary packages. Crush provided cross-built binary packages. The effort has
> been abandoned due to lack of resources.

With unstable now containing cross toolchains for a while now, our focus
has shifted. I have spent considerable effort at making packages in
unstable cross build out of the box. My current numbers are roughly:

 * 12000 source packages build architecture-dependent binary packages.
   (Others are not relevant to cross building.)
 * 6000 have satisfiable cross Build-Depends on at least one
 * I attempted cross building 2300 of those (selection mostly based on
   popcon) for random architectures.
 * 1300 cross built successfully. (Successful builds that failed using
   cross tools are already excluded here.)
 * I filed lots of bugs with patches. Currently, 192 distinct bugs
   affecting cross buildability of 630 source packages are recorded.
   A larger number (~400) of bugs has been fixed in stretch already.

Thus we can safely conclude that 10% of stretch will be cross buildable
out of the box.

> In other words, companies continue using Debian in their products [9], it's too
> valuable to re-invent it in-house. The efforts are somewhat distributed ATM,
> although we are talking how we could improve things. A central policy and
> co-ordination are currently missing, and I find that ok for the time being --
> implementation and persistence first. So, if you would like to use Debian for
> your projects -- that's a good choice, and there are tools for that; at the
> same time, it's still a long way to a bigger and more organized community. If
> we could help, please let us know.

Help would be welcome indeed. There are very many loose ends that
require effort to turn vanilla Debian into a reasonable
Yocto-competitor. Let me try to summarize a few that I have been working
 * Generally making packages cross buildable. There are a number of very
   simple bug classes that simply require someone doing the work:
   + Autotools-based and cmake-based  packages that do not use
     dh_auto_configure often lack relevant --host flags (or their cmake
   + .pc files shipped in /usr/lib/pkgconfig are not found during cross
     compilation and need to be moved to /usr/lib/<triplet>/pkgconfig.
   + Build-Dependencies on perl/python/ruby and other tools are often
     used as build tools rather than libraries. Thus many (but not all)
     need to be annotated with :any or :native.
   + A lot of packages call into plain pkg-config when they need to call
     the <hosttriplet>-pkg-config. Sometimes this happens due to using
     AC_CHECK_PROG instead of AC_CHECK_TOOL or because a Makefile
     hardcodes use of pkg-config.
   In general, the class of bugs that can be fixed in less than an hour
   is still very much populated. But there are also a number of harder
 * Making Debian "bootstrappable", i.e. making it possible to cross
   compile a base system from source. This mostly requires adding build
   profiles in the relevant places and largely is a solved problem. For
   the embedded use case though, more profiles will be needed to
   optionally disable unnecessary features (i.e. what Emdebian Crush
   did). For instance, disabling selinux could noticeably reduce the
   build time. For many architectures, cross boostrapping mostly
   works[1] already.
 * Simplifying foreign installations. As far as I can tell, the current
   state of the art in installing packages still is multistrap. It
   allows foreign package installation albeit leaving them unconfigured.
   Configuration can be done using qemu-user-static or by rewriting the
   relevant scripts. I think neither approach scales and we should move
   to running standard maintainer scripts outside the chroot they are
   operating on. That feature goes by the name DPKG_ROOT[2] and
   currently, you can only experiment with it by passing
   --force-script-chrootless to dpkg. Adding support for DPKG_ROOT to
   maintainer scripts and figuring out its semantics is present work.
 * Cross compiling to from linux to non-linux or from glibc to non-glibc
   (e.g. musl) currently fails, because there are file conflicts in
   headers. We'll have to move[3] standard headers (e.g. limits.h) to
   /usr/include/<triplet>. The initial patches are there, but someone
   needs to perform an archive rebuild to see what breaks.

I am also interested in hearing more from cross compilation users to be
able to use a better prioritization mechanism than popcon. Anyone
interested in working on these topic is invited to get in touch with me.
The preferred ways are the #debian-bootstrap IRC channel on OFTC and
debian-cross@lists.debian.org. I am not subscribed to d-embedded@l.d.o.


[1] https://wiki.debian.org/HelmutGrohne/rebootstrap
[2] https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap
[3] https://wiki.debian.org/Multiarch/GlibcHeaders

Reply to: