Re: State of emdebian / Future plans
(Please CC me in replies.)
On Sun, Mar 12, 2017 at 07:00:28PM +0100, Baurzhan Ismagulov wrote:
> Wookey provides the toolchains. @Wookey: Does "as long as jessie is supported"
> mean that stretch provides cross-compiler sources or binaries out of the box?
Yes, stretch includes cross compilers maintained by gcc maintainer
Matthias Klose targeting a fair number of architectures.
> Neil has pushed the effort on making packages cross-buildable out of the box
> [2, 3]. He released Emdebian distributions [4]. Grip provided stripped Debian
> binary packages. Crush provided cross-built binary packages. The effort has
> been abandoned due to lack of resources.
With unstable now containing cross toolchains for a while now, our focus
has shifted. I have spent considerable effort at making packages in
unstable cross build out of the box. My current numbers are roughly:
* 12000 source packages build architecture-dependent binary packages.
(Others are not relevant to cross building.)
* 6000 have satisfiable cross Build-Depends on at least one
architecture.
* I attempted cross building 2300 of those (selection mostly based on
popcon) for random architectures.
* 1300 cross built successfully. (Successful builds that failed using
cross tools are already excluded here.)
* I filed lots of bugs with patches. Currently, 192 distinct bugs
affecting cross buildability of 630 source packages are recorded.
A larger number (~400) of bugs has been fixed in stretch already.
Thus we can safely conclude that 10% of stretch will be cross buildable
out of the box.
> In other words, companies continue using Debian in their products [9], it's too
> valuable to re-invent it in-house. The efforts are somewhat distributed ATM,
> although we are talking how we could improve things. A central policy and
> co-ordination are currently missing, and I find that ok for the time being --
> implementation and persistence first. So, if you would like to use Debian for
> your projects -- that's a good choice, and there are tools for that; at the
> same time, it's still a long way to a bigger and more organized community. If
> we could help, please let us know.
Help would be welcome indeed. There are very many loose ends that
require effort to turn vanilla Debian into a reasonable
Yocto-competitor. Let me try to summarize a few that I have been working
on:
* Generally making packages cross buildable. There are a number of very
simple bug classes that simply require someone doing the work:
+ Autotools-based and cmake-based packages that do not use
dh_auto_configure often lack relevant --host flags (or their cmake
equivalent).
+ .pc files shipped in /usr/lib/pkgconfig are not found during cross
compilation and need to be moved to /usr/lib/<triplet>/pkgconfig.
+ Build-Dependencies on perl/python/ruby and other tools are often
used as build tools rather than libraries. Thus many (but not all)
need to be annotated with :any or :native.
+ A lot of packages call into plain pkg-config when they need to call
the <hosttriplet>-pkg-config. Sometimes this happens due to using
AC_CHECK_PROG instead of AC_CHECK_TOOL or because a Makefile
hardcodes use of pkg-config.
In general, the class of bugs that can be fixed in less than an hour
is still very much populated. But there are also a number of harder
issues.
* Making Debian "bootstrappable", i.e. making it possible to cross
compile a base system from source. This mostly requires adding build
profiles in the relevant places and largely is a solved problem. For
the embedded use case though, more profiles will be needed to
optionally disable unnecessary features (i.e. what Emdebian Crush
did). For instance, disabling selinux could noticeably reduce the
build time. For many architectures, cross boostrapping mostly
works[1] already.
* Simplifying foreign installations. As far as I can tell, the current
state of the art in installing packages still is multistrap. It
allows foreign package installation albeit leaving them unconfigured.
Configuration can be done using qemu-user-static or by rewriting the
relevant scripts. I think neither approach scales and we should move
to running standard maintainer scripts outside the chroot they are
operating on. That feature goes by the name DPKG_ROOT[2] and
currently, you can only experiment with it by passing
--force-script-chrootless to dpkg. Adding support for DPKG_ROOT to
maintainer scripts and figuring out its semantics is present work.
* Cross compiling to from linux to non-linux or from glibc to non-glibc
(e.g. musl) currently fails, because there are file conflicts in
headers. We'll have to move[3] standard headers (e.g. limits.h) to
/usr/include/<triplet>. The initial patches are there, but someone
needs to perform an archive rebuild to see what breaks.
I am also interested in hearing more from cross compilation users to be
able to use a better prioritization mechanism than popcon. Anyone
interested in working on these topic is invited to get in touch with me.
The preferred ways are the #debian-bootstrap IRC channel on OFTC and
debian-cross@lists.debian.org. I am not subscribed to d-embedded@l.d.o.
Helmut
[1] https://wiki.debian.org/HelmutGrohne/rebootstrap
[2] https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap
[3] https://wiki.debian.org/Multiarch/GlibcHeaders
Reply to: