[sprint] openembedded / dpkg-cross-buildpackage hybrid
ok, i've been working with openembedded continuously now for a good
couple of weeks, so have a better handle on what it is, can, and
cannot solve. it's definitely a competitor to the "dpkg-cross build
system in combination with buildd", and is in many ways superior to
it.
a comment was made the last time i mentioned the idea of combining
openembedded with dpkg-buildpackage, which was, to roughly quote,
"openembedded is source-based whereas debian is binary-based". i made
a rebuttal of this at the time but i had no idea that openembedded
actually creates -dev packages "by default" as well as creating
cross-compile corresponding "-dev" packages at the same time!
so, going through it:
* debian maintainers "upload" completed packages, from across the
world. openembedded you put the rules/recipes into a git repository,
and absolutely anyone can build an entire image, from scratch (with
patience...). this key difference is a major cause of confusion and
belief that the two systems are too wildly disparate.
* debian produces .deb "dev" packages. so does openembedded.
* debian produces .deb "doc" packages. so does openembedded.
* debian produces .deb "dbg" packages. so does openembedded.
* debian produces .dev (binary) packages. so does openembedded.
* debian has a bootstrap process. openembedded doesn't exactly, but
you could create a basic image that would be the equivalent.
* debian has a _package_ called debootstrap. nobody in their right
mind actually bootstraps openembedded to build openembedded inside
openembedded in order to _get_ that far! (ok, nobody's ever tried it:
all the distros so far created are, duh, embedded ones)
* (someone please correct me if i'm wrong on this) - debian does NOT
have an automated "compile-from-scratch" system to create bootstrap
images (but it does have buildd). openembedded does.
* debian does NOT have an automated "cross-compile-from-scratch"
system to create bootstrap images. openembedded does.
the penultimate item is not so much of a serious problem: the
bootstrapping which was originally "unpack this .tgz" from 15+ years
ago has allowed people to, over time, create a bootstrap install (by
hand, initially? of which there are only about 13 anyway) for new
architectures.
the latter item _is_ a serious problem, for multi-arch and weird
hardware capability combinations not yet imagined. as mentioned
previously the sheer number of combinations means that whatever you
specify as the defaults (-no-neon, -thumb, -no-thumb, -mmx, -no-mmx)
it's GOING to be wrong.
so, basically, openembedded "has it all" - the only bit that's missing
is a capability to understand debian/control and debian/rules files,
and the task-running-dependency system would be perfect for building
the packages, staging the -dev packages in a cross-compile area so
that the dependent .debs can then proceed: off it would trundle; at
the end you'd have not only a complete set of .debs but also a
bootstrap image, debian-installer image, vmlinuz, initrd and u-boot
files, all done.
issues:
* openembedded's dependency system is a bit... quirky. as it reads
all the .bb files under the sun (i'm looking at 8,000 right now) it's
a leeetle time-consuming to do even the most basic extensions to
parsing, so they have avoided allowing in regex matching on the
dependencies like the plague. however, i believe that something like
the debian dependency format - package-dev >= N.N.N - should not be
spectacularly onerous. right now the way to specify that a build is
to be dependent on one particular version of a library is to
"blacklist" all the other versions of the same library! that's...
truly dreadful :)
* openembedded's build system doesn't have anything like the sandbox
build (using debootsrap / chroot) that debian typically uses. ok,
correction: nobody _wants_ to use it when doing cross-compiles that
can take potentially days as it is. however, this is merely
"procedure" - it isn't something that's hard-coded into bitbake or
openembedded "thou shalt not do chroots for builds".
rough principle:
* once bitbake has been modified to understand versioned dependencies,
a build would, instead of reading .bitbake files, check,
update/download an apt-cache repository and INTERNALLY create bitbake
recipe objects based not on .bb files but on the debian package
information.
* some new tasks would be created which understand debian/rules and
can issue dpkg-buildpackage and/or dpkg-cross-buildpackage commands to
complete SUB-tasks (not the entire dpkg-buildpackage "run") that
correspond to present openembedded tasks (download, patch, configure,
compile, install, staging) etc. the reason for this is to potentially
allow overrides in a .bb file (which will then eventually go into the
debian package instead) and also to run intermediate automated tasks
such as "sanity checks" to be done on the configure, compile and
install stages.
this latter is very very important: i've had my bacon saved more than
once even in the short time i've been using openembedded, on
discovering LDFLAGS "--no-install" in a Makefile.am which buggers up
the binary with a hard-coded path (one of the many qa sanity checks
checks rpath in binaries to make sure that they don't contain
/home/...oe/staging/....{libraryname}/.libs")
* recipes would be created which create a debian-installer image and a
debootstrap image, and any others required.
* especially needed would be a special task to unpack and chroot a
debootstrap image for a particular package which is known to be
problematic, or just to ensure reliability/repeatability.
advantages:
* openembedded is full of tricks such as replacement libtools and
replacement pkgconfig scripts which understand cross-compiling: the
replacement pkgconfig script can even create simultaneous equivalents
of dpkg-cross {libraryname}.pc files as well as a {libraryname}.pc
file which will end up in the corresponding "-dev" package
* whilst it's not a panacea, many of the equivalent openembedded
packages to debian packages are pretty much up-to-date and have had
the cross-compilation problems *solved* already. including tracking
down very specific combinations of options that will only work on a
particular CPU, for that specific package. so by "getting involved"
with openembedded, debian developers will not exactly have an
automated multi-arch cross-build system "dropped into their laps" but
they will at least have recipes with associated patches that are known
and proven to work, from which it will be possible to adapt the debian
ones to do the same sort of things. (there's strong evidence to show
that openembedded borrows a lot of expertise from debian - packages
which use debian-maintainer-created patches - i'm just surprised that
it's not the other way round as well.)
disadvantages:
* bitbake is _quirky_ :) recipes are a hybrid mix of shell script,
python, makefile-like variable declaration rules, and can inherit
"classes" and declare "tasks" which end up being inserted into the
chain and executed on the package!
* despite cacheing, it's the nature of having to deal with so many
recipes that the initial startup can be truly dreadfully slow (8,000
recipes) especially if the cache has to be thrown away for any reason.
but, if you think about it: no average developer really does complete
from-scratch debian cross-compiles (you always use the debootstrap and
go from there) because that debootstrap was done manually _once_
anyway! so when you tackle this mammoth task, what would you expect?
:)
i'm racking my brains to think of any other down-sides... nnope. i
think "bitbake is quirky" covers it :)
ok - enough! thoughts anyone?
l.
Reply to: