On Thu, 15 Oct 2009 21:23:04 +0100
Colin Watson <cjwatson@ubuntu.com> wrote:
> Package: dpkg-dev
> Version: 1.15.4
> Severity: normal
> User: ubuntu-devel@lists.ubuntu.com
> Usertags: origin-ubuntu karmic
>
> The PKG_CONFIG_LIBDIR override introduced for cross-compilation
> purposes due to #439979 (so courtesy-copying Neil as the reporter of
> that bug, although ISTR he's trying to reduce his involvement with
> Emdebian at the moment) has a somewhat awkward consequence. I'm not
> sure I have a good solution for this right now, but would like to
> have a bug open for discussion.
Day-to-day cross-building for Emdebian has stalled, not just because of
my reduced involvement but also because the cross-building system
has critical problems and the solution means migrating to a fully
operational multiarch - after Squeeze. The system used for Emdebian
Crush 1.0 (based on Lenny) had severe limitations (principally in
correct handling of build-dependencies) and can take us no further.
http://lists.debian.org/debian-embedded/2009/08/msg00005.html
How will multiarch affect pkg-config? I'm not sure that the detail has
been decided yet, so CC:'ing debian-embedded for info.
FTR there are other cases that are similar - pango and gtk both build
tools that are expected to be run during the build and these tools were
simply not built (or at least not executed) in the cross-builds, using
Emdebian-specific patches where necessary, causing a loss of
functionality. I haven't been able to solve that problem (but there
were workarounds for Crush 1.0 that relied on copying the data files
from other systems instead of generating new ones. Not ideal.)
> It's true that setting PKG_CONFIG_LIBDIR when cross-compiling makes
> it a lot easier to build the majority of packages that use pkg-config
> in simple ways. You ask pkg-config for libraries, it automatically
> looks them up for the host architecture, and your job's done.
PKG_CONFIG_LIBDIR was not the complete solution but it did/does do a
good job for most packages that use pkg-config. There were still
workarounds needed for some packages (using path mangling with gccross)
and dbus was one of these packages. Emdebian used a field in the
debian/xcontrol file (X-Build-Cross-Libtool=yes) that changed the PATH
so that the cross-build always used the right compiler. In effect, this
prevented dbus from building or executing the binding-tool using the
build compiler and only allowed the host compiler to be "seen".
http://www.emdebian.org/emdebian/intro.html#gccross
i.e. this is a known bug that I haven't been able to solve without a
truly horrible hack (much worse than Colin's suggestion) and I fully
agree that it is a corner case of the more general solution.
Nevertheless, dbus seemed to work at runtime in Emdebian Crush 1.0 -
whether that was by design or just because the limited package
selection in Emdebian Crush 1.0 simply didn't ask dbus to do much, I
wouldn't like to say.
Sadly, with the current state of cross-building for Emdebian, it is
hard to test fixes until multiarch is at least partially in place.
> However, consider the case where you need to build a binary for use
> during the build that needs to link against external
> build-architecture libraries that should be discovered using
> pkg-config. An example of this is dbus-glib, which builds a binary
> (dbus-binding-tool) both for use during its build and to ship in one
> of its .debs. This binary uses the dbus-1 and gobject-2.0 pkg-config
> modules, linking against the corresponding libraries. In the
> cross-compilation case, we clearly need to build this binary twice -
> but now we need to persuade pkg-config to give us build libraries!
>
> I've approached this in a hacky fashion in my experiments by using
> things like this in configure.ac:
>
> DBUS_CFLAGS_FOR_BUILD=`unset PKG_CONFIG_LIBDIR; pkg-config --cflags
> dbus-1`
>
> The problem with this is that it's horribly Debian-specific
However, the use of PKG_CONFIG_LIBDIR is also Debian-specific
(specifically cross-building in Debian specific). Other teams
cross-building dbus are likely to be using qemu or other staging
methods where at least some of the build is actually achieved on the
embedded device or something approaching compatibility with the
device. I'm not sure that fixing this upstream would actually make any
difference to those who cross-build the upstream dbus using non-Debian
methods or staging methods involving things like qemu or
builds-on-the-device.
The only team trying to cross-build Debian packages using dpkg-dev and
not using emulators or requiring access to the device until after the
installation tarball has been created, is Emdebian. This is principally
because Emdebian seeks to create a binary distribution (i.e.
Debian-like) where the packages and the installation image can be
prepared independently. (Using staging tends to require that you build
all your packages yourself and then build one specific installation
from that one build, making all build errors fatal for the installation
testing and fatally undermining reproducibility testing of the final
packages. Debugging embedded stuff is hard enough without every user
having different builds of all packages.)
>, and it
> would really be easier for everyone if this kind of change (which I am
> able to implement in Autoconf/Automake without resorting to
> packaging-level changes, although it's not all that pretty) could go
> upstream.
A Debian-specific fix would seem appropriate to me. There are a lot of
corner cases right now; dbus, pango and gtk are ostensibly similar
cases, gcc and perl are much harder ones to solve (but gcc is a
higher priority because not every cross-built system needs a GUI or
dbus but everything needs libgcc1, sadly, whilst Crush 1.0 did without
perl entirely - reasonably well too).
Other corner cases may well crop up during the adoption of multiarch.
Once there is a working implementation, we can see about spreading
multiarch to various upstreams as a complete task instead of the
historical patchwork of incremental hacks.
> The standard upstream approach to this kind of thing is
> documented in /usr/share/doc/pkg-config/README:
>
> Users can define the PKG_CONFIG environment variable to point at the
> right one, or if they cross-compile and have a correctly named
> pkg-config (eg. arm-linux-pkg-config) in their PATH that will be used
> in preference.
>
> So, really what it seems that we ought to do is require
> $host-pkg-config on all cross-building systems just like we require
> $host-gcc etc. and then *not* set PKG_CONFIG_LIBDIR; maybe as a
> fallback we could set PKG_CONFIG_LIBDIR in the event that
> $host-pkg-config is not on the path. With that done, it would be
> possible to use $host-pkg-config and pkg-config depending on whether
> cross or native builds are needed, the way it was clearly intended to
> work upstream.
I can't remember now quite why this didn't originally work (it may
simply have been that Emdebian didn't try this level of change) but
with all the changes needed for the adoption of multiarch, I think it
is time to ditch all workarounds (like dpkg-cross mangling of .pc files
and PKG_CONFIG_LIBDIR in dpkg and gccross in emdebian-tools) and
attempt to implement a new, generalised, solution. True, this may
result in a new set of corner cases and a new set of workarounds,
but I think it's a net gain.
It should be possible to
implement /usr/bin/arm-linux-gnueabi-pkg-config as a shell script
provided by a package that is made into a dependency of the
cross-building environment (ala binutils-multiarch). Indeed, the new
package could simply divert /usr/bin/pkg-config and use
dpkg-architecture values to detect which environment to set in the
wrapper with $host-pkg-config being a symlink - possibly even generated
in the postinst or at runtime so that new triplets can be supported
cleanly.
The binary pkg-config-multiarch package could be built from the
dpkg-cross source package to make it easier to keep it in step with
supported triplets or it could be part of dpkg-dev from the start.
(dpkg-cross will still need to be merged into dpkg-dev eventually, the
question is how much of it is to be thrown away first. Most of it,
hopefully.)
> I'm not sure how to get there from here, though. Perhaps get
> cross-targeted pkg-config packages onto buildd.emdebian.org and allow
> a while for people to install them?
I think this needs to be tied in with the other changes being worked on
for the .pc files themselves in order to achieve multiarch support.
It is achievable - it would also provide a possible home for changes to
pkg-config that could make it unnecessary to modify the .pc files in
the first place.
It's not just autotools either, CMake needs similar changes:
The main changes involve setting PKG_CONFIG_LIBDIR within CMake an
setting the include directories to locate the cross libraries installed
by dpkg-cross e.g.
#Make pkg-config look in the right place
SET(ENV{PKG_CONFIG_LIBDIR} ${CMAKE_FIND_ROOT_PATH}/lib/pkgconfig/)
ELSE (CMAKE_CROSSCOMPILING)
dpkg-cross (1)
Those changes could also be made in the multiarch pkg-config wrapper.
A little test and unpacking:
# diff -u usr/lib/pkgconfig/dbus-glib-1.pc usr/arm-linux-gnueabi/lib/pkgconfig/dbus-glib-1.pc
--- usr/lib/pkgconfig/dbus-glib-1.pc 2009-09-29 23:39:48.000000000 +0000
+++ usr/arm-linux-gnueabi/lib/pkgconfig/dbus-glib-1.pc 2009-10-16 08:17:10.000000000 +0000
@@ -1,4 +1,4 @@
-prefix=/usr
+prefix=/usr/arm-linux-gnueabi
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include
Multiarch support already requires that this behaviour is modified. It
may be useful to implement /usr/bin/arm-linux-gnueabi-pkg-config as a
wrapper for /usr/bin/pkg-config that ensures that prefixes and libdir
settings are correctly set *only* within that wrapper.
The wrapper could have support for particular options / environment
variables that support overriding the detection of the build and host
arguments.
> A further wrinkle is that $host-pkg-config will indeed be
> automatically used for anyone who's using the PKG_CHECK_MODULES
> Autoconf macro (which I suspect is the majority, but certainly won't
> be everyone), but not for anyone who's just calling pkg-config
> directly.
A wrapper that knows about dpkg-architecture should be able to handle
that.
> Should we just say that if they're doing that then they
> need to compare build and host architectures directly, the way
> they're probably already doing for gcc and friends? Would this be a
> serious regression for Emdebian that would need to be tackled in
> advance?
Emdebian Crush (the cross-built distribution) is already in a serious
regression but this provides an opportunity for major changes whilst
everything is "up for grabs".
Fixing this will not cause any regression in Emdebian that is not
already present - the implementation of multiarch is the ideal time
to do "TheRightThing".
--
Neil Williams
=============
http://www.data-freedom.org/
http://www.linux.codehelp.co.uk/
http://e-mail.is-not-s.ms/
Attachment:
pgpYi9shfOAVA.pgp
Description: PGP signature