next generation apt/dpkg-cross
Hi,
I've been working on a next generation apt/dpkg-cross that will
support apt, aptitude, synaptic (anything libapt based) and dpkg
directly. And the upcoming multiarch changes. This involves a number
of changes. Some are in the interface and some just in the
implementation. I'm currently waiting for a patch in apt (support to
download index files for multiple architectures) before roling this
out but I heard the patch has been added to the apt multiarch branch
now. So that hurdle should go away any-day-now[tm].
So here is what you can look forward too if you want to switch from
the current apt/dpkg-cross to mine. Using arm is just an example (see
Configuration below).
Interface:
==========
- dpkg-cross just like you are used to.
- Wrappers in /usr/lib/apt-cross/. Add that dir to your PATH or link
them from /usr/local/bin and you can install foreign packages
transparently with just dpkg. This has some sideeffects so maybe don't
use this.
- apt-get, aptitude, synaptic (anything libapt based) are
automatically setup to support -cross packages (just needs
configuration to know which archs).
Using those you can manage your system just like you are use to. The
following will all work fine including for -cross packages:
apt-get update
apt-get upgrade
apt-get dist-upgrade
apt-get install foo libbar-arm-cross
dpkg[-cross] -i libbaz_1.2-3.arm.deb
For every library the -arm-cross package will apear in
aptitude or synaptic as well and you can simply select
them there.
Now for something new:
apt-get build-dep bash-arm-cross
This will install all the build-depends of bash but install the
libraries as arm-cross instead/on top of the native libs so you can
cross compile bash. This is still experimental but it looks like it
works well so far.
With some extra configuration you can also do:
apt-get install bzip2=1.0.5-3~i386.cross.1
This would install a 32bit bzip2 on amd64 including any needed
libraries. It requires a non-default configuration saying that it is
OK to have foreign binaries though and is only usefull for e.g. i386
binaries on amd64 where they can be executed. But you could use the
same with qemu to run them. Usefull to test a bug that only happens on
one architecture. But, let me repeat it, this will require extra
configuration to prevent accidentally replacing native binaries with
foreign ones.
Last there is also a tool (convert-cross) to convert deb packages
instead of installing them. The same mechanisms used during install
(see implementation below) are used to convert libfoo_1.2-3_arm.deb
into libfoo-arm-cross_1.2-3_all.deb (or a dummy
libfoo-arm-cross_1.2-3_arm.deb if multiarch). Those packages can then
be put into a repository and installed with plain apt and dpkg. Those
that don't like on-the-fly conversion can use that to get identical
results.
Configuration:
==============
Configuring this is real easy. You probably only need this in
/etc/apt/apt.conf[.d]:
APT::Architectures { "amd64"; "arm"; };
or
APT::Architectures:: "amd64";
APT::Architectures:: "arm";
That is usualy it. That automatically enables all apt sources for the
archs listed. But you can do more in /etc/apt/sources.list[.d]:
deb [arch=amd64,i386] http://ftp.somewhere.debian.org/debian stable main
deb [arch=arm] http://ftp.debian.org/debian stable main
This tells apt to get the amd64 packages from the "somwhere" mirror
while getting the arm packages from the main archive. With [arch=...]
you limit that source to the listed architectures. If no architectures
are listed apt will default to all configured architectures.
Note: All this is the pending patch for apt.
On top of that there are some conffiles users should not need to
touch:
/etc/apt-cross/black.list:
Filter blacklisting packages from conversion. By default this prevents
any binary available natively to be converted. It further explicitly
excludes dpkg, apt, aptitude, type-handling and a few more from ever
being converted. They would really mess up the system if you
tried. This is used only by "apt-get update".
/etc/apt-cross/rename.list:
List of patterns that decide what packages are libraries and what are
binaries. Also handles packages available natively under lib32* or
lib64* name (or other) where the native one will be prefered. E.g. on
amd64 libc6-i386 is to be used instead of libc6-i386-cross. This is
the only external and global information used.
The list currently contains less than 90 entries and has been
sufficient to run a number of 32bit apps on amd64. Use for cross
compiling will certainly produce a few more patterns but the list
should be pretty good already. Any change in this list might require
reconverting packages as this rules the renaming of package
relationships but package updates can handle that
automatically. Errors (missing entries) could be found automatically
by converting all of the debian archive once and checking where the
rename.list guesses wrong. But I haven't done that yet due to lack of
time and architectures on my mirror.
Implementation:
===============
The implementation of all this is a bit tricky and scary at first but
keep an open mind. The main idea is to change as little as possible
while getting all the features needed. All the changes are
concentrated in 2 places by adding a config sniplet to apt and
wrapping dpkg-deb. No changes are made to apt or dpkg itself. The only
reason dpkg needs a wrapper is so that it used the dpkg-deb wrapper.
Apt:
The before mentioned pending patch adds support to apt to download
index files for multiple architectures. This is required for multiarch
and the first step towards it. I rely on that patch because then
everything else can be done with a simple apt.conf.d sniplet. What the
sniplet does is process the downloaded Packages and Sources files
after apt has downloaded them and correct the package names and
package relationships (depends, conflicts, ...) so the foreign
packages will use libfoo-arm-cross names. If multiarch packages are
found they are left as is and a -arm-cross dummy package is added that
depends on them. This will only be done during "update". The other
thing the sniplet does is tell apt to use /usr/lib/apt-cross/dpkg as
dpkg backend. That way arm debs can actualy be installed.
Dpkg:
The wrapper around dpkg just adds /usr/lib/apt-cross/ to PATH so
/usr/lib/apt-cross/dpkg-deb is used to unpack debs.
Dpkg-deb:
Two actions in dpkg-deb need to be changed to handle foreign
architecture packages. Namely --fsys-tarfile and -e, --control.
Any other uses are passed straight through to dpkg-deb as is. In the
future other options might be handled as well but I would rather have
true multiarch by then.
With -e, --control the control.tar.gz is first unpacked and then the
control file is processed. The package name and package relationships
are changed just like for apt. So "dpkg -e libfoo_1.2-3_arm.deb" will
result in a DEBIAN/control file for libfoo-arm-cross that depends on
libbar-arm-cross and so on. The renaming happens like the old
dpkg-cross did with the changes recently mentioned about multiarch
packages. The big change is that this happens when the control.tar.gz
is unpacked by dpkg instead as seperate process between downloading
and installing. If it exists /usr/share/apt-cross/<package>.control is
executed to do extra modifications needed.
With --fsys-tarfile first the control.tar.gz file is unpacked and
checked if this is a native package, foreign package and if it is
multiarch or not. Native packages are left unchanged. Multiarch
foreign packages without -cross in the name are also left unchanged.
Multiarch foreign packages with -cross in the name are changed to
dummy packages just containing a changelog. Non multiarch foreign
packages are unpacked to a temporary directory and then files are
moved around. Include files to /usr/include/arm-linux-gnu, libraries
to /usr/lib/arm-linux-gnu, binaries thrown away and so on the same way
dpkg-cross moves files around now (except with multiarch dirs now).
If it exists /usr/share/apt-cross/<package>.data is executed to do
extra modifications needed.
Not all packages can be transformed fully automatically. Some need
some extra handholding. For those cases the <package>.control and
<package>.data hooks are available to do whatever custom
transformation they need. For example libgtk2.0-0 needs this for i386
debs on amd64 (old hook from ia32-libs):
# Fix path for plugins
sed -i 's,/usr/lib/,/usr/lib32/,' usr/lib32/gtk-2.0/2.10.0/immodule-files.d/libgtk2.0-0.immodules
sed -i 's,/usr/lib/,/usr/lib32/,' usr/lib32/gtk-2.0/2.10.0/loader-files.d/libgtk2.0-0.loaders
Those custom transformations are usualy only needed to run binaries of
that arch and not for cross compiling itself but I want to support
running 32bit apps on amd64 as well and it is trivial to
support. Writing the hooks is left to those that find they need them.
So that's it. Let the flame fest begin.
MfG
Goswin
Reply to: