Upcoming Debian multiarch support (amd64, sparc64, s390x, mips64) [affects sarge slightly]

[Goswin von Brederlow <brederlo@informatik.uni-tuebingen.de>]

Hi,

after fiddling around with the multiarch support for amd64 for some
time now it looks like our developement plans and ideas for amd64 (and
hopefully the rest too) have stabelized. Since this involves several
changes to key components I would like to give everyone the heads up
about what we try to do and how we plan to do it. One minor change to
the /var/lib/dpkg/status file I would like to get implemented and
included in sarge. 

Lets review the problem first: (I use mostly amd64 as an example but
the other archs are very similar)

On amd64 systems the architecture can run i386 just fine. But for some
reasons one might want to use packages compiled for the full
architecture. Most prominent reason would be speed (only amd64 in
general) and the larger address space of the 64 bit cpu. For similar
reasons - speed for non amd64 and code size - one might still want to
use the compatible older archs. There is also the problem of
availability of 3rd party software.

In the end the system has to support both i386 and amd64 compiled
binaries. This is different from i686 optimized binaries since the ABI
differs for i386 and amd64. You can link a i686 binary to a i386
library and vice versa. The same won't work with amd64 and i386. That
leads to the problem that at least the essential libraries must be
available in both i386 and amd64 format and be installable at the same
time.

The currently implemented idea was to rename the amd64 package of
libfoobar to lib64foobar and have amd64 binary packages depend on that
name instead. libfoobar.so goes to /lib and lib64foobar.so to
/lib64. That works so far. The same renaming happens to the
libfoobar-dev package giving lib64foobar-dev. As you might see now
nearly every package has to be changed to

 Build-Depend: libfoobar-dev [!amd64, !sparc64, !s390x, !mips64],
               lib64foobar-dev [amd64, sparc64, s390x, mips64]

Every lib has to be changed to build lib64foobar and lib64foobar-dev
packages for the new archs, which means changing the control file and
rules file drastically. Every versioned depends on a library has
to be changed too for the new lib64* name.

Renaming packages would work but causes a lot of work, build-failures
and broken packages. So, we asked, why do we have to rename packages
in the first place?

Currently dpkg identifies packages uniquely by its name alone. There
can never be two packages with the same name insalled at the same
time. This will now change. Packages that provide an ABI (indicated by
'ABI: strict' set in the control file) with the same name can be
installed provided the provided ABIs differ. Packages that provide an
ABI only match for a Depends (Conflicts) if the provided ABI fits the
architecture of the packages needing the Depends (Conflicts) has the
same ABI. Lets take an example:

Package: libc6          (1)
Architecture: i386
ABI: strict
Version: 3.2.3

Package: libc6          (2)
Architecture: i686
ABI: strict
Version: 3.2.3

Package: libc6          (3)
Architecture: i386
ABI: strict
Version: 4.5.6

Package: libc6          (4)
Architecture: amd64
ABI: strict
Version: 3.2.3

Only one of 1, 2 or 3 can be installed and 3 is the prefered one
(highest version). But 4 can be installed alone or in combination with
any one of 1, 2 or 3.

Package: make           (5)
Architecture: i386
Depends: libc6 (>= 2.3.2-1)

Package: make           (6)
Architecture: amd64
Depends: libc6 (>= 2.3.2-1)

Either 5 or 6 can be installed. But only 1, 2 or 3 fullfills the
Depends for 5 and only 4 fullfills the depends for 6.

The current libc6 package has to be patched to set "ABI: strict" in
its control file. But no changes have to be made to the existing make
source or binary package. Only a limited number of packages (libs that
need to be patched for amd64 anyway for other reasons) need to be
worked on and existing 3rd party products that Debian can't adapt for
amd64 can still be used. Compared to having to patch nearly every
package thats a big step forward. Further more the "ABI: strict" line
can be checked by the autobuilder easily (Is there any library in the
package but no ABI: strict line?).



What needs to changes for this to work?

The first thing that needs to happen is that the Architecture of each
installed package is recorded in the /var/lib/dpkg/status file. When
upgrading from an i386 to a multiarch amd64 system that information is
needed to figure out the correct depends. But at the stage where a
multiarch amd64 capable dpkg is installed and run for the first time
the status file would be polluted with amd64 packages already so
guessing the correct architecture in retrospect is difficult. It is a
minor change to dpkg to record the architecture of each deb in the
status file and I would realy like that to happen for sarge. It is
enough to record the architecture for any newly installed package or
dpkg could add the architecture to each entry from the available file
when missing. A possible "ABI: strict" line should also be kept in the
status file for much the same reasons. dpkg is already equiped to
handle extra lines in the status file it has no meaning for so
backward compatibility is not broken. Making sarge systems keep track
of the Architecture and ABI lines would greatly simplify upgrading a
sarge system to a sarge+1 multiarch system.

Any other changes can be done in debian-amd64s own dpkg package and be
merged into debian sid once sarge is out of the way. Apt and the
dpkg/apt frontends need to be adapted to multiarch. For all its much
the same change.

And the biggest job is changing every library package to build the
amd64 flavour and put its libraries into /lib64/. But that can be done
step by step as needed to compile binaries for amd64. The i386
binaries already work perfectly so there is no pressing need and for
non amd64 there isn't even much demand for it. Amd64 seems to be
faster with amd64 binaries (due to more registers than i386). The
other archs tend to be slower (due to greater code size). But anything
that could benefit from / need >4G ram, like mysql, should be made
available. Each time such a binary is found the libs needed can be
ported.


Affected packages:
(Entries marked with * are targeted for the sarge release, + already
have working experimental code)

dpkg: + support for multiple packages with the same name
      + support for multiple architectures and abis
      * keeping track of the architecture of each installed deb
      + autorename some package files or directories that otherwise
        collide (copyright, changelog, pre/postinst/rm of lib packages)
      - package:arch syntax

apt: - package:arch syntax
     +/- support for multiple binary-<arch> dirs
     + support for multiple architectures and abis
     - support for multiple packages with the same name
     - strict abi matching for libs, downloading the right abi

dselect, aptitude, ...: - support for multiple architectures and abis
                        - support for multiple packages with the same name

libfoobar.deb: - Add ABI: strict to control file (amd64 buildd will FTBFS
                 if missing to get things started)
               - split out any and all common files into a binary-all package
               - make -dev package binary-all or split it into
                 arch:any and arch:all
(The splitting out is needed if we rename or not so don't bitch about
that)


Small note on the package:arch syntax:

Apt has the following syntax to specify what "flavour" of package to
install:

apt-get install foobar/stable
apt-get install foobar=1.2.3

Now we add

apt-get install foobar:i386
apt-get install foobar:amd64

The same syntax (package:arch) will be added to the Depends,
Build-Depends, Conflicts, ... lines in the control file. The only
example of when this is currently needed is the build-essential
package. Build-essential has to pull in the right gcc/g++ packages,
which means 32 bit biarch compiler capable of compiling 32 and 64 bit
binaries. The 64bit gcc is not fit for official builds yet.


Ideas, complains, objections?

MfG
        Goswin