[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: multiarchitecture binaries - technical obstacles?

(Multiply replies collapsed and addressed)

Years ago, NeXT modified GCC and the rest of the GNU tools to allow
them to produce multi-architecture binaries, so that a single binary
executable could run on both 68k and i386 platforms.  They also had a
tool that could strip out hunks for unwanted architectures.

Most of the tools only have minimal changes to support fat builds above and beyond the necessary support for the Mach-O file format. For instance, when you invoke "cc -arch ppc -arch i386 -c test.c", the compiler backend is invoked twice (one is a native compiler, the other a cross compiler to x86). Each assembly source files is assembled separately (again using a native and a cross assembler), and then lipo(1) is used to join the two object files into a single Mach-O fat file (with a fat header). Most of the heavy lifting is just parsing the -arch flags and knowing the correct path to invoke, and when to
call lipo(1).

The current Mac OS X kernel, when preparing to execve() a file, reads in the Mach-O header. If the header is a fat header, one level of indirection is employed to find the
correct architecture and start executing the correct code.

If you are interested in where and how this code operates, you may want to look at some of the Darwin source code at <http://developer.apple.com/darwin/>. 'xnu' is the Mac OS X/Darwin kernel, 'gcc3' is the gcc 3.1-based compiler used for Mac OS X 10.2, 'cctools' includes
ld(1), as(1) and lipo(1).

Maybe that code has decayed, and maybe it had problems ... but it sure
would be nice if it could be resurrected and used by us, so that
Debian became able to produce multiplatform .deb containing such
multi-architecture executables.

It is alive and well in Mac OS X. The entire Darwin x86 release was cross compiled from Mac OS X 10.2. Are you suggesting adding support to multi-architecture ELF files to the Linux kernel, or just some metadata in .deb files such that only executables of the correct architecture are installed at install time?

Here are a few potential benefits: it would reduce the total volume of the archive;

I assume you mean because a lot of non-executable information like documentation and support files would not need to be replicated in a bunch of arch-specific debs. Keep in mind that there maybe be issues with non-executable data files that are endian-specific which would be implicitly correct during native building but perhaps not for cross building. I can't think of any examples of this offhand though.

eliminate separate i386/Pentium/686/MMX-optimized packages

Mach-O support generic architecture types as well as cpu subtypes. On Mac OS X, I've created a fat file with an object code for a G3 subtype and a G4 subtype. The kernel executes the most specific subtype, so a single executable does the right thing for each type. Arguably, this is less efficient than using primarily G3 code and just doing (ifG4) { g4optimized() } else { g3optimized() }, since the majority of the code need not be optimized.

reduce the burden on the autobuilders;

Using cross building like this would indeed allow a farm of x86 machines to compile all architectures, instead of needing to support lots of builder types, which may or may not be as stable as x86. This is quite a bit different than traditional cross-compiling, which generates code that can never execute on the host. With the Mac OS X toolchain, since intermediate and final binaries have a native portion as well as a cross portion, way more programs can be trivially cross compiled. For instance, consider a program like XFree86 which compiles lots of auxiliary tools to process the source, before actually invoking the compiler. This would be difficult to cross compile without hacking the build system to death to figure out what needs to be compiled native and what should be cross-compiled.

reduce cross-architecture version-skew; catch
cross-architecture compilation problems earlier; ensure widespread
availability of binary packages for multiple architectures; make the
upcoming deployment of competing 64-bit CPUs from Intel and AMD less
painful for our users; and be a unique feather in Debian's cap

These would be pretty cool advantages. Depending on the maturity of IA64 and x86-64 linux ports, it might be advantageous to defer using native builders.

Doing this might reduce the archive size, but it would increase
bandwidth use on the mirrors, since either way the packages would be

Not necessarily. You could build all packages fat, and then post-process them to thin out to all the various support architectures before uploading to mirrors. This way, you get the binary packages built without version skew, but the traditional delivery method is still used.

Shantonu Sen

Reply to: