Implementing binary compatibility with Linux, *BSD, ...
[Sorry to mail to both lists, but the topic _is_ relevant in both cases.
BTW, could we _please_ reunify both lists?]
Abstract: The following mail contains a proposal, discussion and technical
description on implementing binary compatibility of the Hurd to
binaries compiled for Linux-, *BSD- or other OS, without the need
to recompile those binaries from scratch. With binary compat.,
the Hurd servers can be installed in an existing OS distribution
as a simple package and would be ready-to-run.
Thanks to your efforts, more and more programs are being ported to the Hurd
right now. As a consequence, the Debian/Hurd system is getting better all
the time. However, there are still many open issues like e.g. ppp, XFree86,
networking, ... which are not yet (fully) resolved.
While recompiling programs from source is definitly preferred (not only for
performance reasons, but also to help fix bugs and missing parts in the Hurd),
providing binary compatibility to say Linux-, *BSD- or programs compiled for
other OS can be also very useful:
* not all programs are available in source code (alas!),
* embedding the Hurd in existing OS (see below) can help avoid
installing/building a distribution from scratch and may help
get more users to give the Hurd a try inside their Linux/FreeBSD/other
systems.
* implementing a dos emulator or something like this may also be easy to do.
There are currently some systems that are trying to emulate _binary_ compat.
to misc. OS:
1. CPU-emulators a la bochs, vmware, ... booting an OS.
[of course, they are too slow and quite "virtual" ;-)]
2. Replacing dynamic libraries with own versions like in wine
[this is restricted to dynamic linking and is only of limited use]
3. Redirecting system call traps into the right library.
This approach is taken by the linux(8) emulator of FreeBSD
and by the Lites OS-Server in Mach.
[It is of course only applicable to processors of the same type]
Of course, 3. is the right way to do it. One of the most amazing facts
about this emulation technique is, that the emulated programs run at
100% speed as long as they don't invoke a (real) system call [many calls
to the C- or other libraries don't involve a trap in the kernel at all].
There are however performance penalties associated with redirecting the
trap from kernel-mode back to a user-land library, which may herself RPC
the emulated-OS server(s?) if the answer can't be retrieved from a cache.
But we expect the binary emulation to be slower than native programs anyway,
don't we?
Let's consider this technique more closely. It's not really hard to
understand:
* A program requests the execution of another program through exec(), which
resides in glibc and which is either directed to the Lites server (Lites)
or to the exec server (Hurd).
* Lites or the exec server loads the program into the VM space of the
requesting task as usual.
* Now, exec() [a.k.a Lites or the exec server to be more precise, please
read appropriately subsequent references to exec()] examines the signature
of the file to determine the OS that is needed. Depending on the type of
OS, it links the appropriate emulation stubs to the binary and registers
them with Mach. Should the binary trap later into Linux/BSD/..., Mach would
redirect the trap to the emulation stub for further processing. Of course,
native programs won't need any special treatment here.
* Finally, exec() arranges for the right loader to be called, depending
on the type of OS. The *BSD loader would dynamically link the regular
*BSD C- and other libs to the binary, the Linux loader would link the
Linux C- and other libs etc... This can be done by dynamically rewriting
the crt0.o part of the binary or by any other means.
* Now, exec() starts the first thread and lets the binary run.
* The binary runs merrily through its normal code, also descending into
the libc of its own OS without interference from Mach or from Lites/Hurd.
Please note that the code in the OS-dependent libc (and others) also
doesn't require any intervention of Mach/Lites/Hurd, as long as no system
call trap is required.
* Once the binary (usually from inside the OS-dependent libc) generates a
*BSD-, Linux- or other OS trap, Mach catches this trap and redirects it
to the emulation stub, that was previously linked to the binary by exec().
[As far as Mach is concerned, the handling is over and the actual implemen-
tation of the foreign OS system call is (mostly) handled in user-land
by the stub and other servers.]
* The emulation stub tries to generate an answer from its cache if possible
to avoid further overhead. One example of such cached result may be
a second, third, ... call to getpid(). If the answer can't be obtained
from the cache, the stub will have to direct the request to the appropriate
OS-personality for further processing (e.g. open(), read(), ...).
* In the case of Lites, the emulation stub RPCs directly into the Lites
server, which will take care of everything. In the Hurd, the stub could
either direct each call (via Hurd's glibc) to the right server, or could
RPC an OS-Service-Mapper-Server, which would itself call the right servers
on behalf of the stub.
* The emulating OS-personality (be it Lites or the Hurd servers through
Hurd/glibc) performs the (mapped) request on behalf of the foreign binary
and returns a result to the stub. [footnote 1]
* The stub returns the result to the binary it is linked with, which also
happens to be the binary that requested the service.
Please note that binaries _and_ the libraries of the emulated OS don't
have to be changed in any way! Actually, they can be borrowed directly from
an existing Linux-, *BSD- or other OS distribution.
The following work is needed to implement this kind of binary compatibility
in the Hurd:
0. Read Johannes Helander's Master's Thesis describing the design
and implementation of the Lites server and emulator, then play with
the RT-Mach system to get a feeling of Lites itself. Look at the
sources of Lites, paying special attention to the emulation library.
1. Modify the exec server, so that it can recognize native- (Hurd), Linux-,
FreeBSD- and other binaries by examining their signature.
2. Have the exec server add emulation stubs to the address space of the
binary depending on the type of OS.
3. Write simple dummy stubs, that would simply print debug messages in
the first place. Later, replace the dummies with real code as explained
below.
4. Register the stubs with Mach, so that Linux-/*BSD/other traps are
redirected to the stubs. Have the stubs print the kind of trap and
its arguments as debug message.
5. Install the original dynamic libraries of Linux/FreeBSD/other somewhere.
6. Look at the emulation library of the Lites server. This library
consist of real working stubs that RPC into the Lites server.
Now, replace the RPC calls of the Lites server with simple calls to
Hurd's glibc _as appropriate_. Note that most system calls of generic
Unices like Linux and FreeBSD are already implemented in Hurd's glibc.
7. Modify the exec server, so that the right ld.so is called,
according to the type of emulated OS (native Hurd, Linux, FreeBSD)
and ascertain, that the right ld.so links the binary against the
appropriate dynamic libraries that have been installed in step 5.
8. Test some foreign binaries from inside Debian/Hurd.
9. Install the so modified Hurd servers and libraries into an existing OS
distribution (Debian, FreeBSD, SCO, Solaris x86, ...) as a package and
try the binary compatibility by booting the Hurd, then diverting to the
/etc/rc* procedures of the host OS (possibly slighly modified or enhanced).
Remaining problems:
* Most OS provide specialized kernel device drivers, that are needed by
some of their binaries (e.g.: lkm/kld, fb, scsi-passthrough [cdrecord!],
tun [userland ppp!], bpf [snoop, tcpdump, libpcap!], accessing registers
of hardware [(foreign) XFree86 and graphics adapters] ...). Mapping such
drivers/hw-access into appropriate Hurd/Mach semantics can range from
tricky to hairy. IMHO, they should be implemented mostly in user-land
in a residual OS server, accessed by the stubs [see footnote 1].
* Some OS provide/require advanced features like kernel threads (no problems
with Hurd/Mach) and priority control, a.k.a. real time scheduling like
in Solaris (use Hurd/RT-Mach for this). Some OS provide ABI/API support
for multiple CPUs (no problems with Mach threads and processor sets).
Other features may be harder up to impossible to emulate, if they can't
be realized in the Hurd/Mach.
[Examples please?]
* Implementing binary compatibility for non-Unix systems can be quite hard
if not downright impossible or inpractical. E.g. emulating DOS binaries
will only work as long as they use well defined interfaces to access
the system (int 21h, bios ints, ...). Such interfaces can be caught in
an emulation library. Direct access to the hardware like writing into
memory mapped registers and the like would be much harder to emulate
correctly. Protected CPU instructions that differ from traps would be
also quite hard to emulate correctly (possibly by having exec()
dynamically rewrite the offending instruction in the binary, possibly
converting it into a registered Mach trap? Hmmm...).
* Some OS are not very well documented, concerning their behaviour. If
you don't know the kind of traps and their meanings, it's quite hard to
write an appropriate emulation stub and library. Just another reason
to concentrate on free software anyway ;-).
* With the proposed scheme, binaries from different OS can run at the
same time within the Hurd (e.g. native-, Linux-, *BSD- and other
binaries all running as Hurd processes). This is useful, practical and
easy to implement. However, some OS _distributions_ (tools) can get
confused if they see binaries in the (Hurd's) process list, that they
can't handle. Just imagine an SCO kill(1) command sending a SCO
specific signal to a Linux binary running in the Hurd! It must be
investigated, wether each OS should have its own process list, or
if a global process list (from proc server) is sufficient.
I'd like to see most foreign binaries sharing the Hurd like other Hurd
processes, so that they can e.g. communicate via shared memory, pipes,
etc... as far as applicable. On the other hand, it may be useful to
create an isolated environment (on demand) for a whole OS or single
invocation of a binary, like in the jail(2) system call of FreeBSD.
* <... add other problemes here ...>
--------------
Footnote 1: Not every foreign OS service can be directly mapped to an
existing Hurd service. In this case, the remaining functionality will
have to be implemented either in the stubs themselves (if they don't
require any global state) or in a small residual OS-Server (if special
global state is required). To give a concrete example: networking can
be performed directly by the Hurd with pfinet (Linux-based IP) or indirectly
in another OS-Server like Lites (4.4 BSD-Lite based IP). pfinet can be
considered a "residual OS-Server" in this context.
Discussion and feedback highly appreciated.
Anyone with enough time and resources to start working on this project?
-Farid.
--
Farid Hajji -- Unix Systems and Network Admin | Phone: +49-2131-67-555
Broicherdorfstr. 83, D-41564 Kaarst, Germany | farid.hajji@ob.kamp.net
- - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - -
Murphy's Law fails only when you try to demonstrate it, and thus succeeds.
Reply to: