[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Implementing binary compatibility with Linux, *BSD, ...

[Sorry to mail to both lists, but the topic _is_ relevant in both cases.
 BTW, could we _please_ reunify both lists?]

Abstract: The following mail contains a proposal, discussion and technical
          description on implementing binary compatibility of the Hurd to
          binaries compiled for Linux-, *BSD- or other OS, without the need
          to recompile those binaries from scratch. With binary compat.,
          the Hurd servers can be installed in an existing OS distribution
          as a simple package and would be ready-to-run.

Thanks to your efforts, more and more programs are being ported to the Hurd
right now. As a consequence, the Debian/Hurd system is getting better all
the time. However, there are still many open issues like e.g. ppp, XFree86,
networking, ... which are not yet (fully) resolved.

While recompiling programs from source is definitly preferred (not only for
performance reasons, but also to help fix bugs and missing parts in the Hurd),
providing binary compatibility to say Linux-, *BSD- or programs compiled for
other OS can be also very useful:
  * not all programs are available in source code (alas!),
  * embedding the Hurd in existing OS (see below) can help avoid
    installing/building a distribution from scratch and may help
    get more users to give the Hurd a try inside their Linux/FreeBSD/other
  * implementing a dos emulator or something like this may also be easy to do.

There are currently some systems that are trying to emulate _binary_ compat.
to misc. OS:
  1. CPU-emulators a la bochs, vmware, ... booting an OS.
     [of course, they are too slow and quite "virtual" ;-)]
  2. Replacing dynamic libraries with own versions like in wine
     [this is restricted to dynamic linking and is only of limited use]
  3. Redirecting system call traps into the right library.
     This approach is taken by the linux(8) emulator of FreeBSD
     and by the Lites OS-Server in Mach.
     [It is of course only applicable to processors of the same type]

Of course, 3. is the right way to do it. One of the most amazing facts
about this emulation technique is, that the emulated programs run at
100% speed as long as they don't invoke a (real) system call [many calls
to the C- or other libraries don't involve a trap in the kernel at all].
There are however performance penalties associated with redirecting the
trap from kernel-mode back to a user-land library, which may herself RPC
the emulated-OS server(s?) if the answer can't be retrieved from a cache.
But we expect the binary emulation to be slower than native programs anyway,
don't we?

Let's consider this technique more closely. It's not really hard to

* A program requests the execution of another program through exec(), which
  resides in glibc and which is either directed to the Lites server (Lites)
  or to the exec server (Hurd).

* Lites or the exec server loads the program into the VM space of the
  requesting task as usual.

* Now, exec() [a.k.a Lites or the exec server to be more precise, please
  read appropriately subsequent references to exec()] examines the signature
  of the file to determine the OS that is needed. Depending on the type of
  OS, it links the appropriate emulation stubs to the binary and registers
  them with Mach. Should the binary trap later into Linux/BSD/..., Mach would
  redirect the trap to the emulation stub for further processing. Of course,
  native programs won't need any special treatment here.

* Finally, exec() arranges for the right loader to be called, depending
  on the type of OS. The *BSD loader would dynamically link the regular
  *BSD C- and other libs to the binary, the Linux loader would link the
  Linux C- and other libs etc... This can be done by dynamically rewriting
  the crt0.o part of the binary or by any other means.

* Now, exec() starts the first thread and lets the binary run.

* The binary runs merrily through its normal code, also descending into
  the libc of its own OS without interference from Mach or from Lites/Hurd.
  Please note that the code in the OS-dependent libc (and others) also
  doesn't require any intervention of Mach/Lites/Hurd, as long as no system
  call trap is required.

* Once the binary (usually from inside the OS-dependent libc) generates a
  *BSD-, Linux- or other OS trap, Mach catches this trap and redirects it
  to the emulation stub, that was previously linked to the binary by exec().
  [As far as Mach is concerned, the handling is over and the actual implemen-
  tation of the foreign OS system call is (mostly) handled in user-land
  by the stub and other servers.]

* The emulation stub tries to generate an answer from its cache if possible
  to avoid further overhead. One example of such cached result may be
  a second, third, ... call to getpid(). If the answer can't be obtained
  from the cache, the stub will have to direct the request to the appropriate
  OS-personality for further processing (e.g. open(), read(), ...).

* In the case of Lites, the emulation stub RPCs directly into the Lites
  server, which will take care of everything. In the Hurd, the stub could
  either direct each call (via Hurd's glibc) to the right server, or could
  RPC an OS-Service-Mapper-Server, which would itself call the right servers
  on behalf of the stub.

* The emulating OS-personality (be it Lites or the Hurd servers through
  Hurd/glibc) performs the (mapped) request on behalf of the foreign binary
  and returns a result to the stub. [footnote 1]

* The stub returns the result to the binary it is linked with, which also
  happens to be the binary that requested the service.

Please note that binaries _and_ the libraries of the emulated OS don't
have to be changed in any way! Actually, they can be borrowed directly from
an existing Linux-, *BSD- or other OS distribution.

The following work is needed to implement this kind of binary compatibility
in the Hurd:

0. Read Johannes Helander's Master's Thesis describing the design
   and implementation of the Lites server and emulator, then play with
   the RT-Mach system to get a feeling of Lites itself. Look at the
   sources of Lites, paying special attention to the emulation library.

1. Modify the exec server, so that it can recognize native- (Hurd), Linux-,
   FreeBSD- and other binaries by examining their signature.

2. Have the exec server add emulation stubs to the address space of the
   binary depending on the type of OS.

3. Write simple dummy stubs, that would simply print debug messages in
   the first place. Later, replace the dummies with real code as explained

4. Register the stubs with Mach, so that Linux-/*BSD/other traps are
   redirected to the stubs. Have the stubs print the kind of trap and
   its arguments as debug message.

5. Install the original dynamic libraries of Linux/FreeBSD/other somewhere.

6. Look at the emulation library of the Lites server. This library
   consist of real working stubs that RPC into the Lites server.
   Now, replace the RPC calls of the Lites server with simple calls to
   Hurd's glibc _as appropriate_. Note that most system calls of generic
   Unices like Linux and FreeBSD are already implemented in Hurd's glibc.

7. Modify the exec server, so that the right ld.so is called,
   according to the type of emulated OS (native Hurd, Linux, FreeBSD)
   and ascertain, that the right ld.so links the binary against the
   appropriate dynamic libraries that have been installed in step 5.

8. Test some foreign binaries from inside Debian/Hurd.

9. Install the so modified Hurd servers and libraries into an existing OS
   distribution (Debian, FreeBSD, SCO, Solaris x86, ...) as a package and
   try the binary compatibility by booting the Hurd, then diverting to the
   /etc/rc* procedures of the host OS (possibly slighly modified or enhanced).

Remaining problems:

  * Most OS provide specialized kernel device drivers, that are needed by
    some of their binaries (e.g.: lkm/kld, fb, scsi-passthrough [cdrecord!],
    tun [userland ppp!], bpf [snoop, tcpdump, libpcap!], accessing registers
    of hardware [(foreign) XFree86 and graphics adapters] ...). Mapping such
    drivers/hw-access into appropriate Hurd/Mach semantics can range from
    tricky to hairy. IMHO, they should be implemented mostly in user-land
    in a residual OS server, accessed by the stubs [see footnote 1].

  * Some OS provide/require advanced features like kernel threads (no problems
    with Hurd/Mach) and priority control, a.k.a. real time scheduling like
    in Solaris (use Hurd/RT-Mach for this). Some OS provide ABI/API support
    for multiple CPUs (no problems with Mach threads and processor sets).
    Other features may be harder up to impossible to emulate, if they can't
    be realized in the Hurd/Mach.
    [Examples please?]

  * Implementing binary compatibility for non-Unix systems can be quite hard
    if not downright impossible or inpractical. E.g. emulating DOS binaries
    will only work as long as they use well defined interfaces to access
    the system (int 21h, bios ints, ...). Such interfaces can be caught in
    an emulation library. Direct access to the hardware like writing into
    memory mapped registers and the like would be much harder to emulate
    correctly. Protected CPU instructions that differ from traps would be
    also quite hard to emulate correctly (possibly by having exec()
    dynamically rewrite the offending instruction in the binary, possibly
    converting it into a registered Mach trap? Hmmm...).

  * Some OS are not very well documented, concerning their behaviour. If
    you don't know the kind of traps and their meanings, it's quite hard to
    write an appropriate emulation stub and library. Just another reason
    to concentrate on free software anyway ;-).

  * With the proposed scheme, binaries from different OS can run at the
    same time within the Hurd (e.g. native-, Linux-, *BSD- and other
    binaries all running as Hurd processes). This is useful, practical and
    easy to implement. However, some OS _distributions_ (tools) can get
    confused if they see binaries in the (Hurd's) process list, that they
    can't handle. Just imagine an SCO kill(1) command sending a SCO
    specific signal to a Linux binary running in the Hurd! It must be
    investigated, wether each OS should have its own process list, or
    if a global process list (from proc server) is sufficient.

    I'd like to see most foreign binaries sharing the Hurd like other Hurd
    processes, so that they can e.g. communicate via shared memory, pipes,
    etc... as far as applicable. On the other hand, it may be useful to
    create an isolated environment (on demand) for a whole OS or single
    invocation of a binary, like in the jail(2) system call of FreeBSD.

  * <... add other problemes here ...>

Footnote 1: Not every foreign OS service can be directly mapped to an
existing Hurd service. In this case, the remaining functionality will
have to be implemented either in the stubs themselves (if they don't
require any global state) or in a small residual OS-Server (if special
global state is required). To give a concrete example: networking can
be performed directly by the Hurd with pfinet (Linux-based IP) or indirectly
in another OS-Server like Lites (4.4 BSD-Lite based IP). pfinet can be
considered a "residual OS-Server" in this context.

Discussion and feedback highly appreciated.

Anyone with enough time and resources to start working on this project?


Farid Hajji -- Unix Systems and Network Admin | Phone: +49-2131-67-555
Broicherdorfstr. 83, D-41564 Kaarst, Germany  | farid.hajji@ob.kamp.net
- - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - -
Murphy's Law fails only when you try to demonstrate it, and thus succeeds.

Reply to: