Re: Should .a library contains non-reallocatable code?

[Jeff Epler <jepler@unpythonic.net>]

On Thu, Feb 19, 2015 at 05:19:30PM -0600, Jeff Epler wrote:
>  * foomodule is a Python wrapper for libfoo, so it must be shipped
>    as a .so, but if it links libfoo.a, and libfoo.a is not -fPIC,
>    it is not possible to build foomodule at all
> 
>    (The same goes for wrapping the library for most other interpreted
>    languages)

So here is a concrete example of this.  I chose libtomcrypt at
semi-random, because in libtomcrypt-dev 1.17-6 there is both a shared
library and a static library; the latter has non-PIC code.

Begin with a Python module wrapping a single aspect of libtomcrypt:
generating a random byte string using the Fortuna algorithm:

    #include <Python.h>
    #include <tomcrypt.h>

    static PyObject *tom_random(PyObject *noself, PyObject *args) {
        Py_ssize_t sz;

        if(!PyArg_ParseTuple(args, "n:tom.random", &sz)) return NULL;
        PyObject *bytearray = PyByteArray_FromStringAndSize("", 0);
        if(!sz || !bytearray) { return bytearray; }

        prng_state prng;
        int err;
        if((err = rng_make_prng(128, find_prng("fortuna"), &prng, NULL)) != CRYPT_OK)
            return PyErr_Format(PyExc_RuntimeError,
                    "rng_make_rng() returned %d", err);

        if(PyByteArray_Resize(bytearray, sz) < 0) {
            Py_DECREF(bytearray);
            bytearray = NULL;
            goto done;
        }

        if((err = fortuna_read(PyByteArray_AsString(bytearray), sz, &prng))
                != sz) {
            Py_DECREF(bytearray);
            bytearray = NULL;
            return PyErr_Format(PyExc_RuntimeError,
                    "fortuna_read() only read %zd bytes", (size_t)sz);
        }

    done:
        fortuna_done(&prng);
        return bytearray;
    }

    static PyMethodDef meth[] = {
        {"random", (PyCFunction)tom_random, METH_VARARGS,
            "random(n): get N random bytes from a fortuna generator"},
        {}
    };

    void inittom() {
        int err;
        if((err = register_prng(&fortuna_desc)) == -1) {
            PyErr_Format(PyExc_RuntimeError,
                    "register_prng() returned %d", err);
            return;
        }

        PyObject *m = PyImport_AddModule("tom");
        Py_InitModule("tom", meth);
    }

Stir in an 'version' file that exports just the required symbol, inittom:

    {
    global: inittom;
    local: *;
    };

create a Python 2.7 extension module as follows:
    $ gcc -I /usr/include/python2.7 -fPIC -c -o tommodule.o tommodule.c
    $ gcc -shared -o tommodule.so tommodule.o -ltomcrypt -lpython2.7 \
        -Wl,--version-script=version -Wl,--no-allow-shlib-undefined

this .so file has just one exported symbol, thanks to the export file:
    $ nm -D --defined-only tommodule.so 
        0000000000000d0e T inittom
and happily, it works:
    $ python -c 'import tom; print bytes(tom.random(16)).encode("hex")'
    412d02b965f8c1f34c6bbaf2d1beb001

However, if libtomcrypt.a is used instead, the linker errors:
    $ gcc -shared -o tommodule.so tommodule.o \
        -Wl,-Bstatic -ltomcrypt -Wl,-Bdynamic -lpython2.7 \
        -Wl,--version-script=version -Wl,--no-allow-shlib-undefined
    /usr/bin/ld.bfd.real: /usr/lib/x86_64-linux-gnu/libtomcrypt.a(crypt_find_prng.o):
        relocation R_X86_64_32 against `prng_descriptor' can not be used when making a shared object; recompile with -fPIC
    /usr/lib/x86_64-linux-gnu/libtomcrypt.a: error adding symbols: Bad value
    collect2: error: ld returned 1 exit status

If libtomcrypt.a *had* been built with -fPIC, then this would have
worked.  Using the version file prevents *ANY* identifier from
libtomcrypt being a part of the public API of the Python module.  From
the point of view of the user of the Python tom module, it's a 
moot point whether libtomcrypt's C API is stable.

In fact, I went ahead and rebuilt it this way, with just a patch to
debian/rules:
    -export CFLAGS += -DGMP_DESC -DLTM_DESC -DUSE_LTM
    +export CFLAGS += -DGMP_DESC -DLTM_DESC -DUSE_LTM -fPIC
with this version of the package installed, the linker line which
referred to the .a file worked to produce a Python shared library.

All tests done on a Jessie x86-64 system.  I manually trancsribed some
of the commands above, so there's every chance I made an error in
transcrption.

Jeff