fopen(), libc, pthread override nightmare!

[Julien PERVILLE <julien.perville@free.fr>]

Hello list, and please forgive me for posting only an hour after
registring to this mailing list. I promise that I did my homework, and
searched for all the information I could find on my bug before posting
here.

I have been puzzled all this week by the strange behavior of the libc
fopen() on my ibook. Trying to overload fopen() in a shared library just
hangs at the first call to the original library function. This behavior
only happens on my stock debian 3.0 powerpc, using glibc 2.2.5.

The system lives inside a chroot in a gentoo host, and inherits its
2.6.8 kernel.

I managed to replicate the bug with the simpliest test library possible
: an LD_PRELOAD wrapper on fopen() (source below). What did I do wrong?

* Test it with almost every binary, it works as it should : print a
message before calling original fopen().
* Test with /bin/ls (which is linked with libpthread unlike most of
/bin) it hangs, it will never return from the first call to the original
fopen() function.

Relevant part of strace on my example wrapper :

----
write(1, "*wrapping done\n", 15*wrapping done
)        = 15
write(1, "*about to call original fopen on"..., 43*about to call
original fopen on /etc/mtab
) = 43
rt_sigprocmask(SIG_SETMASK, NULL, [32], 8) = 0
rt_sigsuspend([] <unfinished ...>
----

The hang is to be blocked on the re_sigsuspend.

A funny thing, if I test on a directory with not too many entries (up to
7 entries in a directory not counting . and ..), it won't try to fopen()
anything, and it won't hang. Trying to do it in any directory with
enough entries, it will try to fopen() /etc/mtab and others. And it will
hang at the first call to original fopen(). Why would the number of
entries matter...

I tested the exact same source on the gentoo side of the computer with
bleeding edge libc, works just fine. Tested it with debian stable on
x86, works just fine. Tested on debian testing x86. Works. Mandrake's ls
binary doesn't try to fopen() anything so I can't test . The bug only
happens for me on debian stable for ppc.

I tried to find an elegant solution for days now, I can't find anything
satisfying. I searched for that problem and the closest post I can find
about that behavior is
http://lists.debian.org/debian-powerpc/2003/05/msg00329.html, posted
more than one year ago. It seems that pthread is complicating the job of
making preload libraries, but I couldn't find why a simple wrapper to
the original function would fail if linked with pthreads.


If anyone can help me figure out an elegant solution, or even better if
there is bug hidden, help get it fixed, that would make my day.

I know that my project (an ld_preload library to do substitutions on
paths passed to filesystem related functions of libc) will be mostly
used on x86 debian stable, and x86 debian stable is working properly,
but I'm developping on powerpc and I'd rather avoid to have to code ugly
workarounds.


Thank you!

Julien Pervillé
julien.perville@free.fr

note : before posting this, after thinking one more hour : could this
"bug" be the result of the untested relation between a 2.6 kernel from
the gentoo host, and the "old" (as in 2.4 supported only) user land in
debian stable? especially concerning the thread model on the 2.6 kernel.
I looked for infos on this, but can't find much. I'm going to try out a
2.4 kernel see what happens. I'll keep you informed. Julien

ps : Simon I CC this mail to you, you had a similar problem in the past,
did you find a solution?

-----
/*
 * my simpliest test preload library. call it testopen.c :
 *
 * compile with :
 * $ gcc -D_GNU_SOURCE -fpic -shared -o testopen.so testopen.c -ldl
 *
 * and test it with :
 * LD_PRELOAD=/path/to/testopen.so /bin/ls
 */

#include <stdio.h>
#include <dlfcn.h>

static FILE * (*next_fopen)(const char *path, const char *mode);
FILE * fopen(const char *path, const char *mode)
{
  char *msg;

  if (next_fopen == NULL) {
    printf("*fopen : wrapping fopen\n");
    next_fopen = dlsym(RTLD_NEXT, "fopen");
    if ((msg=dlerror())!=NULL)
      printf("**fopen dlopen failed : %s\n", msg);
    printf("*wrapping done\n");
  }

  printf("*about to call original fopen on %s\n", path);
  return next_fopen(path, mode);
}
----