[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Strange bug, help needed



Hi,

I am hitting a very strange bug, and my knowledge is limited to
understand all the ramifications of the issue.

The bug itself, and how to reproduce it:
Run iceweasel (3.0~rc2-2) and keep the window open
Run iceweasel again, which will open a new window in the existing
iceweasel, and... segfault.
I only could reproduce this on amd64 (and not on i386, though i haven't
tested other architectures). I couldn't first reproduce this in a clean
chroot, until I installed bug-buddy (though removing it afterwards
doesn't make it work again. Some dependency bug-buddy pulls must be
catalyzing the problem.
When first debugging this, I could see the crash happening after main()
was returned from..
After installing libc6-dbg and a DEB_BUILD_OPTIONS=nostrip rebuilt
libselinux1, I could get a somewhat significant backtrace:
  #0  0x00007f9d67339160 in ptmalloc_init () from /lib/libc.so.6
  #1  0x00007f9d6733ce37 in memalign_hook_ini () from /lib/libc.so.6
  #2  0x00007f9d67d437d7 in __tls_get_addr () from /lib64/ld-linux-x86-64.so.2
  #3  0x00007f9d6010329d in fini_context_translations () at setrans_client.c:211
  #4  0x00007f9d600f657f in __do_global_dtors_aux () from /lib/libselinux.so.1
  #5  0x0000000000401110 in operator new () #6  0x00007fff6ff4f210 in ?? ()
  #7  0x00007f9d60104ab1 in _fini () from /lib/libselinux.so.1
  #8  0x0000000000000053 in ?? ()
  #9  0x00007f9d67d40e04 in _dl_fini () from /lib64/ld-linux-x86-64.so.2

setrans_client.c:211 reads:
  free(prev_r2t_trans);

Now, it may seem strange that malloc is being initialized at exit()
time, and that is part of what is strange with this bug.

The fact is, xulrunner-stub, which is what iceweasel runs, actually,
re-executes itself with
LD_PRELOAD=/usr/lib/iceweasel/xulrunner/libjemalloc.so.
The same crash happens if I directly link libjemalloc to xulrunner-stub,
revove the re-execution hack and run it with the appropriate
LD_LIBRARY_PATH.
So this is not LD_PRELOAD related.

So, for some reason, libselinux.so.1 is trying to run free() from libc
instead of libjemalloc.

libselinux.so.1, like most things using malloc/free, expect them with a
symbol version, set by the glibc:
$ objdump -T /lib/libselinux.so.1  | grep ' free$'
0000000000000000      DF *UND*  00000000000000d9  GLIBC_2.2.5 free

And libjemalloc exports....
$ objdump -T /usr/lib/iceweasel/xulrunner/libjemalloc.so  | grep .text
00000000000059e0 g    DF .text  0000000000001474  Base        realloc
0000000000002df0 g    DF .text  0000000000000162  Base        malloc_usable_size
0000000000007400 g    DF .text  0000000000000534  Base        malloc
00000000000056f0 g    DF .text  00000000000002ed  Base        free
0000000000007f50 g    DF .text  0000000000000603  Base        valloc
00000000000051b0 g    DF .text  0000000000000540  Base        posix_memalign
0000000000006e60 g    DF .text  0000000000000593  Base        calloc
0000000000007940 g    DF .text  0000000000000603  Base        memalign

I tried building libjemalloc with a version-script so that all these
would get GLIBC_2.2.5 version. It still segfaulted.

Then I tried with LD_DEBUG=all to try to see why selinux would get its
free() symbol from libc...

$ grep 'normal symbol `free' /tmp/ld.so.log
     10925:     binding file /lib/libc.so.6 [0] to /bin/sh [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /lib64/ld-linux-x86-64.so.2 [0] to /bin/sh [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /lib/libc.so.6 [0] to /bin/sh [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /lib/libc.so.6 [0] to /lib/libc.so.6 [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /lib64/ld-linux-x86-64.so.2 [0] to /lib/libc.so.6 [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /lib/libc.so.6 [0] to /lib/libc.so.6 [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /lib/libc.so.6 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /lib64/ld-linux-x86-64.so.2 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /lib/libc.so.6 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /lib/libdl.so.2 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libpcre.so.3 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libexpat.so.1 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libstartup-notification-1.so.0 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libglib-2.0.so.0 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libcairo.so.2 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/iceweasel/xulrunner/libxul.so [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /lib/libselinux.so.1 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libnspr4.so.0d [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/iceweasel/xulrunner/libxul.so [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libstdc++.so.6 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libglib-2.0.so.0 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libX11.so.6 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libxcb.so.1 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libXau.so.6 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libgnutls.so.26 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libxml2.so.2 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/libICE.so.6 [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]
     10925:     binding file /usr/lib/iceweasel/firefox-bin [0] to /usr/lib/iceweasel/xulrunner/libjemalloc.so [0]: normal symbol `free' [GLIBC_2.2.5]

The 6 first lines are insignificant, because they happen before exec()s.
The rest show libselinux is supposed to have got the free symbol from
jemalloc... (note this log was done with original libjemalloc.so,
without symbol versioning).

Sadly, from that point, I don't know what to look for... would you have
ideas ?

Note that before reaching that point, I also found the reason of the
segfault, though I don't understand why.

The assembly for the address where it crashes is:
  0x00007f9d67339160 <ptmalloc_init+384>: cmpb   $0x4d,(%rax)

And %rax points to an address in the range where
/usr/lib/xulrunner-1.9/libxul.so used to be mmap()ed. Except at the
moment the segfault occurs, libxul.so is *not* mmap()ed, because it was
dlclose()d: xulrunner-stub dlopen()s it on startup and dlclose()s it
at the end of its main().

Why would ptmalloc_init() try to read memory in mmap()ed libxul.so ?

Mike

PS: Please Cc me, I'm not subscribed.


Reply to: