Bug#210981: libc6: printf crashes on a certain long double

[Daniel Jacobowitz <dan@debian.org>]

Package: libc6
Version: 2.3.2-7
Severity: normal

Try it:

int data[] = {0x00000000, 0x00000000, 0x00000cd0};
struct ieee {
        unsigned int mantissa1:32;
        unsigned int mantissa0:30;
        unsigned int quiet_nan:1;
        unsigned int one:1;
        unsigned int exponent:15;
        unsigned int negative:1;
        unsigned int empty:16;
} *pieee = data;
long double *doub = (long double *) data;
int main()
{
  char *ret = 0;
  printf ("%.35Lg\n", *doub);
  vasprintf (&ret, "%.35Lg", doub);
  printf ("%s\n", ret);
}

That number is:

$2 = {mantissa1 = 0, mantissa0 = 0, quiet_nan = 0, one = 0, exponent = 3280, negative = 0, empty = 0}

which ought to be a perfectly valid floating point number, as far as I can
tell?

It crashes in __mpn_rshift.

726                           else if (BITS_PER_MP_LIMB - 1 - cnt_h <= cnt_l)
727                             {
728                               (void) __mpn_rshift (frac, tmp + i, tmpsize - i,
729                                                    BITS_PER_MP_LIMB - 1 - cnt_h);
730                               fracsize = tmpsize - i;
731                             }
732                           else
(gdb) p frac
$4 = (mp_limb_t *) 0xbfffe73c
(gdb) p *frac
$5 = 0
(gdb) p tmp + i
$6 = (mp_limb_t *) 0xbfffeb20
(gdb) p tmpsize - i
$7 = -450
(gdb) p cnt_h
$8 = 200


->   64 L(oop): movl    (%esi,%edx,4),%ebx      /* load next higher limb */
     65         shrdl   %cl,%ebx,%eax           /* compute result limb */
     66         movl    %eax,(%edi,%edx,4)      /* store it */

(gdb) p/x $esi
$5 = 0xbfffe418
(gdb) p/x $edx
$6 = 0x6fa
(gdb) p/x $esi + $edx * 4
$7 = 0xc0000000

i.e. it reads off the stack.

It looks like the problem is the fact that i == 665... but how it got there
involves a maze of twisty little gmp routines, and I have no idea.

-- System Information:
Debian Release: testing/unstable
Architecture: i386
Kernel: Linux nevyn 2.6.0-test4-nevyn #1 SMP Sat Aug 23 11:19:59 EDT 2003 i686
Locale: LANG=en_US, LC_CTYPE=en_US

Versions of packages libc6 depends on:
ii  libdb1-compat                 2.1.3-7    The Berkeley database routines [gl

-- no debconf information


-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer