Bug#414045: debugging graphicsmagick-1.1.7 and/or libx11-1.0.3

[ldoolitt@recycle.lbl.gov]

Daniel -

For both the broken.xwd and broken2.xwd files in bug #414045,
the offending operation is in libx11-1.0.3/src/ImUtil.c:505
   dst++ = *src++;
and in fact it's the src pointer that is out of range.
This suggests it's "only" a DOS problem, or at worst an
information leak problem, but no direct exploit is possible.

A few lines earlier, the src pointer is computed as
      src = &ximage->data[ZINDEX(x, y, ximage)];
where ZINDEX is the macro
#define ZINDEX(x, y, img) ((y) * img->bytes_per_line) + \
    (((x) * img->bits_per_pixel) >> 3)

In the broken.xwd case, x = 0, y = 1838, ximage->bytes_per_line = 66148,
and ximage->bits_per_pixel = 24.  So it's no surprise that
attempting to read ximage->data[121580024] generates a segfault.

broken2.xwd is similar, but the overflow is in the x direction.

The call to XGetPixel in both cases is at
graphicsmagick-1.1.7/coders/xwd.c:388 .

I'll try to figure out which routine should check the indexes
(and against what).

   - Larry