Bug#640080: xvfb: Xvfb segfaults when a bit depth other than 8, 15, 16, 24, or 30 requested

[Robert Jacobs <debbtsb06@eamp.org>]

Package: xvfb
Version: 2:1.11.0-1
Severity: normal

If Xvfb is started with a bit depth of anything other than 8, 15, 16,
24, or 30, it segfaults.

This appears to be due to a missing case in
InitOutput.c:vfbScreenInit. At line 832 there's a "switch
(pvfb->depth)" and thus explicit support for only the aforementioned
display depths. When a bit depth other than the above is requested,
the micmap.c:miVisuals array is never filled with useful values and a
null pointer is dereferenced (micmap.c:miInitVisuals at the end where
it says depth[i].vids[j]). 

I'd prefer some attempt to pick sane defaults when the user requests a
nonconventional bitdepth, but erroring out is definitely better than a
segfault. 

Obviously this is not very important -- after all, no one seems to
have mentioned it before -- and the only mode I was actively thinking
about wanting was the 1bpp mode for testing. But since Xvfb is
mentioned as also being used for "testing clients against unusual
depths", it would be nice to actually support said unusual depths. 

Thank you!

 - Robert Jacobs

p.s. Adding support for 1bpp seems to just involves adding:
    case 1:
        miSetVisualTypesAndMasks (1,
                                  (1 << StaticGray),
                                  1, StaticGray, 0, 0, 0);
        break;

I also have tested the following code for some sort of default for all
other depths, but it's rather ugly:
   default:
	    if (pvfb->depth > 8) {
		    int bluemask = (1 << (pvfb->depth/3)) - 1;
		    int greenmask = ((1 << ((2+pvfb->depth)/3)) - 1) 
			    << (pvfb->depth/3);
		    int redmask = ((1 << pvfb->depth)-1) & (~bluemask) & (~greenmask);
		    miSetVisualTypesAndMasks 
			    (pvfb->depth,
			     ((1 << TrueColor) |
			      (1 << DirectColor)),
			     (2+pvfb->depth)/3>8?(2+pvfb->depth)/3:8, 
			     TrueColor, redmask, greenmask, bluemask);
	    } else if (pvfb->depth < 8) {
        miSetVisualTypesAndMasks (pvfb->depth,
                                  ((1 << StaticGray) |
                                   (1 << GrayScale) |
                                   (1 << StaticColor) |
                                   (1 << PseudoColor) |
                                   (1 << TrueColor) |
                                   (1 << DirectColor)),
                                  8, PseudoColor, 0, 0, 0);
	    }
	    break;


-- System Information:
Debian Release: wheezy/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 3.0.0-1-686-pae (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages xvfb depends on:
ii  libaudit0       1.7.18-1  
ii  libc6           2.13-18   
ii  libgcrypt11     1.4.6-9   
ii  libpixman-1-0   0.22.2-1  
ii  libselinux1     2.1.0-1   
ii  libxau6         1:1.0.6-3 
ii  libxdmcp6       1:1.1.0-3 
ii  libxfont1       1:1.4.4-1 
ii  xserver-common  2:1.11.0-1

Versions of packages xvfb recommends:
ii  xauth  1:1.0.6-1

xvfb suggests no packages.

-- no debconf information