Bug#804002: libxcb FTBFS on mips: cc1 segfaults building xkb.c

[Helmut Grohne <helmut@subdivi.de>]

On Wed, Nov 04, 2015 at 06:26:01AM +0100, Helmut Grohne wrote:
> | libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I../../src -D_FORTIFY_SOURCE=2 -Wall -Wpointer-arith -Wmissing-declarations -Wformat=2 -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Wbad-function-cast -Wold-style-definition -Wdeclaration-after-statement -Wunused -Wuninitialized -Wshadow -Wmissing-noreturn -Wmissing-format-attribute -Wredundant-decls -Wlogical-op -Werror=implicit -Werror=nonnull -Werror=init-self -Werror=main -Werror=missing-braces -Werror=sequence-point -Werror=return-type -Werror=trigraphs -Werror=array-bounds -Werror=write-strings -Werror=address -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -c xkb.c -o xkb.o
> | gcc: internal compiler error: Segmentation fault (program cc1)
> | Please submit a full bug report,
> | with preprocessed source if appropriate.
> | See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
> | Makefile:996: recipe for target 'xkb.lo' failed
> 
> The attached file xkb.pre.c is the result of running the same invocation
> past the preprocessor. Running gcc -g -O2 xkb.pre.c also gives that
> segfault. Dropping either -g or -O2 or using -O1 or -O3 makes it go
> away. Thus I cc debian-gcc@l.d.o. This is gcc-5 5.2.1-23.

I am attaching a reduced file crash.c that reproduces the crash with
just 28 lines and no includes. It seems that the bug is in the code
generator or optimizer, because we can only observe a crash for mips and
subtle changes make the issue disappear. Judging from a quick glance at
a traceback (w/o debug symbols), the issue at hand is a stack space
exhaustion of two mutually recursive functions. That's all I know for
now.

Helmut

int xcb_popcount(int mask);
int xcb_xkb_get_map_map_unpack (int *_buffer , int nTypes , int virtualMods , int totalVModMapKeys , int present , int *_aux)
{
	unsigned xcb_buffer_len = 0;
	unsigned xcb_block_len = 0;
	int xcb_pad = 0;
	int xcb_align_to = 0;
	int xcb_tmp_len = 0;
	if(present)
		xcb_block_len++;
	_buffer += xcb_tmp_len;
	if(present & 2)
	{
		xcb_buffer_len += xcb_block_len;
		xcb_popcount(virtualMods);
		xcb_block_len = 0;
	}
	if(present & 1)
	{
		xcb_pad = -xcb_block_len & (xcb_align_to - 1);
		xcb_buffer_len += xcb_pad;
		if (xcb_pad)
			_aux[2] = xcb_pad;
		xcb_block_len += totalVModMapKeys * 4;
	}
	xcb_buffer_len += xcb_block_len + xcb_pad;
	return xcb_buffer_len;
}