Bug#259887: gcc-3.3: Miscompiles automatic dynamic arrays

[<herbert@gondor.apana.org.au>]

Package: gcc-3.3
Version: 1:3.3.4-3
Severity: critical

With the option -mpreferred-stack-boundary=2, gcc 3.3.4 is miscompiling
automatic dynamic arrays.  Unfortunately both are used in the
crypto/IPsec subsystems of the Linux kernel.

Here is a sample program:

#include <string.h>

int bar(char *s);

int foo(char *s, int len, int x)
{
	char buf[x ? len : 0];

	if (x) {
		memcpy(buf, s, len);
		s = buf;
	}

	return bar(s);
}

With gcc 3.3.4, this produces:

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
	.file	"b.c"
	.text
	.p2align 4,,15
.globl foo
	.type	foo, @function
foo:
	pushl	%ebp
	xorl	%eax, %eax
	movl	%esp, %ebp
	subl	$24, %esp
	movl	16(%ebp), %ecx
	movl	%edi, -4(%ebp)
	movl	12(%ebp), %edx
	movl	%esp, %edi
	movl	%ebx, -12(%ebp)
	movl	%esi, -8(%ebp)
	decl	%edx
	movl	8(%ebp), %esi
	testl	%ecx, %ecx
	setne	%al
	decl	%eax
	orl	%eax, %edx
	addl	$19, %edx
	andl	$-4, %edx
---------------------------------------------------------------------
	subl	%edx, %esp
	leal	27(%esp), %ebx
	andl	$-16, %ebx

Note the offset 27.  The same program when compiled with gcc 3.2.3
produces similar output but it uses an offset of 15.

Suppose that len = 16, x != 0, and %esp & 15 = 8 before the subl.

That means %edx = (15 + 19) & ~3 = 32.  So %esp & 15 is still 8
after the subtraction.  That is, %esp = 16x + 8.  Hence
%ebx = (%esp + 27) & ~15 = (16x + 35) & ~15 = 16x + 32 = %esp + 24.

Therefore buf will only contain 8 bytes of space instead of 16
bytes.
---------------------------------------------------------------------
	testl	%ecx, %ecx
	jne	.L5
.L4:
	movl	%esi, (%esp)
	call	bar
	movl	%edi, %esp
	movl	-12(%ebp), %ebx
	movl	-8(%ebp), %esi
	movl	-4(%ebp), %edi
	movl	%ebp, %esp
	popl	%ebp
	ret
	.p2align 4,,7
.L5:
	movl	12(%ebp), %eax
	movl	%esi, 4(%esp)
	movl	%ebx, %esi
	movl	%eax, 8(%esp)
	movl	%ebx, (%esp)
	call	memcpy
	jmp	.L4
	.size	foo, .-foo
	.section	.note.GNU-stack,"",@progbits
	.ident	"GCC: (GNU) 3.3.4 (Debian 1:3.3.4-3)"
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Since this bug can lead to remotely triggered crashes and possibly
exploits I'm rating it as critical.

-- System Information
Debian Release: testing/unstable
Kernel Version: Linux gondolin 2.4.26-1-686-smp #1 SMP Sat May 1 19:17:11 EST 2004 i686 GNU/Linux

Versions of the packages gcc-3.3 depends on:
ii  binutils       2.14.90.0.7-8  The GNU assembler, linker and binary utiliti
ii  cpp-3.3        3.3.4-1        The GNU C preprocessor
ii  gcc-3.3-base   3.3.4-1        The GNU Compiler Collection (base package)
ii  libc6          2.3.2.ds1-13   GNU C Library: Shared libraries and Timezone
ii  libgcc1        3.3.4-1        GCC support library