Bug#247300: libc6: malloc() never fails on 2.4 kernels, making processes crash

[Vincent Lefevre <vincent@vinc17.org>]

Package: libc6
Version: 2.3.2.ds1-12
Severity: grave
Justification: causes non-serious data loss

I've set the overcommit to 0, and the malloc() function never fails,
even when there isn't enough memory left, making processes crash when
they need memory they have already allocated.

I'm not sure whether this is a libc6 or a kernel bug. I'm not familiar
with the glibc source, but could this be due to the following code?

malloc/arena.c:

  /* A memory region aligned to a multiple of HEAP_MAX_SIZE is needed.
     No swap space needs to be reserved for the following large
     mapping (on Linux, this is the case for all non-writable mappings
     anyway). */
  p1 = (char *)MMAP(0, HEAP_MAX_SIZE<<1, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE);
                                                                ^^^^^^^^^^^^^

FYI, here's my test program:

/* $Id: malloc.c 3174 2004-04-28 14:44:41Z lefevre $
 *
 * malloc() test
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define ONEMB 1048576

int main (void)
{
  char *p;
  int i;

  for (i = 1; (p = malloc(ONEMB)) != NULL; i++)
    {
      printf ("Got %d MB\n", i);
      memset (p, 0, ONEMB);
    }
  printf ("malloc() failed - OK\n");
  return 0;
}

-- System Information:
Debian Release: testing/unstable
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing')
Architecture: i386 (i686)
Kernel: Linux 2.4.26
Locale: LANG=POSIX, LC_CTYPE=en_US.ISO8859-1

Versions of packages libc6 depends on:
ii  libdb1-compat                 2.1.3-7    The Berkeley database routines [gl

-- no debconf information