Bug#247300: libc6: malloc() never fails on 2.4 kernels, making processes crash

[Wolfram Gloger <wg@malloc.de>]

Hi,

> Since I don't know enough to continue this discussion, I'm copying
> someone who does!

I hope I can clear this up..

> Wolfram, is there documentation describing the choice of MAP_NORESERVE
> in glibc's malloc, or do you know someone else I should ask?  Thanks in
> advance.

The intention here is to avoid overcommitment accounting for the
mapping with MAP_NORESERVE in malloc.  Since the prot argument for
this mapping is set to PROT_NONE, this is just a second line of
defense, however, as a mapping with this protection mode should not
count against any overcommitment accounting anyway.

I thought that Linux not implementing MAP_NORESERVE did not matter
because it looked at the prot argument before accounting, but I found
the MAP_NORESERVE flag useful e.g. on SGI.

So it's definitely intentional, but not _required_ for correct
malloc operation.  If the creation of a new arena fails, the main
arena is tried secondarily (in fact, this is like a third line
of defense).

But now concerning the bug report in question: I see no bug.  The
MAP_NORESERVE does not matter here at all.  Note that before malloc
hands out memory in a region allocated with MAP_NORESERVE, it _must_
call mprotect(..., PROT_READ|PROT_WRITE) on a smaller subregion, and
_that_ call definitely should be checked by the kernel against
overcommitment accounting, as _then_ (and only then) physical memory
really is potentially allocated.  I believe this to be the case in
Linux.

In general, if you want malloc to return NULL on Linux in a controlled
way, the best advice is to use "ulimit -v" IMHO.

Regards,
Wolfram.