Bug#247300: libc6: malloc() never fails on 2.4 kernels, making processes crash

[Vincent Lefevre <vincent@vinc17.org>]

On 2004-05-06 13:45:45 +0900, GOTO Masanori wrote:
> IIRC, this kind of OOM (Out Of Memory) situation has been discussed at
> linux kernel related lists for a long time.  Your sample code repeats
> to issue mmap() and to fill the acquired pages until it got problem.
> So kernel needs to detect the phisycal memory exhaustion before
> allocating virtual memory.  Please imagine how to do it (to deal with
> vairous types of memories like mmap)?  Yeah, welcome to VM world.

The 2.4 kernel does have such a function:

int vm_enough_memory(long pages)

in mm/mmap.c; looking at strace output, the glibc calls old_mmap,
which has a call to do_mmap_pgoff, which contains the check here:

        /* Private writable mapping? Check memory availability.. */
        if ((vm_flags & (VM_SHARED | VM_WRITE)) == VM_WRITE &&
            !(flags & MAP_NORESERVE)                             &&
            !vm_enough_memory(len >> PAGE_SHIFT))
                return -ENOMEM;

But the check is done only when the MAP_NORESERVE flag isn't set.

> AFAIK, in kernel 2.6, there is strictly overcommit mode
> (/proc/sys/vm/overcommit_memory = 2) to prevent from getting sigkill
> by OOM killer and so on.  It helps you that malloc() should be return
> with the limit (see /proc/sys/vm/overcommit_ratio).
[...]
> I don't think it's glibc bug.  It's linux specification.

In the 2.4 kernel documentation, Documentation/sysctl/vm.txt says:

overcommit_memory:

This value contains a flag that enables memory overcommitment.
When this flag is 0, the kernel checks before each malloc()
to see if there's enough memory left. If the flag is nonzero,
the system pretends there's always enough memory.

Isn't it clear?

So, according to linux specification, the kernel does the check if
overcommit_memory is 0 (my case)... unless the caller asks not to
reserve (but malloc() is a reservation, so I don't see why glibc
sets the MAP_NORESERVE flag, if I've understood correctly).

> I would like to close this bug, ok? If you want to discuss about
> this issue more, I recommend you lists:
> linux-kernel/linux-mm/kernelnewbies that are good place.

Please, don't close the bug now. See my explanations above. It you
think that glibc is doing the right thing, then the source should
probably be more documented.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA