Bug#841669: MAP_NORESERVE vs. ulimit (was: Re: Bug#841669: Acknowledgement (KDE fails to start on kernel 4.7.0-0.bpo.1))

[Alain Knaff <alain@knaff.lu>]

Hi,

After using logging in in "failsafe" mode, and using strace on kde, I 
found out what was going on.

The following call now fails with ENOMEM:

mmap(NULL, 2147483648, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0)

Older kernels did not enforce ulimit -d with MAP_NORESERVE (confirmed 
this on (3.13.0-100-generic), whereas new ones do:

> uname -a ; ulimit -d 64000; ./memalloc-test 69000000 
Linux lll 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Succeeded

> uname -a ; ulimit -d 64000; ./memalloc-test 69000000 
Linux hitchhiker 4.7.0-0.bpo.1-amd64 #1 SMP Debian 4.7.8-1~bpo8+1 (2016-10-19) x86_64 GNU/Linux
Malloc: Cannot allocate memory

I'm not sure where the bug actually is (kde or kernel), as indeed the 
case could be made that KDE should not attempt to reserve a ridiculous 
amount of memory if it never intends to use it (and it doesn't use it, 
or else it would eventually get a SIGSEV on older kernels).

Thanks,

Alain

#include <stdlib.h>
#include <stdio.h>
#include <sys/mman.h>

int main(int argc, char **argv) {
	void *r=mmap(0, atoi(argv[1]),
		     PROT_READ|PROT_WRITE,
		     MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE,
		     -1,
		     0);
	
	if(r != MAP_FAILED) {
		printf("Succeeded\n");
	} else {
		perror("Malloc");
		return -1;
	}
	return 0;
}