Re: recent armel/armhf kernels ENOMEM on mprotect, maxima/axiom FTBFS

[Ian Campbell <ijc@hellion.org.uk>]

On Tue, 2014-08-19 at 18:50 -0400, Camm Maguire wrote:
> Greetings!
> 
> Ian Campbell <ijc@hellion.org.uk> writes:
> 
> > On Tue, 2014-08-19 at 14:47 -0400, Camm Maguire wrote:
> >> Greetings, and thanks so much for your feeback.
> >> 
> >> OK, I've chased this down, and on arm32 only, one can brk memory, but
> >> not mprotect it.
> >> 
> >> I also have a workaround, so this is not critical at the moment, but I
> >> think this is a kernel bug.
> >> 
> >> The last call below fails with ENOMEM:
> >
> > Have you confirmed that it is definitely a kernel change which has
> > caused this (as opposed to e.g. some user level change, ulimits, library
> > using 10x more RAM etc). Assuming so do you have any idea with which
> > kernel version this started happening?
> >
> 
> I have not tracked this to a kernel change.

OK. The "recent...kernels" in $subject made me think this had started
happening with a newer kernel. But it sounds like the kernel version is
irrelevant?

>   Rather a benign change in a
> long building program has exposed an existing issue, maybe a corner
> case, it appears.

What is the benign change? Perhaps it is not so benign after all?

> > Does it fail in a similar way (or invoke OOM etc) if you just touch
> > (e.g. write to) the memory instead of mprotecting it?
> >
> 
> I have not tried that, but can.  In general, I would expect in some
> cases that actual writes to brk'ed memory could trigger the oom killer
> given memory overcommit, but did not expect this on mprotect.  In any
> case, the OOM killer is not triggered, and mprotect simply returns -1
> with ENOMEM.  So you have brk'ed memory which you cannot mprotect, and
> the program continues to run.

I'd expect OOM to kick in too.

> > Are you able to share your reproducer program?
> >
> 
> Yes.  I have not boiled it down to a small test case, but this will do
> it reliably:

Thanks, I'll give this a try.

[...]
> and when run under strace -f will show the brk calls and mprotect calls
> which fail.  Oddly enough, when run under gdb, something is done to the
> runtime environment which prevents the failure from occurring, a mystery
> to me.

How strange!