Re: recent armel/armhf kernels ENOMEM on mprotect, maxima/axiom FTBFS
Greetings!
Ian Campbell <ijc@hellion.org.uk> writes:
> On Tue, 2014-08-19 at 14:47 -0400, Camm Maguire wrote:
>> Greetings, and thanks so much for your feeback.
>>
>> OK, I've chased this down, and on arm32 only, one can brk memory, but
>> not mprotect it.
>>
>> I also have a workaround, so this is not critical at the moment, but I
>> think this is a kernel bug.
>>
>> The last call below fails with ENOMEM:
>
> Have you confirmed that it is definitely a kernel change which has
> caused this (as opposed to e.g. some user level change, ulimits, library
> using 10x more RAM etc). Assuming so do you have any idea with which
> kernel version this started happening?
>
I have not tracked this to a kernel change. Rather a benign change in a
long building program has exposed an existing issue, maybe a corner
case, it appears.
> Does it fail in a similar way (or invoke OOM etc) if you just touch
> (e.g. write to) the memory instead of mprotecting it?
>
I have not tried that, but can. In general, I would expect in some
cases that actual writes to brk'ed memory could trigger the oom killer
given memory overcommit, but did not expect this on mprotect. In any
case, the OOM killer is not triggered, and mprotect simply returns -1
with ENOMEM. So you have brk'ed memory which you cannot mprotect, and
the program continues to run.
> Are you able to share your reproducer program?
>
Yes. I have not boiled it down to a small test case, but this will do
it reliably:
1) git clone git://git.sv.gnu.org/gcl.git
2) cd gcl/gcl
3) git checkout Version_2_6_11pre
4) add the lines
#ifdef NEED_STACK_CHK_GUARD
if (!raw_image) __stack_chk_guard=random_ulong();/*Cannot be safely set inside a function which returns*/
#endif
to main.c, right after
gcl_init_alloc(&argv);
5) debian/rules build-ansi-stamp
6) cd unixport
7) echo '(si::sgc-on t)(si::save-system "ff")' | ./saved_ansi_gcl
Now you have a program "ff" which when executed should print this:
mprotect failure: 0xd49000 305430528 : Cannot allocate memory
sgc disabled: Cannot allocate memory
and when run under strace -f will show the brk calls and mprotect calls
which fail. Oddly enough, when run under gdb, something is done to the
runtime environment which prevents the failure from occurring, a mystery
to me.
Take care,
> Ian.
>
>>
>> prot[2431,2437,(1),writable]
>> mprotect(0x97f000,0x6000), sbrk(0)=0x2cda000
>> prot[2437,2438,(0),not writable]
>> mprotect(0x985000,0x1000), sbrk(0)=0x2cda000
>> prot[2438,2439,(1),writable]
>> mprotect(0x986000,0x1000), sbrk(0)=0x2cda000
>> prot[2439,2472,(0),not writable]
>> mprotect(0x987000,0x21000), sbrk(0)=0x2cda000
>> prot[2472,2474,(1),writable]
>> mprotect(0x9a8000,0x2000), sbrk(0)=0x2cda000
>> prot[2474,2666,(0),not writable]
>> mprotect(0x9aa000,0xc0000), sbrk(0)=0x2cda000
>> prot[2666,2667,(1),writable]
>> mprotect(0xa6a000,0x1000), sbrk(0)=0x2cda000
>> prot[2667,2668,(0),not writable]
>> mprotect(0xa6b000,0x1000), sbrk(0)=0x2cda000
>> prot[2668,2733,(1),writable]
>> mprotect(0xa6c000,0x41000), sbrk(0)=0x2cda000
>> prot[2733,2881,(0),not writable]
>> mprotect(0xaad000,0x94000), sbrk(0)=0x2cda000
>> prot[2881,2884,(1),writable]
>> mprotect(0xb41000,0x3000), sbrk(0)=0x2cda000
>> prot[2884,2887,(0),not writable]
>> mprotect(0xb44000,0x3000), sbrk(0)=0x2cda000
>> prot[2887,2890,(1),writable]
>> mprotect(0xb47000,0x3000), sbrk(0)=0x2cda000
>> prot[2890,2921,(0),not writable]
>> mprotect(0xb4a000,0x1f000), sbrk(0)=0x2cda000
>> prot[2921,2959,(1),writable]
>> mprotect(0xb69000,0x26000), sbrk(0)=0x2cda000
>> prot[2959,3036,(0),not writable]
>> mprotect(0xb8f000,0x4d000), sbrk(0)=0x2cda000
>> prot[3036,3058,(1),writable]
>> mprotect(0xbdc000,0x16000), sbrk(0)=0x2cda000
>> prot[3058,3174,(0),not writable]
>> mprotect(0xbf2000,0x74000), sbrk(0)=0x2cda000
>> prot[3174,3207,(1),writable]
>> mprotect(0xc66000,0x21000), sbrk(0)=0x2cda000
>> prot[3207,3264,(0),not writable]
>> mprotect(0xc87000,0x39000), sbrk(0)=0x2cda000
>> prot[3264,3323,(1),writable]
>> mprotect(0xcc0000,0x3b000), sbrk(0)=0x2cda000
>> prot[3323,3325,(0),not writable]
>> mprotect(0xcfb000,0x2000), sbrk(0)=0x2cda000
>> prot[3325,3400,(1),writable]
>> mprotect(0xcfd000,0x4b000), sbrk(0)=0x2cda000
>> prot[3400,3586,(0),not writable]
>> mprotect(0xd48000,0xba000), sbrk(0)=0x2cda000
>> prot[3586,3587,(1),writable]
>> mprotect(0xe02000,0x1000), sbrk(0)=0x2cda000
>> prot[2431,3587,(3),writable]
>> mprotect(0x97f000,0x484000), sbrk(0)=0x2cda000
>> prot[2431,2437,(1),writable]
>> mprotect(0x97f000,0x6000), sbrk(0)=0x1501f000
>> prot[2437,2438,(0),not writable]
>> mprotect(0x985000,0x1000), sbrk(0)=0x1501f000
>> prot[2438,2439,(1),writable]
>> mprotect(0x986000,0x1000), sbrk(0)=0x1501f000
>> prot[2439,2472,(0),not writable]
>> mprotect(0x987000,0x21000), sbrk(0)=0x1501f000
>> prot[2472,2474,(1),writable]
>> mprotect(0x9a8000,0x2000), sbrk(0)=0x1501f000
>> prot[2474,2668,(0),not writable]
>> mprotect(0x9aa000,0xc2000), sbrk(0)=0x1501f000
>> prot[2668,2733,(1),writable]
>> mprotect(0xa6c000,0x41000), sbrk(0)=0x1501f000
>> prot[2733,2881,(0),not writable]
>> mprotect(0xaad000,0x94000), sbrk(0)=0x1501f000
>> prot[2881,2884,(1),writable]
>> mprotect(0xb41000,0x3000), sbrk(0)=0x1501f000
>> prot[2884,2887,(0),not writable]
>> mprotect(0xb44000,0x3000), sbrk(0)=0x1501f000
>> prot[2887,2890,(1),writable]
>> mprotect(0xb47000,0x3000), sbrk(0)=0x1501f000
>> prot[2890,2921,(0),not writable]
>> mprotect(0xb4a000,0x1f000), sbrk(0)=0x1501f000
>> prot[2921,2959,(1),writable]
>> mprotect(0xb69000,0x26000), sbrk(0)=0x1501f000
>> prot[2959,3036,(0),not writable]
>> mprotect(0xb8f000,0x4d000), sbrk(0)=0x1501f000
>> prot[3036,3058,(1),writable]
>> mprotect(0xbdc000,0x16000), sbrk(0)=0x1501f000
>> prot[3058,3174,(0),not writable]
>> mprotect(0xbf2000,0x74000), sbrk(0)=0x1501f000
>> prot[3174,3207,(1),writable]
>> mprotect(0xc66000,0x21000), sbrk(0)=0x1501f000
>> prot[3207,3264,(0),not writable]
>> mprotect(0xc87000,0x39000), sbrk(0)=0x1501f000
>> prot[3264,3323,(1),writable]
>> mprotect(0xcc0000,0x3b000), sbrk(0)=0x1501f000
>> prot[3323,3325,(0),not writable]
>> mprotect(0xcfb000,0x2000), sbrk(0)=0x1501f000
>> prot[3325,3397,(1),writable]
>> mprotect(0xcfd000,0x48000), sbrk(0)=0x1501f000
>> prot[3397,3398,(0),not writable]
>> mprotect(0xd45000,0x1000), sbrk(0)=0x1501f000
>> prot[3398,3399,(1),writable]
>> mprotect(0xd46000,0x1000), sbrk(0)=0x1501f000
>> prot[3399,3401,(0),not writable]
>> mprotect(0xd47000,0x2000), sbrk(0)=0x1501f000
>> prot[3401,77964,(1),writable]
>> mprotect(0xd49000,0x12343000), sbrk(0)=0x1501f000
>>
>> Take care,
>>
>>
>> Ian Campbell <ijc@hellion.org.uk> writes:
>>
>> > On Sun, 2014-08-17 at 11:29 -0400, Camm Maguire wrote:
>> >> Greetings! Recently, some change has been introduced into the 32bit arm
>> >> kernels (apparently) which is blocking maxima and axiom, and perhaps
>> >> others. These programs rely on mprotecting various pages read-only to
>> >> accelerate garbage collection, an algorithm which has worked on arm for
>> >> many years. Now, after a relatively few calls, mprotect returns ENOMEM,
>> >> indicating (perhaps) that the kernel table size for this information has
>> >> been decreased. When run under gdb, the calls succeed, which is a
>> >> mystery to me.
>> >>
>> >> In any case, I can turn this algorithm off if this is likely to be a
>> >> permanent condition on these machines. If this is rather a bug, please
>> >> let me know if I can assist in fixing it.
>> >
>> > I've not noticed anything of this nature going on (but I could easily
>> > have missed it) but I have a hard time imagining that this would be
>> > deliberate so my money would be on a bug of some sort (AFAIK there is no
>> > "kernel table size" as such since r/w is a bit in the actual page
>> > table).
>> >
>> > Have you confirmed that this is a kernel change rather than a change to
>> > maxima/axoim or some other change? Any idea with which version (or range
>> > of versions) it started?
>> >
>> > Ian.
>> >
>> >
>> >
>> >
>> >
>>
>
>
>
>
>
>
--
Camm Maguire camm@maguirefamily.org
==========================================================================
"The earth is but one country, and mankind its citizens." -- Baha'u'llah
Reply to: