Re: More than 2, but less than 3 GiB per process memory?
Hello,
Ron Johnson wrote:
> I get the same message at 3057MB. Even
>
> This test has to be bogus, though, because my RAM+swap is
> less than 2GiB.
>
> This lets me alloc a 1650MB on a desktop w/ 1GB RAM, 1GB swap,
> GNOME 2.10, lots of pages open in Firefox, etc.
>
> [code that dirties allocated memory via memset]
I've done some further research and found out a few things. Just allocating
memory in 1MB-sized blocks with malloc works up to the amount of free address
space (which is 4 GiB - 1 GiB for the kernel - a few bytes for program code
and shared libraries and used heap/stack etc. That is, roughly 3 GiB of free
address space. Of course, this is overcommitted, otherwise it wouldn't work
on systems with RAM/swap less than that).
The 1650MB amount you get with your program should be consistent with what the
free command shows (Free mem without buffers/cache plus free swap).
With my test program, I found out I can allocate approx. 1.8 Gig, and then
again about 750 MB. So, I can have > 2 GiB allocated, though not with a
single malloc/new invocation. I tried watching the address space of this new
test program. You can find out about the memory map/address space of a
process by cat'ing /proc/pid/maps.
no allocation done yet:
08048000-08049000 r-xp 00000000 03:0b 132427 /home/mc/memcheck-layout
08049000-0804a000 rw-p 00000000 03:0b 132427 /home/mc/memcheck-layout
41000000-41015000 r-xp 00000000 03:0b 164627 /lib/ld-2.3.5.so
41015000-41017000 rw-p 00014000 03:0b 164627 /lib/ld-2.3.5.so
41019000-4114a000 r-xp 00000000 03:0b 392874 /lib/tls/libc-2.3.5.so
4114a000-4114b000 r--p 00130000 03:0b 392874 /lib/tls/libc-2.3.5.so
4114b000-4114e000 rw-p 00131000 03:0b 392874 /lib/tls/libc-2.3.5.so
4114e000-41151000 rw-p 4114e000 00:00 0
b7f49000-b7f4a000 rw-p b7f49000 00:00 0
b7f61000-b7f63000 rw-p b7f61000 00:00 0
bfe4d000-bfe63000 rw-p bfe4d000 00:00 0 [stack]
ffffe000-fffff000 ---p 00000000 00:00 0 [vdso]
1.8 GB allocated:
08048000-08049000 r-xp 00000000 03:0b 132427 /home/mc/memcheck-layout
08049000-0804a000 rw-p 00000000 03:0b 132427 /home/mc/memcheck-layout
41000000-41015000 r-xp 00000000 03:0b 164627 /lib/ld-2.3.5.so
41015000-41017000 rw-p 00014000 03:0b 164627 /lib/ld-2.3.5.so
41019000-4114a000 r-xp 00000000 03:0b 392874 /lib/tls/libc-2.3.5.so
4114a000-4114b000 r--p 00130000 03:0b 392874 /lib/tls/libc-2.3.5.so
4114b000-4114e000 rw-p 00131000 03:0b 392874 /lib/tls/libc-2.3.5.so
4114e000-41151000 rw-p 4114e000 00:00 0
4caab000-b7f4a000 rw-p 4caab000 00:00 0
b7f60000-b7f63000 rw-p b7f60000 00:00 0
bfe4d000-bfe63000 rw-p bfe4d000 00:00 0 [stack]
ffffe000-fffff000 ---p 00000000 00:00 0 [vdso]
Additional 750 MB allocated:
08048000-08049000 r-xp 00000000 03:0b 132427 /home/mc/memcheck-layout
08049000-0804a000 rw-p 00000000 03:0b 132427 /home/mc/memcheck-layout
144be000-41000000 rw-p 144be000 00:00 0
41000000-41015000 r-xp 00000000 03:0b 164627 /lib/ld-2.3.5.so
41015000-41017000 rw-p 00014000 03:0b 164627 /lib/ld-2.3.5.so
41019000-4114a000 r-xp 00000000 03:0b 392874 /lib/tls/libc-2.3.5.so
4114a000-4114b000 r--p 00130000 03:0b 392874 /lib/tls/libc-2.3.5.so
4114b000-4114e000 rw-p 00131000 03:0b 392874 /lib/tls/libc-2.3.5.so
4114e000-41151000 rw-p 4114e000 00:00 0
4caab000-b7f4a000 rw-p 4caab000 00:00 0
b7f60000-b7f63000 rw-p b7f60000 00:00 0
bfe4d000-bfe63000 rw-p bfe4d000 00:00 0 [stack]
ffffe000-fffff000 ---p 00000000 00:00 0 [vdso]
Reduced to the interesting parts:
no allocation:
b7f49000-b7f4a000 rw-p b7f49000 00:00 0
1.8 GB allocated:
4caab000-b7f4a000 rw-p 4caab000 00:00 0
additional 750 MB allocated:
144be000-41000000 rw-p 144be000 00:00 0
[...]
4caab000-b7f4a000 rw-p 4caab000 00:00 0
So, malloc appears to only use contiguos areas of address space with a single
invocation, but allocating more memory in a different "address space hole"
the second time around works fine.
There is some interesting discussion to be found on the kernel mailing list:
http://www.ussg.iu.edu/hypermail/linux/kernel/0002.1/0005.html
One solution to my problem seems to be the 4G/4G patch by Ingo Molnar, which
gives me another GiB of address space previously used by the kernel:
https://lwn.net/Articles/39283/
While the announcement does not actually mention the benefit for the address
space, this is later clarified in more recent versions of the patch itself.
http://people.redhat.com/mingo/4g-patches/4g-2.6.6-B7 :
> [...] There's also a 33% increase in user-space VM size - database
> applications might see a boost from this.[...]
Unfortunately, somewhat recent discussions at
http://www.ussg.iu.edu/hypermail/linux/kernel/0405.2/1424.html
indicate that the patch won't go in. It might be a workaround for desperate
people; maybe we'll even have to fall back to that third party kernel patch
ourselves, but hopefully not. And the effect on malloc has not yet been
tested by me. I'd first have to forward-port the patch to Debian's 2.6.8
Linux kernel.
The question remains why the address space is split like it is. That is, the
program starts at 0x08000000, while the shared libraries are at 0x4100000.
So, approx. 900 MiB can be allocated here. Then, there is free address space
again after the shared libraries, up to approx. 0xb7000000. This is the
largest contiguous amount of free address space (ca. 2 GiB) and is what
limits my malloc test program, I suppose.
After various stuff and the stack, the kernel window into the address space
begins at 0xc0000000 (when not using 4G/4G split patch).
So, why can't glibc's malloc use non-continuous areas of address space for one
allocation? And is there a way around the gratuitous placing of shared
libraries at 0x4100000, resulting in a larger free contiguous address space?
(I tried linking statically, eliminating the shared libraries, but strangely
enough, malloc did not want to go above 0x41000000 even though the address
space was marked as free above that. Maybe there is an additional 2 Gig limit
after all due to unrelated signed int issues...).
Anyone still following me? And should I try asking at glibc-devel or at the
Linux kernel mailing list?
Yours
-Malte
BTW, another interesting resource is at the following URL:
http://www.spack.org/wiki/LinuxRamLimits
Reply to: