Stuck at 1GB again
Some time ago, the list was very helpful in solving my programs failing at the limit of real memory rather than expanding into swap under linux 2.2.
Now my giant app is failing on a new limit.
I have an Athlon 850 with 2GB of RAM running 2.4.0-test7.
The kernel is compiled with the 4GB option. (which I think is the 2/2GB option from 2.2.x kernels). I believe the option is supposed to assign 2GB of address space to real memory, and 2GB to virtual memory (from a per process point of view).
I've also compiled and am using glibc 2.2, though I believe I don't need to in order to use up to 2GB of real memory.
Without glibc 2.2 I should be able to get to 2GB of memory allocated via the heap. I only need glibc 2.2 to start mmap'ing malloc'able pools from VM. I.E. beyond 2GB of malloc'ed memory.
My app running with 1 GB RAM under linux 2.2, with glibc 2.2 successfully malloc's up to 3GB and the app works fine. (though swapping quite a bit).
My app running with 2 GB RAM under linux 2.4.0-test7, with glibc 2.2 dies at 1 GB of memory used. (it also dies at 1 GB using glibc 2.1.2).
The app is compiled -static to make sure I get the right libraries.
I have an program which logs the memory usage of the application from /proc.
These are logs from two runs that died very near the 1GB limit when malloc returned an error:
967780380 986796 kB 0 kB 985216 kB 985636 kB 628 kB 508 kB 0 kB
967661675 1013092 kB 0 kB 1010700 kB 1011184 kB 540 kB 184 kB 1140 kB
Limits are set to:
cputime unlimited filesize unlimited datasize unlimited stacksize 8MB coredumpsize 488MB memoryuse unlimited maxproc 256 descriptors 1024 memorylocked unlimited addressspace unlimited
Any idea what limit I'm running into?
Signature · 7 lines
Brian Litzinger <[email protected]> Copyright (c) 2000 By Brian Litzinger, All Rights Reserved -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux.eu.org/Linux-MM/
Hi,
Quoted 4 lines · On Sat, Sep 02, 2000 at 11:50:32AM -0700, [email protected] wrote:
The kernel is compiled with the 4GB option. (which I think is the 2/2GB option from 2.2.x kernels). I believe the option is supposed to assign 2GB of address space to real memory, and 2GB to virtual memory (from a per process point of view).
No, it's much better than that in 2.4! The 2.4 4GB option still has the 3+1 split, and only maps the first just-under-1GB of physical memory into the kernel's permanent VA space. The rest of the memory is mapped on demand.
Quoted 4 lines
Without glibc 2.2 I should be able to get to 2GB of memory allocated via the heap. I only need glibc 2.2 to start mmap'ing malloc'able pools from VM. I.E. beyond 2GB of malloc'ed memory.
Right.
Quoted 6 lines
My app running with 1 GB RAM under linux 2.2, with glibc 2.2 successfully malloc's up to 3GB and the app works fine. (though swapping quite a bit).
My app running with 2 GB RAM under linux 2.4.0-test7, with glibc 2.2 dies at 1 GB of memory used. (it also dies at 1 GB using glibc 2.1.2).
What happens at this point? If you strace the binary, what fails?
--Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Quoted 5 lines
Some time ago, the list was very helpful in solving my programs failing at the limit of real memory rather than expanding into swap under linux 2.2.
I can;t say what your actual problem is, but in previous experiments, I have seen these as the main cause:
1. shortage of real memory (ram + swap). I don't think this is your problem.
2. resource limit problems: some resource limits were defined as "int/long" instead of "unsigned int/long", but these should have been fixed.
3. inability of malloc to find a contiguous range of virtual space in
userland: this depends on libraries used etc, that eat up chunks of the user space. This might be your problem. (Hint: code a while(1) loop before any malloc happens in your program, then use "cat /proc/pid/maps", where pid is the pid of your running program, to see the user space virtual address allocation; you might not see a contiguous 3Gb chunk for malloc).
Kanoj
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/