Repost: could ia32 mmap() allocations grow downward?
Quoted 2 lines · Wayne Whitney
o Pick a maximum stack size S and change the kernel so the "
mmap()withoutMAP_FIXED" region starts at0xC0000000- S and grows downwards.
How you'll pick S? 8MB? 128MB? Now you can have 1GB brk + 2GB (stack+mmap), after change you have 2.9GB (brk+mmap), but only 128MB stack. And if you'll change your malloc implementation, you can have up to 2GB stack now, or up to 3GB of mmap. After your change your stack is limited to 128MB, and you cannot do anything around that except moving stack somewhere else during libc startup - and in this case couple of argv[] assumptions setproctitle and other do are no longer valid.
Another problem is mremap. Due to way how apps works, you'll have to move VMAs around much more because of you cannot grow your last VMA up without move. And if you shrink your last block, you'll get a gap.
Quoted 4 lines
This seems ideal, as it allows the balance between the
mmap()region and thebrk()region to vary for each process, automatically. What changes would be required to the kernel to implement this properly and efficiently? Is there some downside I am missing?
Nobody can call brk() directly from app, as libc may use brk() for implementing malloc(), and libraries can call malloc. So you have to create your own allocator on the top of brk() results, and this allocator must not release memory back to system, as this could release also chunks you do not own. Writting your allocator on the top of malloc()ed areas is much better idea.
Best regards,
Petr Vandrovec [email protected]
P.S.: I do not think that your app calls directly brk(). I think that your app calls malloc with some small number, and libc decides to use brk() instead of mmap(). And in such case it is bug in your libc that it does not use mmap() after brk() fails.
Quoted 6 lines · Petr Vandrovec
Quoted 2 lines · Wayne Whitney
o Pick a maximum stack size S and change the kernel so the "
mmap()withoutMAP_FIXED" region starts at0xC0000000- S and grows downwards.How you'll pick S? 8MB? 128MB?
Well, Mark Hahn suggests using the stack ulimit. On my bog standard RedHat 7.2, ulimit -a tells me the stack size limit is 8MB. Of course, once an mmap() (sans MAP_FIXED) has occurred, you can't increase S, so a program that wants more stack would have to ensure that the ulimit is set before calling mmap().
Quoted 2 lines
Now you can have 1GB brk + 2GB (stack+mmap), after change you have 2.9GB (brk+mmap), but only 128MB stack.
My (very limited) experience suggests that of the stack, mmap and brk regions, stack is likely the smallest. So if one of the three has to have a predetermined maximum size, and the other two are allowed to grow toward each other from opposite ends of address space, it seems the stack should have the fixed size, not brk.
Quoted 3 lines
Another problem is mremap. Due to way how apps works, you'll have to move VMAs around much more because of you cannot grow your last VMA up without move. And if you shrink your last block, you'll get a gap.
Right now, growing any VMA other than the last requires relocating, and shrinking any VMA other than the last will cause gaps. How big a hit would it be to remove the exception for the last VMA, so that any VMA growth requires relocation, and any VMA shrink leaves a gap? Are there applications that rely on cheap growth and shrinkage of the most recently allocated VMA (when there have been deletions and MAP_FIXED mmap()s)?
Quoted 6 lines
Nobody can call
brk()directly from app, as libc may usebrk()for implementingmalloc(), and libraries can call malloc. So you have to create your own allocator on the top ofbrk()results, and this allocator must not release memory back to system, as this could release also chunks you do not own. Writting your allocator on the top of malloc()ed areas is much better idea.
Assuming the overhead of malloc() is low, I agree that in writing a new program one would be better off writing an allocator over malloc() than over brk(). But there are plenty of legacy programs that use brk(), which may be hard to port to a malloc()-based allocator, or available to some users only as binaries.
So there is a tradeoff between changing the programs and changing the kernel. I'm trying to figure out how expensive the requisite kernel changes would be. For example, I don't grok the structure that holds the VMAs, I think it is in some sense sorted by increasing start address. So if one were to change mmap() to allocate VMAs going downward, would it be appropriate to change the VMA containment structure to be sorted by decreasing start address?
BTW, if one were trying to port some code that uses brk() directly and even frees memory that way, then it seems that with glibc's malloc(), one could make it work by instructing malloc() always to use mmap().
Cheers, Wayne
P.S. I am 100% sure that the particular application of mine that started me thinking about this, MAGMA, uses its own allocator built on top of brk() and never calls malloc() itself.