maximum memory limit

The kernel supports up to 3GB of address space per process. The first 900MB can be allocated by brk() and the rest can be allocated by mmap().

Problem is that libc malloc() appears to use brk() only, so it is limited to 900MB. You can fix that by doing the brk() and malloc() yourself, but I think that in the long run the glibc people may want to change their malloc implementation so that it automatically supports the full 3GB...

regards,

Rik
--
The Internet is not a network of computers. It is a network of people. That is its real strength.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

The glibc manual says that for allocations much greater than the page size (no, it doesn't quantify "much") it will use anonymous mmap of /dev/zero.

It's probably a bad idea to allocate over a gigabyte in 1K chunks anyway...

Matthew.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

modern libc's certainly use mmap for large mallocs. but this can be a serious problem: I corresponded with someone who had a binary app that did many small mallocs, and he was pissed that his 4G box could only malloc 900M or so. this happened because __PAGE_OFFSET and TASK_SIZE were 3G, but TASK_UNMAPPED_BASE, where mmap's start, is TASK_SIZE/3.

a hackish solution that worked was TASK_UNMAPPED_BASE=TASK_SIZE-0x20000000, which just assumes that you won't need >512M of mmaped areas.

since the heap grows up and the stack is generally small and limited, it would be nice to arrange for mmaped areas to grow down. as far as I can tell, we could just sort vmlist in descending order.

would there be some problem with doing this?

regards, mark hahn.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

glibc malloc is able to use mmap, plus have a lot of tunable things. See mallopt(3), particularly M_MMAP_THRESHOLD and M_MMAP_MAX parameters. The default mmap threshold (above which malloc uses mmap) is I think 128K, but you can decrease it. But it does not make much sense to decrease this below PAGE_SIZE, because you then waste a lot of memory.

Cheers,
Jakub
___________________________________________________________________
Jakub Jelinek | [email protected] | http://sunsite.mff.cuni.cz/~jj
Linux version 2.3.42 on a sparc64 machine (1343.49 BogoMips)
___________________________________________________________________
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Another thing to keep in mind is that to allow efficient allocation/deallocation, there may be some rounding going on. The 2.0 kmalloc would round 1024 to 2048 and therefore waste almost half the RAM.

glibc will use mmap to implement "malloc". libc5 probably uses brk.

Roger.

** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2137555 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
 "I didn't say it was your fault. I said I was going to blame it on you."
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Clarification: The problem is the brk interface, which ignores fragmentation. The brk interface assumes all memory is continuous. When brk runs into any mapping it fails. And since ld.so is mapped at 1GB the brk cannot allocate any more memory. This is agravated by the fact that ELF programs appear to be intially mapped at 128M+288K. 0x08048000. (Someone allocated 900MB??? wow!)

It would certainly be a good option if libc could allocate new chunks of memory with mmap, or a combination of mmap and mremap. mremap is functionally a good as brk but will let you work with arbitrary areas of memory.

A good option is to compile programs that need huge amounts of memory through brk statically. If they do not use mmap, or shmat they should be fine until they hit the stack, which is growing in the other direction from 3GB. Because the program is static it's code size is reduced the linker will only pull in needed objects, and performance is also enhanced as you don't need to deal with PIC, and register starvation. So it looks good for compute intensive code.

Eric
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

I imagine we agree that the problem is not due to brk(). It's due to the fragmented memory map in Linux.

We use linux systems in compute farms running programs that often need well in excess of 2GB of memory. Partly allocated in few large blocks, partly as many small blocks.

(So the fact that on Linux the kernel wastes the upper 1GB of the address space does make Solaris/x86 look very attractive, also because it can address all physical 4GB; but that's a different issue)

The current implementation of malloc already does that to a limited extent. The reason why it doesn't go any further than that is because overreliance on mmap() puts all the burden on the kernel, and you're likely to severely fragment your memory map. Remember, mmap has a pagesize resolution.

I have written an extended version that alleviates that problem by jumping across the shared libs when brk runs into them, but there are some fundamental problems with this approach that make it work only in well controlled environments.

So how about getting rid of this memory map dichotomy?

The shared libs could be mapped from 3GB-max_stacksize downwards (rather than from 1GB upwards).

Is there any reason why this cannot be done?

This would allow brk to grow beyond the 1GB stonewall.

True. Unfortunately some libs only come as shared libs. If your program needs them, no sigar.

Raymond Nijssen
[email protected]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Hi,

libc can easily mmap /dev/zero in chunks of multiple MB at a time if it wants to, and can then dole out that memory as if it was a huge piece of the heap without kernel involvement.

You then break all programs which allocate large arrays on the stack.

You cannot have an arbitrarily growable heap, AND an arbitrarily growable stack, AND have the kernel correctly guess where to place mmaps.

One answer may be to start mmaps down near the heap boundary, and to teach glibc to be more willing to use mmap() for even small mallocs. That may break custom malloc libraries but should give the best results for code which uses the standard glibc malloc: it doesn't artificially restrain the stack or the mmap area. Anything relying directly on [s]brk will be affected, of course.

--Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Several allocators already do that (including GCC's), but Glibc doesn't AFAIK.

There are lots of custom malloc libraries. If you're going to teach Glibc something anyway, why not add a new mmap flag?

One flag I think would be quite useful is MAP_NOCLOBBER|MAP_FIXED: there are times when I'd like to be able to _try_ mapping something at a fixed address, but fail if there is something mapped there already. Currently it's necessary to parse /proc/self/maps, and that's far from thread safe.

The obvious use is for pre-relocated shared libraries, which can run at any address but will load faster and share more pages if loaded at a specific address. (Especially if they're non-PIC).

As a natural extension, a map flag which says "try to map at the supplied address, but if there is an object there search for a big enough hole above the supplied address" would simultaneously be useful for malloc optimisations like you're suggesting, and pre-relocated shared libraries. That would be MAP_NOCLOBBER without MAP_FIXED.

-- Jamie
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Hi,

Because by the time glibc has been loaded, it's too late! We need the crt0.o stub to load libdl.so lower in memory or we have already clobbered the address space irretrievably. That's why Ben's proposal for a per-process prctl() for the mmap base would seem to work well --- you can just use floating mmaps and the kernel will automatically place them somewhere that won't interfere with really large mallocs.

Good idea, but it doesn't help this problem.

That's what non-MAP_FIXED mmaps already do. From mmap.c:

	/* Obtain the address to map to. we verify (or select) it and
ensure
	 * that it represents a valid section of the address space.
	 */
	if (flags & MAP_FIXED) {
		if (addr & ~PAGE_MASK)
			return -EINVAL;
	} else {
		addr = get_unmapped_area(addr, len);
		if (!addr)
			return -ENOMEM;
	}

get_unmapped_area looks for the next unmapped area above addr, with the start address defaulting to TASK_UNMAPPED_BASE only if addr==0 going in.

Cheers,
Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

?? /lib/ld-linux.so.2 is loaded by the kernel so you can easily changed the way it is mapped. Everything from there on is part of Glibc.

The only exception is statically linked binaries, and they're not going to switch to a new malloc anyway.

-- Jamie
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Sure enough, mmapping anonymous chunks is easy. It is not easy to efficiently manage those chunks.

This fragmentation (these chunks are oftentimes not abutting -- you have limited control over that) is hard to manage. Make them big and you'll waste tons of resources. Make them small and you'll be unable to consolidate free ranges to satisfy large requests.

Writing an efficient allocator is actually a pretty tough problem. It has to be fast, low-fragmenting, low administration overhead, low waste, etc. Add to that the ability to co-exist with other sbrk()ing allocators, mmap()ing routines and other complicating factors.
An erratic context makes those objectives so much harder to achieve.

Yes some workarounds can may be made to work. They make nices excercises, but it's still putting the horse behind the carriage.
There's no substitute for a large unadorned memory space.

Most programmers are wise enough not to assume a large stack.
For one because the stack has a fixed size on all major unices, typically 8MB (Solaris) or 80MB (HP/UX).

Does anybody know about programs requiring more than that?

Absolutely. It's a matter of choosing which one to confine.

Practically it is better to confine the stack.

The rare exception for which the default max stacksize is not enough can be resolved easily by a wrapper that sets a different resource limit.

I don't understand this fear for limiting the stack: in practice that just isn't a problem. If anything one should fear fragmenting the heap: I have seen many cases in which that is a severe problem.

YMMV of course, and if it does, please share your experiences.

-Raymond
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/