Allocating more than 890MB in the kernel?
vmalloc() fails after about 890MB because the kernel memory map is only for about 1GB. I know there are some hacks and work-arounds to get more than that, but instead of reinventing the wheel, I was hoping some kind soul would tell me how (a few hints would be nice!)
The reason we use vmalloc() is because we need to apply memory pressure during the allocating: memory should be swapped out to make room for our allocation.
We're trying to allocate up to 3GB on a 4GB machine. Thanks in advance!
Isn't this solved by just recompiling the kernel with HIGHMEM support?
John
Quoted 16 lines · Timur Tabi
vmalloc()fails after about 890MB because the kernel memory map is only for about 1GB. I know there are some hacks and work-arounds to get more than that, but instead of reinventing the wheel, I was hoping some kind soul would tell me how (a few hints would be nice!)The reason we use
vmalloc()is because we need to apply memory pressure during the allocating: memory should be swapped out to make room for our allocation.We're trying to allocate up to 3GB on a 4GB machine. Thanks in advance!
-
Kernelnewbies: Help each other learn about the Linux kernel. Archive: http://mail.nl.linux.org/kernelnewbies/ IRC Channel: irc.openprojects.net / #kernelnewbies Web Page: http://www.kernelnewbies.org/
Signature · 2 lines
John Tyner [email protected]
Quoted 1 line · John Tyner
Isn't this solved by just recompiling the kernel with HIGHMEM support?
I don't think so. The Red Hat 7.1 kernel is compiled with "4GB" support, which apparently is the same as HIGHMEM. We see the 890MB kernel vmalloc limit still.
Followup to: <[email protected]> By author: Timur Tabi <[email protected]> In newsgroup: linux.dev.kernel
Quoted 8 lines
Quoted 2 lines
Isn't this solved by just recompiling the kernel with HIGHMEM support?
I don't think so. The Red Hat 7.1 kernel is compiled with "4GB" support, which apparently is the same as HIGHMEM. We see the 890MB kernel vmalloc limit still.
That's because you're running out of address space, not memory. HIGHMEM doesn't do anything for the latter -- it can't. You start running into a lot of fundamental problems when your memory size gets in the same (or higher) ballpark than your address space.
The best solution is go buy a 64-bit CPU. There isn't much else you can do about it.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
Quoted 7 lines · H. Peter Anvin
That's because you're running out of address space, not memory. HIGHMEM doesn't do anything for the latter -- it can't. You start running into a lot of fundamental problems when your memory size gets in the same (or higher) ballpark than your address space.
The best solution is go buy a 64-bit CPU. There isn't much else you can do about it.
That's completely missing the point of my request (which, I admit, I didn't make clear). I need to allocate about 3/4 of available memory in the kernel.
If I had 2GB of RAM, I'd need to allocate 1.5GB. If I had 8 GB of RAM, I'd
need to allocate 6GB. I just used 3GB/4GB because it's our current test platform.
Quoted 7 lines · Timur Tabi
That's completely missing the point of my request (which, I admit, I didn't make clear). I need to allocate about 3/4 of available memory in the kernel. If I had 2GB of RAM, I'd need to allocate 1.5GB. If I had 8 GB of RAM, I'd need to allocate 6GB. I just used 3GB/4GB because it's our current test platform.
There is no way you can allocate 6 GB of address space when you have 4 GB to dole out -- and that includes to userspace. Linux tends to allocate most of the address space (usually 3 GB) to userspace, because it can be re-used between processes and it matches the needs of more users.
This puts fundamental limits on how much space is addressible in the kernel. What you can do if your application permits is allocate HIGHMEM pages, and use kmap()/kmap_atomic()/kunmap() to selectively bring them into the address space on an as-need basis. THIS IS EXPENSIVE, have no illusions about it, and doesn't give you space that is contiguous in either linear nor physical space.
Obviously, on a 64-bit CPU these limitations utterly vanish, since address space is no longer limited.
-hpa
Quoted 6 lines · Timur Tabi
Quoted 5 lines · H. Peter Anvin
That's because you're running out of address space, not memory. HIGHMEM doesn't do anything for the latter -- it can't. You start running into a lot of fundamental problems when your memory size gets in the same (or higher) ballpark than your address space.
Quoted 8 lines
Quoted 2 lines
The best solution is go buy a 64-bit CPU. There isn't much else you can do about it.
That's completely missing the point of my request (which, I admit, I didn't make clear). I need to allocate about 3/4 of available memory in the kernel. If I had 2GB of RAM, I'd need to allocate 1.5GB. If I had 8 GB of RAM, I'd need to allocate 6GB. I just used 3GB/4GB because it's our current test platform.
Each user process has 32 bit pointers for memory. This means they only have 4 gigabytes of virtual address space, regardless of how many physical pages the machine has. The kernel doesn't use segment:offset addressing. It just uses the offset. Flat memory model.
Now your page tables can shuffle memory around (using physical pages, swap, etc), but that's irrelevant. You've still only got 4 gigs of virtual space per process to work with on a 32 bit system. The register you're ultimately dereferencing to access memory isn't big enough to hold a number larger than that.
To simplify the page tables, the kernel is in shared memory. The first gigabyte of every process's address space is the kernel. The kernel's page table is sort of spliced into everybody's page table. And there was much rejoicing.
The problem is, you're trying to allocate kernel memory, meaning you hit the 1 gigabyte limit. And that includes the kernel itself. You can recompile so the kernel has more than 1 gig of everybody's virtual address range reserved to it, but that reduces the amount of virtual memory user space processes have (4 gigs - 1 gig) because the kernel space (shared memory) is mapped into each and every process so the process can access kernel resources easily. (You do NOT want to change page tables for every system call. We're not going there.)
To allocate more memory than that, you need to allocate pages in user space. To allocate more than 4 gigabytes of memory, you MUST use more than one process (you only have 4 gigs of virtual address space per process, so two processes must use overlapping virtual address ranges to point to different pages. Perhaps they could communicate with each other through a shared mmap.
Congratulations, you've just reinvented expanded memory.)
Or, we could get back to the "buy a real computer" answer everybody's been giving you: buy a 64 bit computer that has more than 4 gigabytes of virtual address space per process. Then you have 64 bit pointers for your virtual addresses, and can directly address virtual petabytes or exabytes or whatever comes next...
Rob
Quoted 3 lines · Rob Landley
Each user process has 32 bit pointers for memory. This means they only have 4 gigabytes of virtual address space, regardless of how many physical pages the machine has. The kernel doesn't use segment:offset addressing. It just uses the offset. Flat memory model.
And even if it did (on i386) it wouldn't help... the segment:offset is folded into a single 32-bit space before paging.
-hpa
Quoted 4 lines
I need to allocate about 3/4 of available memory in the kernel. If I had 2GB of RAM, I'd need to allocate 1.5GB. If I had 8 GB of RAM, I'd need to allocate 6GB. I just used 3GB/4GB because it's our current test platform.
The best you can do, IMHO:
1. reserve a 3 GB chunk of memory at boot
2. create a regular user process
3. have that process make a system call which will never return
4. in that system call, wipe out all memory mappings in the process
5. hand-craft a 3 GB memory mapping (0 GB virt --> 1 GB phys)
6. call your desired code, remembering to schedule by hand
Quoted 6 lines · Albert D. Cahalan
1. reserve a 3 GB chunk of memory at boot
2. create a regular user process
3. have that process make a system call which will never return
4. in that system call, wipe out all memory mappings in the process
5. hand-craft a 3 GB memory mapping (0 GB virt --> 1 GB phys)
6. call your desired code, remembering to schedule by hand
Thanks, that's the most useful idea I've gotten. It's crazy, but it just might work!
Too bad no one else on this list can think outside of the box like you just did.