Any limitations on bigmem usage?
I am about to embark on a data processing software project that will require a LOT of memory (about, ohhh, 6GB or so), and I was wondering if there are any limitations to how one can use very large chunks of memory under Linux. Specifically, is there anything to prevent me from malloc()ing 6GB of memory, then accessing that memory as I would any other buffer? FYI, the application will be buffering a stream of data, then performing a variety of calculations on large blocks of data at a time, before writing it out to a socket.
I've been eyeing an 8-way Intel box with gobs of memory, but if there are subtle issues with using that much memory, I need to know now.
I haven't seen this specifcally addressed, so I figured I should ask you folk. Any insights/comments/reccomendations would be greatly appreciated.
Thanks,
Matt
Signature · 4 lines
Matthew Nelson Dynamics Technology, Inc. 21311 Hawthorne Blvd., Suite 300, Torrance, CA 90503-5610 Voice: (310) 543-5433 FAX: (310) 543-2117 Email: [email protected]
Quoted 9 lines · Matt Nelson
I am about to embark on a data processing software project that will require a LOT of memory (about, ohhh, 6GB or so), and I was wondering if there are any limitations to how one can use very large chunks of memory under Linux. Specifically, is there anything to prevent me from malloc()ing 6GB of memory, then accessing that memory as I would any other buffer? FYI, the application
will be buffering a stream of data, then performing a variety of calculations on large blocks of data at a time, before writing it out to a socket.
Pointers on IA32 are still 32 bits, so (as I understand it) each process can address a maximum of 4GB. You would have to allocate multiple chunks (in shared memory or mmap()ed files, so they're persistent) and map them in and out as needed (which could get hairy).
Quoted 2 lines
I've been eyeing an 8-way Intel box with gobs of memory, but if there are subtle issues with using that much memory, I need to know now.
Best bet is probably to buy an Alpha machine, if you don't want to spend time working around the 4GB limitation.
-Doug
Signature · 4 lines
The rain man gave me two cures; he said jump right in, The first was Texas medicine--the second was just railroad gin, And like a fool I mixed them, and it strangled up my mind, Now people just get uglier, and I got no sense of time... --Dylan
Quoted 12 lines · Doug McNaught
Quoted 9 lines · Matt Nelson
I am about to embark on a data processing software project that will require a LOT of memory (about, ohhh, 6GB or so), and I was wondering if there are any limitations to how one can use very large chunks of memory under Linux. Specifically, is there anything to prevent me from malloc()ing 6GB of memory, then accessing that memory as I would any other buffer? FYI, the application will be buffering a stream of data, then performing a variety of calculations on large blocks of data at a time, before writing it out to a socket.
Quoted 4 lines
Pointers on IA32 are still 32 bits, so (as I understand it) each process can address a maximum of 4GB. You would have to allocate multiple chunks (in shared memory or mmap()ed files, so they're persistent) and map them in and out as needed (which could get hairy).
Sorry to followup to myself, but I just had another thought--if you can split your task up into multiple processe such that no process needs to address more than 4GB, an IA32 machine will work fine. It will probably still take more programming work than the naive approach (malloc() huge chink, use it) that you could use on a 64-bit machine...
-Doug
Signature · 4 lines
The rain man gave me two cures; he said jump right in, The first was Texas medicine--the second was just railroad gin, And like a fool I mixed them, and it strangled up my mind, Now people just get uglier, and I got no sense of time... --Dylan
Thanks to all that replied. I've never needed to use so much memory before, and was ignorant to how much magic was implemented in the 64G support on IA32. Unfortunately, there's not quite enough magic in there for my needs... now to find an affordable SMP Alpha system....
Thanks again,
-Matt
Quoted 11 lines · Matt Nelson
I am about to embark on a data processing software project that will require a LOT of memory (about, ohhh, 6GB or so), and I was wondering if there are any limitations to how one can use very large chunks of memory under Linux. Specifically, is there anything to prevent me from malloc()ing 6GB of memory, then accessing that memory as I would any other buffer? FYI, the application will be buffering a stream of data, then performing a variety of calculations on large blocks of data at a time, before writing it out to a socket.
I've been eyeing an 8-way Intel box with gobs of memory, but if there are subtle issues with using that much memory, I need to know now.
I haven't seen this specifcally addressed, so I figured I should ask you folk. Any insights/comments/reccomendations would be greatly appreciated.
Thanks,
Matt
Signature · 4 lines
Matthew Nelson Dynamics Technology, Inc. 21311 Hawthorne Blvd., Suite 300, Torrance, CA 90503-5610 Voice: (310) 543-5433 FAX: (310) 543-2117 Email: [email protected]
Quoted 2 lines
Specifically, is there anything to prevent me from malloc()ing 6GB of memory, then accessing that memory as I would any other buffer? FYI, the application
X86 has no nice way to address over 4Gb of RAM. You can do paging games with multiple banks and shmat (ie like using DOS expanded ram but with bigger pieces).
Quoted 2 lines
I've been eyeing an 8-way Intel box with gobs of memory, but if there are subtle issues with using that much memory, I need to know now.
If your algorithm can work well with say 2Gb windows on the data and only change window evey so often (in computing terms) then it should be ok, if its access is random you need to look at a 64bit box like an Alpha, Sparc64 or eventually IA64
Quoted 4 lines · Alan Cox
If your algorithm can work well with say 2Gb windows on the data and only change window evey so often (in computing terms) then it should be ok, if its access is random you need to look at a 64bit box like an Alpha, Sparc64 or eventually IA64
No, eventually Sledgehammer.
You know that IA64 will never ship, because the performance will always suck.
The design is fundamentally flawed. The real bottleneck is the memory bus.
These days they're clock multiplying the inside of the processor by a factor of what, twelve? Fun as long as you're entirely in L1 cache and aren't task switching. But that's basicaly an embedded system.
CISC instructions are effectively compressed. When you exhaust your cache your next 64 bit gulp can get more than 2 instructions, you can get more like 5 (which still means you're getting about 1/4 the performance of in-cache execution, although L2 and L3 caches help here, of course..)
But optimizing for something that isn't actually your bottleneck is just dumb, and that's exactly what Intel did with the move to VLIW/EPIC/IA64. 3 times 64 bit instructions is 192, whereas Pentium can fit more than that in a single 64 bit chunk. Brilliant. You need what, a 6x larger cache just to break even with the amount of time you're running in-cache? And of course the compiler is GOING to put NOPs in that because it won't always be able to figure out something for the second and third cores to do this clock, regardless of how good a compiler it is. That's just beautiful.
This is why they were so desperate for RAMBUS. They KNOW the memory bus is killing them, they were grasping at straws to fix it. (Currently they're saying that a future vaporware version of iTanium will fix everything, but it's a fundamental design flaw: the new instruction set they've chosen is WAY less efficient going across the memory bus, and that's the real limiting factor in performance.)
Transmeta at least sucks CISC in from main memory to keep the northbridge bottleneck optimized. And they have a big cache, and they're still using 32 bit instructions so they only need 96 bytes per chunk (or atom or whatever they're calling it these days).
Sledgehammer is the interesting x86 64 bit instruction set. You still have the cisc/risc hybrid that made pentium work. (And, of course, backwards compatability that's inherent in the design rather than bolted onto the side with duct tape and crazy glue.) Yeah the circuitry to make it work is probably fairly insane, but at least the Athlon design team made all the racy logic go away so they can migrate the design to new manufacturing processes without four months of redesign work. (And no, making insanely long pipelines in Pentium 4 is not a major improvement there either. Cache exhaustion still kills you, so does branch misprediction. Stalling a longer pipeline is more expensive.)
I saw an 800 mhz iTanium which benchmarked about the same (running 64 bit code) as a Penium III 300 mhz running 32 bit code. That's just sad. Go ahead and blame the compiler. That's still just sad. And a design flaw in the new instruction set is not something that can be easily optimized away.
Rob
Quoted 2 lines · Rob Landley
Brilliant. You need what, a 6x larger cache just to break even with the amount of time you're running in-cache?
That's going to be hard, since the cache will also need to be faster in order to feed the CPU core. Making a cache both larger AND faster at the same time will need some smart people.
Quoted 3 lines
And of course the compiler is GOING to put NOPs in that because it won't always be able to figure out something for the second and third cores to do this clock, regardless of how good a compiler it is.
Compilers are also notoriously bad at runtime optimisations.
Quoted 1 line
That's just beautiful.
I also never expected Intel to dispose of themselves in such a cute way.
cheers,
Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
http://www.surriel.com/ http://distro.conectiva.com/
Send all your spam to [email protected] (spam digging piggy)