V2 ia64 SPARSEMEM
cover V2 ia64 SPARSEMEM
Tony,
ChangeLog V2:
Jesse's review input has been applied to patches 3 and 4. These were all cosmetic changes.
This is a series of patches which enable SPARSEMEM for ia64. It's against rc4-mm2. The patches have been tested against memory configurations FLATMEM, DISCONTIG and SPARSEMEM. This includes NUMA simulated hardware, rx2600 and HPSIM. An early version of ia64 with SPARSEMEM was tested by Jesse. It would be optimal to have another test pass on SGI NUMA hardware.
Ultimately I would like to eliminate DISCONTIG and VIRTUAL_MEM_MAP. Before this can be accomplished, more performance comparisons between SPARSEMEM and DISCONTIG+VIRTUAL_MEM_MAP is required. I did some preliminary performance work with 2 CPU rx2600. The results were comparable and no noticeable regression was evidenced. Further work on multi node NUMA machines is required.
thanks,
bob
On Wed, May 25, 2005 at 11:13:47AM -0400, Bob Picco wrote:
...
Quoted 2 lines
and HPSIM. An early version of ia64 with SPARSEMEM was tested by Jesse. It would be optimal to have another test pass on SGI NUMA hardware.
I'm away from the office this week, but next week, I'll give the patch another spin on a big SGI system.
Quoted 10 lines
VIRTUAL_MEM_MAPwas introduced on ia64 because of memory holes on ia64 platforms. Themem_mapof thepglist_datais a pointer to a virtual contiguous array of page structures in memory for a node. To eliminate memory waste contiguous memory holes don't have page structures. The alternative would be for holes in memory to be represented by reserve page structures. TheVIRTUAL_MEM_MAPsolutions requiresia64_pfn_validand a hook in the buddy allocator to check for holes in memory.
Quoted 3 lines
SPARSEMEM has eliminated
mem_map,ia64_pfn_validand the buddy allocator hook. There is a small cost for SPARSEMEM. Any section which has both memory and holes requires reserved pages for the holes. I can see
Our systems are notorious for having large holes in memory on nodes. For example, a node may have memory at node offsets 0, 16GB, 32GB & 48GB. At each offset, you can have a chunk of 1 to 16GB. For example, a typical system might have memory at the following node offsets:
Memory at
0 - 4GB
16 - 20GB
32 - 36GB
48 - 52GBYou are guaranteed to have memory at node offset 0. Memory may or may not exist at the other offsets.
I haven't had a chance to look at the patch, but will page struct entries will be allocated for the pages in the holes (4GB-16GB, etc).
Signature · 4 lines
Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.
Jack Steiner wrote: [Wed May 25 2005, 06:44:06PM EDT]
Quoted 8 lines · Bob Picco
...
Quoted 2 lines
and HPSIM. An early version of ia64 with SPARSEMEM was tested by Jesse. It would be optimal to have another test pass on SGI NUMA hardware.
I'm away from the office this week, but next week, I'll give the patch another spin on a big SGI system.
Great!
Quoted 25 lines
Quoted 11 lines
VIRTUAL_MEM_MAPwas introduced on ia64 because of memory holes on ia64 platforms. Themem_mapof thepglist_datais a pointer to a virtual contiguous array of page structures in memory for a node. To eliminate memory waste contiguous memory holes don't have page structures. The alternative would be for holes in memory to be represented by reserve page structures. TheVIRTUAL_MEM_MAPsolutions requiresia64_pfn_validand a hook in the buddy allocator to check for holes in memory.Quoted 3 lines
SPARSEMEM has eliminated
mem_map,ia64_pfn_validand the buddy allocator hook. There is a small cost for SPARSEMEM. Any section which has both memory and holes requires reserved pages for the holes. I can seeOur systems are notorious for having large holes in memory on nodes. For example, a node may have memory at node offsets 0, 16GB, 32GB & 48GB. At each offset, you can have a chunk of 1 to 16GB. For example, a typical system might have memory at the following node offsets:
Memory at 0 - 4GB 16 - 20GB 32 - 36GB 48 - 52GB
Quoted 5 lines
You are guaranteed to have memory at node offset 0. Memory may or may not exist at the other offsets.
I haven't had a chance to look at the patch, but will page struct entries will be allocated for the pages in the holes (4GB-16GB, etc).
No there won't be page structures allocated for regions which are holes. The default SECTION_BITS value of 28 is for a 256Mb aligned section. So memory not reported (via efi memory descriptor walk) in a section won't have page structs. The function register_sparse_mem in discontig.c calls SPARSEMEM to validate a memory section. Only valid sections are subject to page structure allocation later in initialization.
Quoted 9 lines
Signature · 5 lines
-- Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.
your welcome and thanks,
bob
Quoted 1 line
Quoted 9 lines · Jack Steiner
On Wed, May 25, 2005 at 11:13:47AM -0400, Bob Picco wrote:
...Quoted 2 lines
and HPSIM. An early version of ia64 with SPARSEMEM was tested by Jesse. It would be optimal to have another test pass on SGI NUMA hardware.
I'm away from the office this week, but next week, I'll give the patch another spin on a big SGI system.
Sorry to have taken so long but I had a number of problems getting the latest mm kernel to run reliably on our large systems.
All of the page fault intensive tests that I ran on a 64p showed essentially the same performance with & w/o SPARSEMEM enabled.
I also ran AIM7 but was unsucessful drawing any conclusions from the results. Something in the latest tree is causing large run-to-run variations - almost 2:1 at some load points. I'm trying to determine the cause of these variations.
I have not had a chance to pull down the latest SPARSEMEM patches. I'll do that once I understand the cause of the run-to-run variations.
Signature · 4 lines
Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.
Jack Steiner wrote: [Thu Jun 09 2005, 03:47:13PM EDT]
Quoted 17 lines · Jack Steiner
Quoted 9 lines · Bob Picco
...
Quoted 2 lines
and HPSIM. An early version of ia64 with SPARSEMEM was tested by Jesse. It would be optimal to have another test pass on SGI NUMA hardware.
I'm away from the office this week, but next week, I'll give the patch another spin on a big SGI system.
Sorry to have taken so long but I had a number of problems getting the latest mm kernel to run reliably on our large systems.
All of the page fault intensive tests that I ran on a 64p showed essentially the same performance with & w/o SPARSEMEM enabled.
Quoted 6 lines
I also ran AIM7 but was unsucessful drawing any conclusions from the results. Something in the latest tree is causing large run-to-run variations - almost 2:1 at some load points. I'm trying to determine the cause of these variations.
I have not had a chance to pull down the latest SPARSEMEM patches. I'll do that once I understand the cause of the run-to-run variations.
Jack,
Great. thanks.
Please hold off on fetching latest SPARSEMEM. I'm about to post V3 of ia64 SPARSEMEM. It depends on an Andy -mm patch acked by Andrew and two -mm patches by me not acked by Andrew.
thanks,
bob
Quoted 8 lines
Signature · 5 lines
-- Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.