Skip to contentSkip to conversation

V2 ia64 SPARSEMEM

PATCH In review Bob Picco 2 participants 5 messages May 25, 2005 – Jun 9, 2005

lore
Share & tools
lore

Bob Picco

author
#0 lore raw

cover V2 ia64 SPARSEMEM

Tony,

ChangeLog V2:
Jesse's review input has been applied to patches 3 and 4. These were all cosmetic changes.

This is a series of patches which enable SPARSEMEM for ia64. It's against rc4-mm2. The patches have been tested against memory configurations FLATMEM, DISCONTIG and SPARSEMEM. This includes NUMA simulated hardware, rx2600 and HPSIM. An early version of ia64 with SPARSEMEM was tested by Jesse. It would be optimal to have another test pass on SGI NUMA hardware.

Ultimately I would like to eliminate DISCONTIG and VIRTUAL_MEM_MAP. Before this can be accomplished, more performance comparisons between SPARSEMEM and DISCONTIG+VIRTUAL_MEM_MAP is required. I did some preliminary performance work with 2 CPU rx2600. The results were comparable and no noticeable regression was evidenced. Further work on multi node NUMA machines is required.

thanks,

bob

#1 lore raw

On Wed, May 25, 2005 at 11:13:47AM -0400, Bob Picco wrote:
...

Quoted 2 lines

and HPSIM. An early version of ia64 with SPARSEMEM was tested by Jesse. It would be optimal to have another test pass on SGI NUMA hardware.

I'm away from the office this week, but next week, I'll give the patch another spin on a big SGI system.

Quoted 10 lines

VIRTUAL_MEM_MAP was introduced on ia64 because of memory holes on ia64 platforms. The mem_map of the pglist_data is a pointer to a virtual contiguous array of page structures in memory for a node. To eliminate memory waste contiguous memory holes don't have page structures. The alternative would be for holes in memory to be represented by reserve page structures. The VIRTUAL_MEM_MAP solutions requires ia64_pfn_valid and a hook in the buddy allocator to check for holes in memory.

Quoted 3 lines

SPARSEMEM has eliminated mem_map, ia64_pfn_valid and the buddy allocator hook. There is a small cost for SPARSEMEM. Any section which has both memory and holes requires reserved pages for the holes. I can see

Our systems are notorious for having large holes in memory on nodes. For example, a node may have memory at node offsets 0, 16GB, 32GB & 48GB. At each offset, you can have a chunk of 1 to 16GB. For example, a typical system might have memory at the following node offsets:

    Memory at
         0 -  4GB
	16 - 20GB
	32 - 36GB
	48 - 52GB

You are guaranteed to have memory at node offset 0. Memory may or may not exist at the other offsets.

I haven't had a chance to look at the patch, but will page struct entries will be allocated for the pages in the holes (4GB-16GB, etc).

Signature · 4 lines
Thanks

Jack Steiner ([email protected])          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.
author
#2 lore raw
In reply to Bob Picco

Jack Steiner wrote: [Wed May 25 2005, 06:44:06PM EDT]

Quoted 8 lines · Bob Picco

...

Quoted 2 lines

and HPSIM. An early version of ia64 with SPARSEMEM was tested by Jesse. It would be optimal to have another test pass on SGI NUMA hardware.

I'm away from the office this week, but next week, I'll give the patch another spin on a big SGI system.

Great!

Quoted 25 lines
Quoted 11 lines

VIRTUAL_MEM_MAP was introduced on ia64 because of memory holes on ia64 platforms. The mem_map of the pglist_data is a pointer to a virtual contiguous array of page structures in memory for a node. To eliminate memory waste contiguous memory holes don't have page structures. The alternative would be for holes in memory to be represented by reserve page structures. The VIRTUAL_MEM_MAP solutions requires ia64_pfn_valid and a hook in the buddy allocator to check for holes in memory.

Quoted 3 lines

SPARSEMEM has eliminated mem_map, ia64_pfn_valid and the buddy allocator hook. There is a small cost for SPARSEMEM. Any section which has both memory and holes requires reserved pages for the holes. I can see

Our systems are notorious for having large holes in memory on nodes. For example, a node may have memory at node offsets 0, 16GB, 32GB & 48GB. At each offset, you can have a chunk of 1 to 16GB. For example, a typical system might have memory at the following node offsets:

    Memory at
         0 -  4GB
	16 - 20GB
	32 - 36GB
	48 - 52GB
Quoted 5 lines

You are guaranteed to have memory at node offset 0. Memory may or may not exist at the other offsets.

I haven't had a chance to look at the patch, but will page struct entries will be allocated for the pages in the holes (4GB-16GB, etc).

No there won't be page structures allocated for regions which are holes. The default SECTION_BITS value of 28 is for a 256Mb aligned section. So memory not reported (via efi memory descriptor walk) in a section won't have page structs. The function register_sparse_mem in discontig.c calls SPARSEMEM to validate a memory section. Only valid sections are subject to page structure allocation later in initialization.

Quoted 9 lines
Signature · 5 lines
-- 
Thanks

Jack Steiner ([email protected])          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.

your welcome and thanks,

bob

Quoted 1 line
#3 lore raw
In reply to Bob Picco
Quoted 9 lines · Jack Steiner

On Wed, May 25, 2005 at 11:13:47AM -0400, Bob Picco wrote:
...

Quoted 2 lines

and HPSIM. An early version of ia64 with SPARSEMEM was tested by Jesse. It would be optimal to have another test pass on SGI NUMA hardware.

I'm away from the office this week, but next week, I'll give the patch another spin on a big SGI system.

Sorry to have taken so long but I had a number of problems getting the latest mm kernel to run reliably on our large systems.

All of the page fault intensive tests that I ran on a 64p showed essentially the same performance with & w/o SPARSEMEM enabled.

I also ran AIM7 but was unsucessful drawing any conclusions from the results. Something in the latest tree is causing large run-to-run variations - almost 2:1 at some load points. I'm trying to determine the cause of these variations.

I have not had a chance to pull down the latest SPARSEMEM patches. I'll do that once I understand the cause of the run-to-run variations.

Signature · 4 lines
Thanks

Jack Steiner ([email protected])          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.
author
#4 lore raw
In reply to Bob Picco

Jack Steiner wrote: [Thu Jun 09 2005, 03:47:13PM EDT]

Quoted 17 lines · Jack Steiner
Quoted 9 lines · Bob Picco

...

Quoted 2 lines

and HPSIM. An early version of ia64 with SPARSEMEM was tested by Jesse. It would be optimal to have another test pass on SGI NUMA hardware.

I'm away from the office this week, but next week, I'll give the patch another spin on a big SGI system.

Sorry to have taken so long but I had a number of problems getting the latest mm kernel to run reliably on our large systems.

All of the page fault intensive tests that I ran on a 64p showed essentially the same performance with & w/o SPARSEMEM enabled.

Quoted 6 lines

I also ran AIM7 but was unsucessful drawing any conclusions from the results. Something in the latest tree is causing large run-to-run variations - almost 2:1 at some load points. I'm trying to determine the cause of these variations.

I have not had a chance to pull down the latest SPARSEMEM patches. I'll do that once I understand the cause of the run-to-run variations.

Jack,

Great. thanks.

Please hold off on fetching latest SPARSEMEM. I'm about to post V3 of ia64 SPARSEMEM. It depends on an Andy -mm patch acked by Andrew and two -mm patches by me not acked by Andrew.

thanks,

bob

Quoted 8 lines
Signature · 5 lines
-- 
Thanks

Jack Steiner ([email protected])          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.