: fake numa for x86_64 patch
This patch provides a IO hole size in a given address range.
include/asm-x86_64/e820.h +1 -0
@@ -46,6 +46,7 @@ extern void e820_mark_nosave_regions(voiextern void e820_print_map (char *who);extern int e820_any_mapped (unsigned long start,unsigned long end,unsigned type);extern int e820_all_mapped (unsigned long start,unsigned long end,unsigned type); +extern unsigned long e820_hole_size (unsigned long start,unsigned long end);extern void e820_setup_gap (void ); extern void e820_register_active_regions(int nid,
arch/x86_64/kernel/e820.c +32 -0
@@ -184,6 +184,38 @@ unsigned long __init e820_end_of_ram(voi }/* +* Find the hole size in the range. +*/ +unsigned long __init e820_hole_size (unsigned long start,unsigned long end) +{ +unsigned long ram =0 ; +int i; + +for (i =0 ; i < e820.nr_map; i++) { +struct e820entry *ei = &e820.map[i]; +unsigned long last, addr; + +if (ei->type !=E820_RAM || + ei->addr+ei->size <= start || + ei->addr >= end) +continue ; + + addr =round_up (ei->addr,PAGE_SIZE ); +if (addr < start) + addr = start; + + last =round_down (ei->addr + ei->size,PAGE_SIZE ); +if (last >= end) + last = end; + +if (last > addr) + ram += last - addr; + } +return ((end - start) - ram); +} + + +/* * Mark e820 reserved areas as busy for the resource manager. */ void __init e820_reserve_resources (void )
Quoted 2 lines · Rohit Seth
This patch provides a IO hole size in a given address range.
Hi,
This patch reintroduces a function that doubles up what absent_pages_in_range(start_pfn, end_pfn). I recognise you do this because you are interested in hole sizes before add_active_range() is called. However, what is not clear is why these patches are so specific to x86_64.
It looks possible to do the work of functions like split_nodes_equal() in an architecture-independent manner using early_node_map rather than dealing with the arch-specific nodes array. That would open the possibility of providing fake nodes on more than one architecture in the future.
What I think can be done is that you register memory as normal and then split up the nodes into fake nodes. This would remove the need for having e820_hole_size() reintroduced.
Quoted 57 lines
Signed-off-by David Rientjes <[email protected]>Signed-off-by Paul Menage <[email protected]>Signed-off-by Rohit Seth <[email protected]>include/asm-x86_64/e820.h +1
-0@@ -46,6 +46,7 @@ extern void e820_mark_nosave_regions(voiextern void e820_print_map (char *who);extern int e820_any_mapped (unsigned long start,unsigned long end,unsigned type);extern int e820_all_mapped (unsigned long start,unsigned long end,unsigned type); +extern unsigned long e820_hole_size (unsigned long start,unsigned long end);extern void e820_setup_gap (void ); extern void e820_register_active_regions(int nid,arch/x86_64/kernel/e820.c +32
-0@@ -184,6 +184,38 @@ unsigned long __init e820_end_of_ram(voi }/* +* Find the hole size in the range. +*/ +unsigned long __init e820_hole_size (unsigned long start,unsigned long end) +{ +unsigned long ram =0 ; +int i; + +for (i =0 ; i < e820.nr_map; i++) { +struct e820entry *ei = &e820.map[i]; +unsigned long last, addr; + +if (ei->type !=E820_RAM || + ei->addr+ei->size <= start || + ei->addr >= end) +continue ; + + addr =round_up (ei->addr,PAGE_SIZE ); +if (addr < start) + addr = start; + + last =round_down (ei->addr + ei->size,PAGE_SIZE ); +if (last >= end) + last = end; + +if (last > addr) + ram += last - addr; + } +return ((end - start) - ram); +} + + +/* * Mark e820 reserved areas as busy for the resource manager. */ void __init e820_reserve_resources (void )
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Hi Mel,
Quoted 10 lines · Mel Gorman
Quoted 2 lines · Rohit Seth
This patch provides a IO hole size in a given address range.
Hi,
This patch reintroduces a function that doubles up what
absent_pages_in_range(start_pfn, end_pfn). I recognise you do this because you are interested in hole sizes beforeadd_active_range()is called.
Right.
Quoted 3 lines
However, what is not clear is why these patches are so specific to
x86_64.
Specifically in the fake numa case, we want to make sure that we don't carve fake nodes that only have IO holes in it. Unlike the real NUMA case, here we don't have SRAT etc. to know the memory layout beforehand.
Quoted 5 lines
It looks possible to do the work of functions like
split_nodes_equal()in an architecture-independent manner usingearly_node_maprather than dealing with the arch-specific nodes array. That would open the possibility of providing fake nodes on more than one architecture in the future.
The functions like splti_nodes_equal etc. can be abstracted out to arch independent part. I think the only API it needs from arch dependent part is to find out how much real RAM is present in range without have to first do add_active_range.
Though as a first step, let us fix the x86_64 (as it doesn't boot when you have sizeable chunk of IO hole and nodes > 4).
I'm also not sure if other archs actually want to have this functionality.
Quoted 3 lines
What I think can be done is that you register memory as normal and then split up the nodes into fake nodes. This would remove the need for having
e820_hole_size()reintroduced.
Are you saying first let the system find out real numa topology and then build fake numa on top of it?
-rohit
Quoted 31 lines · Rohit Seth
Hi Mel,
Quoted 10 lines · Mel Gorman
Quoted 2 lines · Rohit Seth
This patch provides a IO hole size in a given address range.
Hi,
This patch reintroduces a function that doubles up what
absent_pages_in_range(start_pfn, end_pfn). I recognise you do this because you are interested in hole sizes beforeadd_active_range()is called.Right.
Quoted 3 lines
However, what is not clear is why these patches are so specific to
x86_64.Specifically in the fake numa case, we want to make sure that we don't carve fake nodes that only have IO holes in it. Unlike the real NUMA case, here we don't have SRAT etc. to know the memory layout beforehand.
Quoted 5 lines
It looks possible to do the work of functions like
split_nodes_equal()in an architecture-independent manner usingearly_node_maprather than dealing with the arch-specific nodes array. That would open the possibility of providing fake nodes on more than one architecture in the future.
Quoted 5 lines
The functions like
splti_nodes_equaletc. can be abstracted out to arch independent part. I think the only API it needs from arch dependent part is to find out how much real RAM is present in range without have to first doadd_active_range.
That is a problem because the ranges must be registered with add_active_range() to work out how much real RAM is present.
Quoted 3 lines
Though as a first step, let us fix the
x86_64(as it doesn't boot when you have sizeable chunk of IO hole and nodes > 4).
Ok.
Quoted 3 lines
I'm also not sure if other archs actually want to have this functionality.
It's possible that the containers people are interested in the possibility of setting up fake nodes as part of a memory controller.
Quoted 7 lines
Quoted 3 lines
What I think can be done is that you register memory as normal and then split up the nodes into fake nodes. This would remove the need for having
e820_hole_size()reintroduced.Are you saying first let the system find out real numa topology and then build fake numa on top of it?
Yes, there is nothing stopping you altering the early_node_map[] before free_area_init_node() initialises the node_mem_map. If you do hit a problem, it'll be because x86_64 allocates it's own node_mem_map with CONFIG_FLAT_NODE_MEM_MAP is set. Is that set when setting up fake nodes?
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Quoted 33 lines · Mel Gorman
Quoted 31 lines · Rohit Seth
Hi Mel,
Quoted 10 lines · Mel Gorman
Quoted 2 lines · Rohit Seth
This patch provides a IO hole size in a given address range.
Hi,
This patch reintroduces a function that doubles up what
absent_pages_in_range(start_pfn, end_pfn). I recognise you do this because you are interested in hole sizes beforeadd_active_range()is called.Right.
Quoted 3 lines
However, what is not clear is why these patches are so specific to
x86_64.Specifically in the fake numa case, we want to make sure that we don't carve fake nodes that only have IO holes in it. Unlike the real NUMA case, here we don't have SRAT etc. to know the memory layout beforehand.
Quoted 5 lines
It looks possible to do the work of functions like
split_nodes_equal()in an architecture-independent manner usingearly_node_maprather than dealing with the arch-specific nodes array. That would open the possibility of providing fake nodes on more than one architecture in the future.
Quoted 9 lines
Quoted 5 lines
The functions like
splti_nodes_equaletc. can be abstracted out to arch independent part. I think the only API it needs from arch dependent part is to find out how much real RAM is present in range without have to first doadd_active_range.That is a problem because the ranges must be registered with
add_active_range()to work out how much real RAM is present.
Right. And that is why I need e820_hole_size functionality. BTW, what is the concern in having that function?
Quoted 4 lines
Quoted 3 lines
Though as a first step, let us fix the
x86_64(as it doesn't boot when you have sizeable chunk of IO hole and nodes > 4).
Quoted 9 lines
Ok.
Quoted 3 lines
I'm also not sure if other archs actually want to have this functionality.
It's possible that the containers people are interested in the possibility of setting up fake nodes as part of a memory controller.
That is precisely why I'm doing it :-)
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
What I think can be done is that you register memory as normal and then split up the nodes into fake nodes. This would remove the need for having
e820_hole_size()reintroduced.
Quoted 9 lines
Quoted 3 lines
Are you saying first let the system find out real numa topology and then build fake numa on top of it?
Yes, there is nothing stopping you altering the early_node_map[] before
free_area_init_node()initialises thenode_mem_map. If you do hit a problem, it'll be becausex86_64allocates it's ownnode_mem_mapwithCONFIG_FLAT_NODE_MEM_MAPis set. Is that set when setting up fake nodes?
I thought they both (real numa + fake numa) operate on same data structures. I'll have to double check.
-rohit
Quoted 40 lines · Rohit Seth
Quoted 39 lines · Mel Gorman
Quoted 36 lines · Rohit Seth
Hi Mel,
Quoted 10 lines · Mel Gorman
Quoted 2 lines · Rohit Seth
This patch provides a IO hole size in a given address range.
Hi,
This patch reintroduces a function that doubles up what
absent_pages_in_range(start_pfn, end_pfn). I recognise you do this because you are interested in hole sizes beforeadd_active_range()is called.Right.
Quoted 3 lines
However, what is not clear is why these patches are so specific to
x86_64.Specifically in the fake numa case, we want to make sure that we don't carve fake nodes that only have IO holes in it. Unlike the real NUMA case, here we don't have SRAT etc. to know the memory layout beforehand.
Quoted 5 lines
It looks possible to do the work of functions like
split_nodes_equal()in an architecture-independent manner usingearly_node_maprather than dealing with the arch-specific nodes array. That would open the possibility of providing fake nodes on more than one architecture in the future.The functions like
splti_nodes_equaletc. can be abstracted out to arch independent part. I think the only API it needs from arch dependent part is to find out how much real RAM is present in range without have to first doadd_active_range.
Quoted 7 lines
Quoted 3 lines
That is a problem because the ranges must be registered with
add_active_range()to work out how much real RAM is present.Right. And that is why I need
e820_hole_sizefunctionality. BTW, what is the concern in having that function?
Because it provides almost identical functionality to another function. If that can be avoided, it's preferable.
Quoted 34 lines
Quoted 4 lines
Quoted 3 lines
Though as a first step, let us fix the
x86_64(as it doesn't boot when you have sizeable chunk of IO hole and nodes > 4).Quoted 9 lines
Ok.
Quoted 3 lines
I'm also not sure if other archs actually want to have this functionality.
It's possible that the containers people are interested in the possibility of setting up fake nodes as part of a memory controller.
That is precisely why I'm doing it :-)
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
What I think can be done is that you register memory as normal and then split up the nodes into fake nodes. This would remove the need for having
e820_hole_size()reintroduced.Quoted 9 lines
Quoted 3 lines
Are you saying first let the system find out real numa topology and then build fake numa on top of it?
Yes, there is nothing stopping you altering the early_node_map[] before
free_area_init_node()initialises thenode_mem_map. If you do hit a problem, it'll be becausex86_64allocates it's ownnode_mem_mapwithCONFIG_FLAT_NODE_MEM_MAPis set. Is that set when setting up fake nodes?I thought they both (real numa + fake numa) operate on same data structures. I'll have to double check.
-rohit
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Quoted 46 lines · Mel Gorman
Quoted 44 lines · Rohit Seth
Quoted 42 lines · Mel Gorman
Quoted 31 lines · Rohit Seth
Hi Mel,
Quoted 10 lines · Mel Gorman
Quoted 2 lines · Rohit Seth
This patch provides a IO hole size in a given address range.
Hi,
This patch reintroduces a function that doubles up what
absent_pages_in_range(start_pfn, end_pfn). I recognise you do this because you are interested in hole sizes beforeadd_active_range()is called.Right.
Quoted 3 lines
However, what is not clear is why these patches are so specific to
x86_64.Specifically in the fake numa case, we want to make sure that we don't carve fake nodes that only have IO holes in it. Unlike the real NUMA case, here we don't have SRAT etc. to know the memory layout beforehand.
Quoted 5 lines
It looks possible to do the work of functions like
split_nodes_equal()in an architecture-independent manner usingearly_node_maprather than dealing with the arch-specific nodes array. That would open the possibility of providing fake nodes on more than one architecture in the future.Quoted 5 lines
The functions like
splti_nodes_equaletc. can be abstracted out to arch independent part. I think the only API it needs from arch dependent part is to find out how much real RAM is present in range without have to first doadd_active_range.That is a problem because the ranges must be registered with
add_active_range()to work out how much real RAM is present.
Quoted 7 lines
Quoted 3 lines
Right. And that is why I need
e820_hole_sizefunctionality. BTW, what is the concern in having that function?Because it provides almost identical functionality to another function. If that can be avoided, it's preferable.
There are subtle difference in the way two function can be used. They are operating in two different environments. absent_pages work when memory layout is already registered. The e820_hole_size is the (low level arch dependent) function that will be used to find out how the memory lay out is going to be set for the cases when kernel has to itself decide about the layout.
-rohit