make memblock allocator utilize the node's fallback info
Subsystems (5)
cover make memblock allocator utilize the node's fallback info
There are NUMA machines with memory-less node. At present page allocator builds the full fallback info by build_zonelists(). But memblock allocator does not utilize this info. And for memory-less node, memblock allocator just falls back "node 0", without utilizing the nearest node. Unfortunately, the percpu section is allocated by memblock, which is accessed frequently after bootup.
This series aims to improve the performance of per cpu section on memory-less node by feeding node's fallback info to memblock allocator on x86, like we do for page allocator. On other archs, it requires independent effort to setup node to cpumask map ahead.
Pingfan Liu (6):
mm/numa: extract the code of building node fall back list mm/memblock: make full utilization of numa info x86/numa: define numa_init_array() conditional on CONFIG_NUMA x86/numa: concentrate the code of setting cpu to node map x86/numa: push forward the setup of node to cpumask map x86/numa: build node fallback info after setting up node to cpumask
map
arch/x86/include/asm/topology.h | 4 --- arch/x86/kernel/setup.c | 2 ++ arch/x86/kernel/setup_percpu.c | 3 -- arch/x86/mm/numa.c | 40 +++++++++++------------- include/linux/memblock.h | 3 ++ mm/memblock.c | 68 ++++++++++++++++++++++++++++++++++++++--- mm/page_alloc.c | 48 +++++++++++++++++------------ 7 files changed, 114 insertions(+), 54 deletions(-)
Signature · 1 line
2.7.4
PATCH 1/6 mm/numa: extract the code of building node fall back list
In coming patch, memblock allocator also utilizes node fall back list info. Hence extracting the related code for reusing.
--- mm/page_alloc.c | 48 +++++++++++++++++++++++++++++------------------- 1 file changed, 29 insertions(+), 19 deletions(-)
mm/page_alloc.c +29 -19
@@ -5380,6 +5380,32 @@ static void build_thisnode_zonelists(pg_data_t *pgdat) zonerefs->zone_idx =0 ; } +int build_node_order (int *node_oder_array,int sz, +int local_node,nodemask_t *used_mask) +{ +int node, nr_nodes =0 ; +int prev_node = local_node; +int load = nr_online_nodes; + + +while ((node =find_next_best_node (local_node, used_mask)) >=0 + && nr_nodes < sz) { +/* +* We don't want to pressure a particular node. +* So adding penalty to the first node in same +* distance group to make it round-robin. +*/ +if (node_distance (local_node, node) != +node_distance (local_node, prev_node)) + node_load[node] = load; + + node_oder_array[nr_nodes++] = node; + prev_node = node; + load--; + } +return nr_nodes; +} +/* * Build zonelists ordered by zone and nodes within zones. * This results in conserving DMA zone[s] until all Normal memory is
@@ -5390,32 +5416,16 @@ static void build_thisnode_zonelists(pg_data_t *pgdat)static void build_zonelists (pg_data_t *pgdat) {static int node_order[MAX_NUMNODES ]; -int node, load, nr_nodes =0 ; +int local_node, nr_nodes =0 ;nodemask_t used_mask; -int local_node, prev_node;/* NUMA-aware ordering of nodes */ local_node = pgdat->node_id; - load = nr_online_nodes; - prev_node = local_node;nodes_clear (used_mask);memset (node_order,0 ,sizeof (node_order)); -while ((node =find_next_best_node (local_node, &used_mask)) >=0 ) { -/* -* We don't want to pressure a particular node. -* So adding penalty to the first node in same -* distance group to make it round-robin. -*/ -if (node_distance (local_node, node) != -node_distance (local_node, prev_node)) - node_load[node] = load; - - node_order[nr_nodes++] = node; - prev_node = node; - load--; - } - + nr_nodes =build_node_order (node_order,MAX_NUMNODES , + local_node, &used_mask);build_zonelists_in_node_order (pgdat, node_order, nr_nodes);build_thisnode_zonelists (pgdat); }
Signature · 1 line
2.7.4
PATCH 2/6 mm/memblock: make full utilization of numa info
In reply to Pingfan LiuThere are numa machines with memory-less node. When allocating memory for the memory-less node, memblock allocator falls back to 'Node 0' without fully utilizing the nearest node. This hurts the performance, especially for per cpu section. Suppressing this defect by building the full node fall back info for memblock allocator, like what we have done for page allocator.
--- include/linux/memblock.h | 3 +++ mm/memblock.c | 68 ++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 66 insertions(+), 5 deletions(-)
include/linux/memblock.h +3 -0
@@ -342,6 +342,9 @@ void *memblock_alloc_try_nid_nopanic(phys_addr_t size, phys_addr_t align,void *memblock_alloc_try_nid (phys_addr_t size,phys_addr_t align,phys_addr_t min_addr,phys_addr_t max_addr,int nid); +extern int build_node_order (int *node_oder_array,int sz, +int local_node,nodemask_t *used_mask); +void memblock_build_node_order (void );static inline void *__init memblock_alloc (phys_addr_t size,phys_addr_t align) {
mm/memblock.c +63 -5
92 lines not on this page · raw message
Signature · 1 line
2.7.4
kbuild test robot
build botRe: [PATCH 2/6] mm/memblock: make full utilization of numa info
Thank you for the patch! Yet something to improve:
Hi Pingfan,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v5.0-rc4]
[cannot apply to next-20190222]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Pingfan-Liu/mm-numa-extract-the-code-of-building-node-fall-back-list/20190225-143613
config: i386-tinyconfig (attached as .config)
compiler: gcc-8 (Debian 8.2.0-20) 8.2.0 reproduce:
# save the attached .config to linux build tree
make ARCH=i386 All errors (new ones prefixed by >>):
ld: mm/memblock.o: in function `memblock_build_node_order':
>> memblock.c:(.init.text+0x310): undefined reference to `build_node_order'
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
kbuild test robot
build botRe: [PATCH 2/6] mm/memblock: make full utilization of numa info
In reply to Pingfan LiuThank you for the patch! Yet something to improve:
Hi Pingfan,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v5.0-rc4]
[cannot apply to next-20190222]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Pingfan-Liu/mm-numa-extract-the-code-of-building-node-fall-back-list/20190225-143613
config: i386-randconfig-a1-201908 (attached as .config)
compiler: gcc-4.9 (Debian 4.9.4-2) 4.9.4 reproduce:
# save the attached .config to linux build tree
make ARCH=i386 All errors (new ones prefixed by >>):
ld: mm/memblock.o: in function `memblock_build_node_order':
>> mm/memblock.c:1364: undefined reference to `build_node_order'
vim +1364 mm/memblock.c
1342
1343 /*
1344 * build_node_order() relies on cpumask_of_node(), hence arch should set up
1345 * cpumask before calling this func.
1346 */
1347 void __init memblock_build_node_order(void)
1348 {
1349 int nid, i;
1350 nodemask_t used_mask;
1351
1352 node_fallback = memblock_alloc(MAX_NUMNODES * sizeof(int *),
1353 sizeof(int *));
1354 for_each_online_node(nid) {
1355 node_fallback[nid] = memblock_alloc(
1356 num_online_nodes() * sizeof(int), sizeof(int));
1357 for (i = 0; i < num_online_nodes(); i++)
1358 node_fallback[nid][i] = NUMA_NO_NODE;
1359 }
1360
1361 for_each_online_node(nid) {
1362 nodes_clear(used_mask);
1363 node_set(nid, used_mask);
> 1364 build_node_order(node_fallback[nid], num_online_nodes(),
1365 nid, &used_mask);
1366 }
1367 }
1368
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Re: [PATCH 2/6] mm/memblock: make full utilization of numa info
In reply to Pingfan LiuQuoted 4 lines · Pingfan Liu
+/*
+ *build_node_order()relies oncpumask_of_node(), hence arch should
+ * set up cpumask before calling this func.
+ */
Whenever I see comments like this, I wonder what happens if the arch doesn't do this? Do we just crash in early boot in wonderful new ways?
Or do we get a nice message telling us?
Quoted 11 lines
mm/memblock.c +11
-0+void __init memblock_build_node_order (void ) +{ +int nid, i; +nodemask_t used_mask; + + node_fallback =memblock_alloc (MAX_NUMNODES *sizeof (int *), +sizeof (int *)); +for_each_online_node (nid) { + node_fallback[nid] =memblock_alloc ( +num_online_nodes () *sizeof (int ),sizeof (int )); +for (i =0 ; i <num_online_nodes (); i++)
mm/memblock.c phys_addr_t __init memblock_phys_alloc_try_nid(phys_addr_t size, phys_addr_t ali · line 1367
Quoted 10 lines
mm/memblock.c +10
-0+ node_fallback[nid][i] =NUMA_NO_NODE ; + } + +for_each_online_node (nid) { +nodes_clear (used_mask); +node_set (nid, used_mask); +build_node_order (node_fallback[nid],num_online_nodes (), + nid, &used_mask); + } +}
This doesn't get used until patch 6 as far as I can tell. Was there a reason to define it here?
Re: [PATCH 2/6] mm/memblock: make full utilization of numa info
Quoted 11 lines · Dave Hansen
Quoted 4 lines · Pingfan Liu
+/*
+ *build_node_order()relies oncpumask_of_node(), hence arch should
+ * set up cpumask before calling this func.
+ */Whenever I see comments like this, I wonder what happens if the arch doesn't do this? Do we just crash in early boot in wonderful new ways?
Or do we get a nice message telling us?
If doesn't do this, this function will crash. It is a shame but a little hard to work around, since this function is called at early boot stage, things like cpumask_of_node(cpu_to_node(cpu)) can not work reliably, and we lack of an abstract interface to get such information from all archs. So I leave this to arch's developer.
Quoted 16 lines
Quoted 16 lines
mm/memblock.c +16
-0+void __init memblock_build_node_order (void ) +{ +int nid, i; +nodemask_t used_mask; + + node_fallback =memblock_alloc (MAX_NUMNODES *sizeof (int *), +sizeof (int *)); +for_each_online_node (nid) { + node_fallback[nid] =memblock_alloc ( +num_online_nodes () *sizeof (int ),sizeof (int )); +for (i =0 ; i <num_online_nodes (); i++) + node_fallback[nid][i] =NUMA_NO_NODE ; + } + +for_each_online_node (nid) { +nodes_clear (used_mask);
Quoted 9 lines
mm/memblock.c phys_addr_t __init memblock_phys_alloc_try_nid(phys_addr_t size, phys_addr_t ali · line 1367
Quoted 5 lines
mm/memblock.c +5
-0+node_set (nid, used_mask); +build_node_order (node_fallback[nid],num_online_nodes (), + nid, &used_mask); + } +}This doesn't get used until patch 6 as far as I can tell. Was there a reason to define it here?
Yes, it gets used until patch 6. Patch 6 has two groups of pre-requirements [1-2] and [3-5]. Do you think reorder the patches and moving [3-5] ahead of [1-2] is a better choice?
Thanks and regards,
Pingfan
Re: [PATCH 2/6] mm/memblock: make full utilization of numa info
Quoted 6 lines · Pingfan Liu
Quoted 3 lines
This doesn't get used until patch 6 as far as I can tell. Was there a reason to define it here?
Yes, it gets used until patch 6. Patch 6 has two groups of pre-requirements [1-2] and [3-5]. Do you think reorder the patches and moving [3-5] ahead of [1-2] is a better choice?
I'd rather that you just introduce the code along with its first user.
Re: [PATCH 2/6] mm/memblock: make full utilization of numa info
In reply to Pingfan LiuQuoted 5 lines · Pingfan Liu
There are numa machines with memory-less node. When allocating memory for the memory-less node, memblock allocator falls back to 'Node 0' without fully utilizing the nearest node. This hurts the performance, especially for per cpu section. Suppressing this defect by building the full node fall back info for memblock allocator, like what we have done for page allocator.
Is it really necessary to build full node fallback info for memblock and then rebuild it again for the page allocator?
I think it should be possible to split parts of build_all_zonelists_init() that do not touch per-cpu areas into a separate function and call that function after topology detection. Then it would be possible to use local_memory_node() when calling memblock.
Quoted 142 lines
Signed-off-by Pingfan Liu <[email protected]>CC Thomas Gleixner <[email protected]>CC Ingo Molnar <[email protected]>CC Borislav Petkov <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Dave Hansen <[email protected]>CC Vlastimil Babka <[email protected]>CC Mike Rapoport <[email protected]>CC Andrew Morton <[email protected]>CC Mel Gorman <[email protected]>CC Joonsoo Kim <[email protected]>CC Andy Lutomirski <[email protected]>CC Andi Kleen <[email protected]>CC Petr Tesarik <[email protected]>CC Michal Hocko <[email protected]>CC Stephen Rothwell <[email protected]>CC Jonathan Corbet <[email protected]>CC Nicholas Piggin <[email protected]>CC Daniel Vacek <[email protected]>--- include/linux/memblock.h | 3 +++ mm/memblock.c | 68 ++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 66 insertions(+), 5 deletions(-)include/linux/memblock.h +3
-0@@ -342,6 +342,9 @@ void *memblock_alloc_try_nid_nopanic(phys_addr_t size, phys_addr_t align,void *memblock_alloc_try_nid (phys_addr_t size,phys_addr_t align,phys_addr_t min_addr,phys_addr_t max_addr,int nid); +extern int build_node_order (int *node_oder_array,int sz, +int local_node,nodemask_t *used_mask); +void memblock_build_node_order (void );static inline void *__init memblock_alloc (phys_addr_t size,phys_addr_t align) {mm/memblock.c +63
-5@@ -1338,6 +1338,47 @@ phys_addr_t __init memblock_phys_alloc_try_nid(phys_addr_t size, phys_addr_t alireturn memblock_alloc_base (size, align,MEMBLOCK_ALLOC_ACCESSIBLE ); } +static int **node_fallback__initdata ; + +/* +* build_node_order() relies on cpumask_of_node(), hence arch should set up +* cpumask before calling this func. +*/ +void __init memblock_build_node_order (void ) +{ +int nid, i; +nodemask_t used_mask; + + node_fallback =memblock_alloc (MAX_NUMNODES *sizeof (int *), +sizeof (int *)); +for_each_online_node (nid) { + node_fallback[nid] =memblock_alloc ( +num_online_nodes () *sizeof (int ),sizeof (int )); +for (i =0 ; i <num_online_nodes (); i++) + node_fallback[nid][i] =NUMA_NO_NODE ; + } + +for_each_online_node (nid) { +nodes_clear (used_mask); +node_set (nid, used_mask); +build_node_order (node_fallback[nid],num_online_nodes (), + nid, &used_mask); + } +} + +static void __init memblock_free_node_order (void ) +{ +int nid; + +if (!node_fallback) +return ; +for_each_online_node (nid) +memblock_free (__pa (node_fallback[nid]), +num_online_nodes () *sizeof (int )); +memblock_free (__pa (node_fallback),MAX_NUMNODES *sizeof (int *)); + node_fallback =NULL ; +} +/** * memblock_alloc_internal - allocate boot memory block * @size: size of memory block to be allocated in bytes @@ -1370,6 +1411,7 @@ static void * __init memblock_alloc_internal( {phys_addr_t alloc;void *ptr; +int node;enum memblock_flags flags =choose_memblock_flags ();if (WARN_ONCE (nid ==MAX_NUMNODES ,"Usage of MAX_NUMNODES is deprecated. Use NUMA_NO_NODE instead\n" ))@@ -1397,11 +1439,26 @@ static void * __init memblock_alloc_internal( goto done;if (nid !=NUMA_NO_NODE ) { - alloc =memblock_find_in_range_node (size, align, min_addr, - max_addr,NUMA_NO_NODE , - flags); -if (alloc && !memblock_reserve (alloc, size)) - goto done; +if (!node_fallback) { + alloc =memblock_find_in_range_node (size, align, + min_addr, max_addr, +NUMA_NO_NODE , flags); +if (alloc && !memblock_reserve (alloc, size)) + goto done; + }else { +int i; +for (i =0 ; i <num_online_nodes (); i++) { + node = node_fallback[nid][i]; +/* fallback list has all memory nodes */ +if (node ==NUMA_NO_NODE ) +break ; + alloc =memblock_find_in_range_node (size, + align, min_addr, max_addr, + node, flags); +if (alloc && !memblock_reserve (alloc, size)) + goto done; + } + } }if (min_addr) {@@ -1969,6 +2026,7 @@ unsigned long __init memblock_free_all(void)reset_all_zones_managed_pages (); +memblock_free_node_order (); pages =free_low_memory_core_early ();totalram_pages_add (pages);Signature · 1 line
2.7.4
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH 2/6] mm/memblock: make full utilization of numa info
Quoted 11 lines · Mike Rapoport
Quoted 5 lines · Pingfan Liu
There are numa machines with memory-less node. When allocating memory for the memory-less node, memblock allocator falls back to 'Node 0' without fully utilizing the nearest node. This hurts the performance, especially for per cpu section. Suppressing this defect by building the full node fall back info for memblock allocator, like what we have done for page allocator.
Is it really necessary to build full node fallback info for memblock and then rebuild it again for the page allocator?
Do you mean building the full node fallback info once, and share it by both memblock and page allocator? If it is, then node online/offline is the corner case to block this design.
Quoted 5 lines
I think it should be possible to split parts of
build_all_zonelists_init()that do not touch per-cpu areas into a separate function and call that function after topology detection. Then it would be possible to uselocal_memory_node()when calling memblock.
Yes, this is one way but may be with higher pay of changing the code. I will try it.
Thank your for your suggestion.
Best regards,
Pingfan
Quoted 147 lines
Quoted 140 lines
Signed-off-by Pingfan Liu <[email protected]>CC Thomas Gleixner <[email protected]>CC Ingo Molnar <[email protected]>CC Borislav Petkov <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Dave Hansen <[email protected]>CC Vlastimil Babka <[email protected]>CC Mike Rapoport <[email protected]>CC Andrew Morton <[email protected]>CC Mel Gorman <[email protected]>CC Joonsoo Kim <[email protected]>CC Andy Lutomirski <[email protected]>CC Andi Kleen <[email protected]>CC Petr Tesarik <[email protected]>CC Michal Hocko <[email protected]>CC Stephen Rothwell <[email protected]>CC Jonathan Corbet <[email protected]>CC Nicholas Piggin <[email protected]>CC Daniel Vacek <[email protected]>--- include/linux/memblock.h | 3 +++ mm/memblock.c | 68 ++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 66 insertions(+), 5 deletions(-)include/linux/memblock.h +3
-0@@ -342,6 +342,9 @@ void *memblock_alloc_try_nid_nopanic(phys_addr_t size, phys_addr_t align,void *memblock_alloc_try_nid (phys_addr_t size,phys_addr_t align,phys_addr_t min_addr,phys_addr_t max_addr,int nid); +extern int build_node_order (int *node_oder_array,int sz, +int local_node,nodemask_t *used_mask); +void memblock_build_node_order (void );static inline void *__init memblock_alloc (phys_addr_t size,phys_addr_t align) {mm/memblock.c +63
-6@@ -1338,6 +1338,47 @@ phys_addr_t __init memblock_phys_alloc_try_nid(phys_addr_t size, phys_addr_t alireturn memblock_alloc_base (size, align,MEMBLOCK_ALLOC_ACCESSIBLE ); } +static int **node_fallback__initdata ; + +/* +* build_node_order() relies on cpumask_of_node(), hence arch should set up +* cpumask before calling this func. +*/ +void __init memblock_build_node_order (void ) +{ +int nid, i; +nodemask_t used_mask; + + node_fallback =memblock_alloc (MAX_NUMNODES *sizeof (int *), +sizeof (int *)); +for_each_online_node (nid) { + node_fallback[nid] =memblock_alloc ( +num_online_nodes () *sizeof (int ),sizeof (int )); +for (i =0 ; i <num_online_nodes (); i++) + node_fallback[nid][i] =NUMA_NO_NODE ; + } + +for_each_online_node (nid) { +nodes_clear (used_mask); +node_set (nid, used_mask); +build_node_order (node_fallback[nid],num_online_nodes (), + nid, &used_mask); + } +} + +static void __init memblock_free_node_order (void ) +{ +int nid; + +if (!node_fallback) +return ; +for_each_online_node (nid) +memblock_free (__pa (node_fallback[nid]), +num_online_nodes () *sizeof (int )); +memblock_free (__pa (node_fallback),MAX_NUMNODES *sizeof (int *)); + node_fallback =NULL ; +} +/** * memblock_alloc_internal - allocate boot memory block * @size: size of memory block to be allocated in bytes @@ -1370,6 +1411,7 @@ static void * __init memblock_alloc_internal( {phys_addr_t alloc;void *ptr; +int node;enum memblock_flags flags =choose_memblock_flags ();if (WARN_ONCE (nid ==MAX_NUMNODES ,"Usage of MAX_NUMNODES is deprecated. Use NUMA_NO_NODE instead\n" ))@@ -1397,11 +1439,26 @@ static void * __init memblock_alloc_internal( goto done;if (nid !=NUMA_NO_NODE ) { - alloc =memblock_find_in_range_node (size, align, min_addr, - max_addr,NUMA_NO_NODE , - flags); -if (alloc && !memblock_reserve (alloc, size)) - goto done; +if (!node_fallback) { + alloc =memblock_find_in_range_node (size, align, + min_addr, max_addr, +NUMA_NO_NODE , flags); +if (alloc && !memblock_reserve (alloc, size)) + goto done; + }else { +int i; +for (i =0 ; i <num_online_nodes (); i++) { + node = node_fallback[nid][i]; +/* fallback list has all memory nodes */ +if (node ==NUMA_NO_NODE ) +break ; + alloc =memblock_find_in_range_node (size, + align, min_addr, max_addr, + node, flags); +if (alloc && !memblock_reserve (alloc, size)) + goto done; + } + } }if (min_addr) {@@ -1969,6 +2026,7 @@ unsigned long __init memblock_free_all(void)reset_all_zones_managed_pages (); +memblock_free_node_order (); pages =free_low_memory_core_early ();totalram_pages_add (pages); --Quoted 2 lines
2.7.4
--
Sincerely yours,
Mike.
PATCH 3/6 x86/numa: define numa_init_array() conditional on CONFIG_NUMA
In reply to Pingfan LiuFor non-NUMA, it turns out that numa_init_array() has no operations. Make separated definition for non-NUMA and NUMA, so later they can be combined into their counterpart init_cpu_to_node().
--- arch/x86/mm/numa.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/x86/mm/numa.c +4 -0
@@ -599,6 +599,7 @@ static int __init numa_register_memblks(struct numa_meminfo *mi)return 0 ; } +#ifdef CONFIG_NUMA /* * There are unfortunately some poorly designed mainboards around that * only connect memory to a single CPU. This breaks the 1:1 cpu->node
@@ -618,6 +619,9 @@ static void __init numa_init_array(void) rr = next_node_in(rr, node_online_map); } } +#else +static void __init numa_init_array (void ) {} +#endif static int __init numa_init (int (*init_func)(void )) {
Signature · 1 line
2.7.4
Re: [PATCH 3/6] x86/numa: define numa_init_array() conditional on CONFIG_NUMA
arch/x86/mm/numa.c static void __init numa_init_array(void) · line 624
Quoted 11 lines · Pingfan Liu
arch/x86/mm/numa.c +4
-0+#ifdef CONFIG_NUMA /* * There are unfortunately some poorly designed mainboards around that * only connect memory to a single CPU. This breaks the 1:1 cpu->node @@ -618,6 +619,9 @@ static void __init numa_init_array(void) rr = next_node_in(rr, node_online_map); } } +#else +static void __init numa_init_array (void ) {} +#endif
What functional effect does this #ifdef have?
Let's look at the code:
Quoted 4 lines
(quoted diff) +0
-0static void __init numa_init_array(void) { int rr, i;
Quoted 8 lines
rr = first_node(node_online_map);
for (i = 0; i < nr_cpu_ids; i++) {
if (early_cpu_to_node(i) != NUMA_NO_NODE)
continue;
numa_set_node(i, rr);
rr = next_node_in(rr, node_online_map);
}
}and "play compiler" for a bit.
The first iteration will see early_cpu_to_node(i)==1 because:
static inline int early_cpu_to_node(int cpu)
{
return 0;
}if CONFIG_NUMA=n.
In other words, I'm not sure this patch does *anything*.
Re: [PATCH 3/6] x86/numa: define numa_init_array() conditional on CONFIG_NUMA
Quoted 35 lines · Dave Hansen
arch/x86/mm/numa.c static void __init numa_init_array(void) · line 624
Quoted 11 lines · Pingfan Liu
arch/x86/mm/numa.c +4
-0+#ifdef CONFIG_NUMA /* * There are unfortunately some poorly designed mainboards around that * only connect memory to a single CPU. This breaks the 1:1 cpu->node @@ -618,6 +619,9 @@ static void __init numa_init_array(void) rr = next_node_in(rr, node_online_map); } } +#else +static void __init numa_init_array (void ) {} +#endif What functional effect does this #ifdef have?
Let's look at the code:
Quoted 4 lines
(quoted diff) +0
-0static void __init numa_init_array(void) { int rr, i;Quoted 8 lines
rr = first_node(node_online_map); for (i = 0; i < nr_cpu_ids; i++) { if (early_cpu_to_node(i) != NUMA_NO_NODE) continue; numa_set_node(i, rr); rr = next_node_in(rr, node_online_map); } }and "play compiler" for a bit.
The first iteration will see early_cpu_to_node(i)==1 because:
Quoted 8 lines
static inline int
early_cpu_to_node(int cpu){ return 0; }if
CONFIG_NUMA=n.In other words, I'm not sure this patch does *anything*.
I had thought separating [3/6] and [4/6] can ease the review. And I will merge them in next version.
Thanks and regards,
Pingfan
PATCH 4/6 x86/numa: concentrate the code of setting cpu to node map
In reply to Pingfan LiuBoth numa_init_array() and init_cpu_to_node() aim at setting up the cpu to node map, so combining them. And the coming patch will set up node to cpumask map in the combined function.
--- arch/x86/mm/numa.c | 39 +++++++++++++-------------------------- 1 file changed, 13 insertions(+), 26 deletions(-)
arch/x86/mm/numa.c +13 -26
@@ -599,30 +599,6 @@ static int __init numa_register_memblks(struct numa_meminfo *mi)return 0 ; } -#ifdef CONFIG_NUMA -/* -* There are unfortunately some poorly designed mainboards around that -* only connect memory to a single CPU. This breaks the 1:1 cpu->node -* mapping. To avoid this fill in the mapping for all possible CPUs, -* as the number of CPUs is not known yet. We round robin the existing -* nodes. -*/ -static void __init numa_init_array (void ) -{ -int rr, i; - - rr =first_node (node_online_map); -for (i =0 ; i < nr_cpu_ids; i++) { -if (early_cpu_to_node (i) !=NUMA_NO_NODE ) -continue ; -numa_set_node (i, rr); - rr =next_node_in (rr, node_online_map); - } -} -#else -static void __init numa_init_array (void ) {} -#endif -static int __init numa_init (int (*init_func)(void )) {int i;
@@ -675,7 +651,6 @@ static int __init numa_init(int (*init_func)(void))if (!node_online (nid))numa_clear_node (i); } -numa_init_array ();return 0 ; }
@@ -758,14 +733,26 @@ void __init init_cpu_to_node(void) {int cpu;u16 *cpu_to_apicid =early_per_cpu_ptr (x86_cpu_to_apicid); +int rr;BUG_ON (cpu_to_apicid ==NULL ); + rr =first_node (node_online_map);for_each_possible_cpu (cpu) {int node =numa_cpu_node (cpu); -if (node ==NUMA_NO_NODE ) +/* +* There are unfortunately some poorly designed mainboards +* around that only connect memory to a single CPU. This +* breaks the 1:1 cpu->node mapping. To avoid this fill in +* the mapping for all possible CPUs, as the number of CPUs +* is not known yet. We round robin the existing nodes. +*/ +if (node ==NUMA_NO_NODE ) { +numa_set_node (cpu, rr); + rr =next_node_in (rr, node_online_map);continue ; + }if (!node_online (node))init_memory_less_node (node);
Signature · 1 line
2.7.4
Re: [PATCH 4/6] x86/numa: concentrate the code of setting cpu to node map
Quoted 8 lines · Pingfan Liu
arch/x86/mm/numa.c +0
-4-#else -static void __init numa_init_array (void ) {} -#endif -static int __init numa_init (int (*init_func)(void )) {int i;
arch/x86/mm/numa.c static int __init numa_init(int (*init_func)(void)) · line 656
Quoted 7 lines
arch/x86/mm/numa.c +0
-1@@ -675,7 +651,6 @@ static int __init numa_init(int (*init_func)(void))if (!node_online (nid))numa_clear_node (i); } -numa_init_array ();return 0 ; }
Huh, and now the (stub) code that was just added gets removed?
PATCH 5/6 x86/numa: push forward the setup of node to cpumask map
In reply to Pingfan LiuAt present the node to cpumask map is set up until the secondary cpu boot up. But it is too late for the purpose of building node fall back list at early boot stage. Considering that init_cpu_to_node() already owns cpu to node map, it is a good place to set up node to cpumask map too. So do it by calling numa_add_cpu(cpu) in init_cpu_to_node().
--- arch/x86/include/asm/topology.h | 4 ---- arch/x86/kernel/setup_percpu.c | 3 --- arch/x86/mm/numa.c | 5 ++++- 3 files changed, 4 insertions(+), 8 deletions(-)
arch/x86/include/asm/topology.h +0 -4
@@ -73,8 +73,6 @@ static inline const struct cpumask *cpumask_of_node(int node) }#endif -extern void setup_node_to_cpumask_map (void ); -#define pcibus_to_node (bus) __pcibus_to_node(bus)extern int __node_distance (int ,int );
@@ -96,8 +94,6 @@ static inline int early_cpu_to_node(int cpu) return 0; } -static inline void setup_node_to_cpumask_map (void ) { } -#endif #include <asm-generic/topology.h>
arch/x86/kernel/setup_percpu.c +0 -3
@@ -283,9 +283,6 @@ void __init setup_per_cpu_areas(void)early_per_cpu_ptr (x86_cpu_to_node_map) =NULL ;#endif -/* Setup node to cpumask map */ -setup_node_to_cpumask_map (); -/* Setup cpu initialized, callin, callout masks */ setup_cpu_local_masks ();
arch/x86/mm/numa.c +4 -1
@@ -110,7 +110,7 @@ void numa_clear_node(int cpu)* Note: cpumask_of_node() is not valid until after this is done. * (Use CONFIG_DEBUG_PER_CPU_MAPS to check this.) */ -void __init setup_node_to_cpumask_map (void ) +static void __init setup_node_to_cpumask_map (void ) {unsigned int node;
@@ -738,6 +738,7 @@ void __init init_cpu_to_node(void)BUG_ON (cpu_to_apicid ==NULL ); rr =first_node (node_online_map); +setup_node_to_cpumask_map ();for_each_possible_cpu (cpu) {int node =numa_cpu_node (cpu);
@@ -750,6 +751,7 @@ void __init init_cpu_to_node(void)*/ if (node ==NUMA_NO_NODE ) {numa_set_node (cpu, rr); +numa_add_cpu (cpu); rr =next_node_in (rr, node_online_map);continue ; }
@@ -758,6 +760,7 @@ void __init init_cpu_to_node(void)init_memory_less_node (node);numa_set_node (cpu, node); +numa_add_cpu (cpu); } }
Signature · 1 line
2.7.4
Re: [PATCH 5/6] x86/numa: push forward the setup of node to cpumask map
Quoted 5 lines · Pingfan Liu
At present the node to cpumask map is set up until the secondary cpu boot up. But it is too late for the purpose of building node fall back list at early boot stage. Considering that
init_cpu_to_node()already owns cpu to node map, it is a good place to set up node to cpumask map too. So do it by callingnuma_add_cpu(cpu)ininit_cpu_to_node().
It sounds like you have carefully considered the ordering and dependencies here. However, none of that consideration has made it into the code.
Could you please add some comments to the new call-sites to explain why the *must* be where they are?
Re: [PATCH 5/6] x86/numa: push forward the setup of node to cpumask map
Quoted 8 lines · Dave Hansen
Quoted 5 lines · Pingfan Liu
At present the node to cpumask map is set up until the secondary cpu boot up. But it is too late for the purpose of building node fall back list at early boot stage. Considering that
init_cpu_to_node()already owns cpu to node map, it is a good place to set up node to cpumask map too. So do it by callingnuma_add_cpu(cpu)ininit_cpu_to_node().
Quoted 6 lines
It sounds like you have carefully considered the ordering and dependencies here. However, none of that consideration has made it into the code.
Could you please add some comments to the new call-sites to explain why the *must* be where they are?
OK. How about: "building up node fallback list needs cpumask info, so filling cpumask info here"
Thanks for your kindly review.
Regards,
Pingfan
PATCH 6/6 x86/numa: build node fallback info after setting up node to cpumask map
In reply to Pingfan LiuAfter the previous patches, on x86, it is safe to call memblock_build_node_order() after init_cpu_to_node(), which has set up node to cpumask map. So calling memblock_build_node_order() to feed memblock with numa node fall back info.
--- arch/x86/kernel/setup.c | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kernel/setup.c +2 -0
@@ -1245,6 +1245,8 @@ void __init setup_arch(char **cmdline_p)prefill_possible_map ();init_cpu_to_node (); +/* node to cpumask map is ready */ +memblock_build_node_order ();io_apic_init_mappings ();
Signature · 1 line
2.7.4
Quoted 10 lines · Pingfan Liu
There are NUMA machines with memory-less node. At present page allocator builds the full fallback info by
build_zonelists(). But memblock allocator does not utilize this info. And for memory-less node, memblock allocator just falls back "node 0", without utilizing the nearest node. Unfortunately, the percpu section is allocated by memblock, which is accessed frequently after bootup.This series aims to improve the performance of per cpu section on memory-less node by feeding node's fallback info to memblock allocator on x86, like we do for page allocator. On other archs, it requires independent effort to setup node to cpumask map ahead.
Do you have any numbers to tell us how much does this improve the situation?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 8 lines · Michal Hocko
Quoted 6 lines · Pingfan Liu
There are NUMA machines with memory-less node. At present page allocator builds the full fallback info by
build_zonelists(). But memblock allocator does not utilize this info. And for memory-less node, memblock allocator just falls back "node 0", without utilizing the nearest node. Unfortunately, the percpu section is allocated by memblock, which is accessed frequently after bootup.
Quoted 7 lines
Quoted 4 lines
This series aims to improve the performance of per cpu section on memory-less node by feeding node's fallback info to memblock allocator on x86, like we do for page allocator. On other archs, it requires independent effort to setup node to cpumask map ahead.
Do you have any numbers to tell us how much does this improve the situation?
Not yet. At present just based on the fact that we prefer to allocate per cpu area on local node.
Thanks,
Pingfan
Quoted 14 lines · Pingfan Liu
Quoted 13 lines · Michal Hocko
Quoted 10 lines · Pingfan Liu
There are NUMA machines with memory-less node. At present page allocator builds the full fallback info by
build_zonelists(). But memblock allocator does not utilize this info. And for memory-less node, memblock allocator just falls back "node 0", without utilizing the nearest node. Unfortunately, the percpu section is allocated by memblock, which is accessed frequently after bootup.This series aims to improve the performance of per cpu section on memory-less node by feeding node's fallback info to memblock allocator on x86, like we do for page allocator. On other archs, it requires independent effort to setup node to cpumask map ahead.
Quoted 5 lines
Quoted 2 lines
Do you have any numbers to tell us how much does this improve the situation?
Not yet. At present just based on the fact that we prefer to allocate per cpu area on local node.
Yes, we _usually_ do. But the additional complexity should be worth it. And if we find out that the final improvement is not all that great and considering that memory-less setups are crippled anyway then it might turn out we just do not care all that much.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 19 lines · Michal Hocko
Quoted 17 lines · Pingfan Liu
Quoted 15 lines · Michal Hocko
Quoted 10 lines · Pingfan Liu
There are NUMA machines with memory-less node. At present page allocator builds the full fallback info by
build_zonelists(). But memblock allocator does not utilize this info. And for memory-less node, memblock allocator just falls back "node 0", without utilizing the nearest node. Unfortunately, the percpu section is allocated by memblock, which is accessed frequently after bootup.This series aims to improve the performance of per cpu section on memory-less node by feeding node's fallback info to memblock allocator on x86, like we do for page allocator. On other archs, it requires independent effort to setup node to cpumask map ahead.
Do you have any numbers to tell us how much does this improve the situation?
Quoted 8 lines
Quoted 2 lines
Not yet. At present just based on the fact that we prefer to allocate per cpu area on local node.
Yes, we _usually_ do. But the additional complexity should be worth it. And if we find out that the final improvement is not all that great and considering that memory-less setups are crippled anyway then it might turn out we just do not care all that much.
--
I had finished some test on a "Dell Inc. PowerEdge R7425/02MJ3T" machine, which owns 8 numa node. and the topology is:
L1d cache: 32K L1i cache: 64K L2 cache: 512K L3 cache: 4096K NUMA node0 CPU(s): 0,8,16,24 NUMA node1 CPU(s): 2,10,18,26 NUMA node2 CPU(s): 4,12,20,28 NUMA node3 CPU(s): 6,14,22,30 NUMA node4 CPU(s): 1,9,17,25 NUMA node5 CPU(s): 3,11,19,27 NUMA node6 CPU(s): 5,13,21,29 NUMA node7 CPU(s): 7,15,23,31
Here is the basic info about the NUMA machine. cpu 0 and 16 share the same L3 cache. Only node 1 and 5 own memory. Using local node as baseline, the memory write performance suffer 25% drop to nearest node (i.e. writing data from node 0 to 1), and 78% drop to farthest node (i.e. writing from 0 to 5).
I used a user space test case to get the performance difference between the nearest node and the farthest. The case pins two tasks on cpu 0 and 16. The case used two memory chunks, A which emulates a small footprint of per cpu section, and B which emulates a large footprint. Chunk B is always allocated on nearest node, while chunk A switch between nearest node and the farthest to render comparable result. To emulate around 2.5% access to per cpu area, the case composes two groups of writing, 1 time to memory chunk A, then 40 times to chunk B.
On the nearest node, I used 4MB foot print, which is the same size as L3 cache. And varying foot print from 2K -> 4K ->8K to emulate the access to the per cpu section. For 2K and 4K, perf result can not tell the difference exactly, due to the difference is smaller than the variance. For 8K: 1.8% improvement, then the larger footprint, the higher improvement in performance. But 8K means that a module allocates 4K/per cpu in the section. This is not in practice.
So the changes may be not need.
Regards,
Pingfan