Skip to contentSkip to conversation

make memblock allocator utilize the node's fallback info

PATCH In review Pingfan Liu 5 participants 23 messages Feb 24, 2019 – Mar 5, 2019

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover make memblock allocator utilize the node's fallback info

There are NUMA machines with memory-less node. At present page allocator builds the full fallback info by build_zonelists(). But memblock allocator does not utilize this info. And for memory-less node, memblock allocator just falls back "node 0", without utilizing the nearest node. Unfortunately, the percpu section is allocated by memblock, which is accessed frequently after bootup.

This series aims to improve the performance of per cpu section on memory-less node by feeding node's fallback info to memblock allocator on x86, like we do for page allocator. On other archs, it requires independent effort to setup node to cpumask map ahead.

CC Thomas Gleixner <[email protected]>
CC Ingo Molnar <[email protected]>
CC Borislav Petkov <[email protected]>
CC "H. Peter Anvin" <[email protected]>
CC Dave Hansen <[email protected]>
CC Vlastimil Babka <[email protected]>
CC Mike Rapoport <[email protected]>
CC Andrew Morton <[email protected]>
CC Mel Gorman <[email protected]>
CC Joonsoo Kim <[email protected]>
CC Andy Lutomirski <[email protected]>
CC Andi Kleen <[email protected]>
CC Petr Tesarik <[email protected]>
CC Michal Hocko <[email protected]>
CC Stephen Rothwell <[email protected]>
CC Jonathan Corbet <[email protected]>
CC Nicholas Piggin <[email protected]>
CC Daniel Vacek <[email protected]>

Pingfan Liu (6):
mm/numa: extract the code of building node fall back list mm/memblock: make full utilization of numa info x86/numa: define numa_init_array() conditional on CONFIG_NUMA x86/numa: concentrate the code of setting cpu to node map x86/numa: push forward the setup of node to cpumask map x86/numa: build node fallback info after setting up node to cpumask
map

 arch/x86/include/asm/topology.h |  4 ---
 arch/x86/kernel/setup.c         |  2 ++
 arch/x86/kernel/setup_percpu.c  |  3 --
 arch/x86/mm/numa.c              | 40 +++++++++++-------------
 include/linux/memblock.h        |  3 ++
 mm/memblock.c                   | 68 ++++++++++++++++++++++++++++++++++++++---
 mm/page_alloc.c                 | 48 +++++++++++++++++------------
 7 files changed, 114 insertions(+), 54 deletions(-)
Signature · 1 line
2.7.4
#1 lore raw

PATCH 1/6 mm/numa: extract the code of building node fall back list

In coming patch, memblock allocator also utilizes node fall back list info. Hence extracting the related code for reusing.

CC Thomas Gleixner <[email protected]>
CC Ingo Molnar <[email protected]>
CC Borislav Petkov <[email protected]>
CC "H. Peter Anvin" <[email protected]>
CC Dave Hansen <[email protected]>
CC Vlastimil Babka <[email protected]>
CC Mike Rapoport <[email protected]>
CC Andrew Morton <[email protected]>
CC Mel Gorman <[email protected]>
CC Joonsoo Kim <[email protected]>
CC Andy Lutomirski <[email protected]>
CC Andi Kleen <[email protected]>
CC Petr Tesarik <[email protected]>
CC Michal Hocko <[email protected]>
CC Stephen Rothwell <[email protected]>
CC Jonathan Corbet <[email protected]>
CC Nicholas Piggin <[email protected]>
CC Daniel Vacek <[email protected]>
---
 mm/page_alloc.c | 48 +++++++++++++++++++++++++++++-------------------
 1 file changed, 29 insertions(+), 19 deletions(-)
mm/page_alloc.c +29 -19
index 35fdde0..a6967a1 100644
@@ -5380,6 +5380,32 @@ static void build_thisnode_zonelists(pg_data_t *pgdat)
 	zonerefs->zone_idx = 0;
 }
 
+int build_node_order(int *node_oder_array, int sz,
+	int local_node, nodemask_t *used_mask)
+{
+	int node, nr_nodes = 0;
+	int prev_node = local_node;
+	int load = nr_online_nodes;
+
+
+	while ((node = find_next_best_node(local_node, used_mask)) >= 0
+		&& nr_nodes < sz) {
+		/*
+		 * We don't want to pressure a particular node.
+		 * So adding penalty to the first node in same
+		 * distance group to make it round-robin.
+		 */
+		if (node_distance(local_node, node) !=
+		    node_distance(local_node, prev_node))
+			node_load[node] = load;
+
+		node_oder_array[nr_nodes++] = node;
+		prev_node = node;
+		load--;
+	}
+	return nr_nodes;
+}
+
 /*
  * Build zonelists ordered by zone and nodes within zones.
  * This results in conserving DMA zone[s] until all Normal memory is
@@ -5390,32 +5416,16 @@ static void build_thisnode_zonelists(pg_data_t *pgdat)
 static void build_zonelists(pg_data_t *pgdat)
 {
 	static int node_order[MAX_NUMNODES];
-	int node, load, nr_nodes = 0;
+	int local_node, nr_nodes = 0;
 	nodemask_t used_mask;
-	int local_node, prev_node;
 
 	/* NUMA-aware ordering of nodes */
 	local_node = pgdat->node_id;
-	load = nr_online_nodes;
-	prev_node = local_node;
 	nodes_clear(used_mask);
 
 	memset(node_order, 0, sizeof(node_order));
-	while ((node = find_next_best_node(local_node, &used_mask)) >= 0) {
-		/*
-		 * We don't want to pressure a particular node.
-		 * So adding penalty to the first node in same
-		 * distance group to make it round-robin.
-		 */
-		if (node_distance(local_node, node) !=
-		    node_distance(local_node, prev_node))
-			node_load[node] = load;
-
-		node_order[nr_nodes++] = node;
-		prev_node = node;
-		load--;
-	}
-
+	nr_nodes = build_node_order(node_order, MAX_NUMNODES,
+		local_node, &used_mask);
 	build_zonelists_in_node_order(pgdat, node_order, nr_nodes);
 	build_thisnode_zonelists(pgdat);
 }
Signature · 1 line
2.7.4
#2 lore raw

PATCH 2/6 mm/memblock: make full utilization of numa info

In reply to Pingfan Liu

There are numa machines with memory-less node. When allocating memory for the memory-less node, memblock allocator falls back to 'Node 0' without fully utilizing the nearest node. This hurts the performance, especially for per cpu section. Suppressing this defect by building the full node fall back info for memblock allocator, like what we have done for page allocator.

CC Thomas Gleixner <[email protected]>
CC Ingo Molnar <[email protected]>
CC Borislav Petkov <[email protected]>
CC "H. Peter Anvin" <[email protected]>
CC Dave Hansen <[email protected]>
CC Vlastimil Babka <[email protected]>
CC Mike Rapoport <[email protected]>
CC Andrew Morton <[email protected]>
CC Mel Gorman <[email protected]>
CC Joonsoo Kim <[email protected]>
CC Andy Lutomirski <[email protected]>
CC Andi Kleen <[email protected]>
CC Petr Tesarik <[email protected]>
CC Michal Hocko <[email protected]>
CC Stephen Rothwell <[email protected]>
CC Jonathan Corbet <[email protected]>
CC Nicholas Piggin <[email protected]>
CC Daniel Vacek <[email protected]>
---
 include/linux/memblock.h |  3 +++
 mm/memblock.c            | 68 ++++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 66 insertions(+), 5 deletions(-)
include/linux/memblock.h +3 -0
index 64c41cf..ee999c5 100644
@@ -342,6 +342,9 @@ void *memblock_alloc_try_nid_nopanic(phys_addr_t size, phys_addr_t align,
 void *memblock_alloc_try_nid(phys_addr_t size, phys_addr_t align,
 			     phys_addr_t min_addr, phys_addr_t max_addr,
 			     int nid);
+extern int build_node_order(int *node_oder_array, int sz,
+	int local_node, nodemask_t *used_mask);
+void memblock_build_node_order(void);
 
 static inline void * __init memblock_alloc(phys_addr_t size,  phys_addr_t align)
 {
mm/memblock.c +63 -5

92 lines not on this page · raw message

Signature · 1 line
2.7.4
#3 lore raw

Re: [PATCH 2/6] mm/memblock: make full utilization of numa info

Thank you for the patch! Yet something to improve:

Hi Pingfan,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on linus/master]
[also build test ERROR on v5.0-rc4]
[cannot apply to next-20190222]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url: https://github.com/0day-ci/linux/commits/Pingfan-Liu/mm-numa-extract-the-code-of-building-node-fall-back-list/20190225-143613
config: i386-tinyconfig (attached as .config)
compiler: gcc-8 (Debian 8.2.0-20) 8.2.0 reproduce:

        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

ld: mm/memblock.o: in function `memblock_build_node_order':

>> memblock.c:(.init.text+0x310): undefined reference to `build_node_order'

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#4 lore raw

Re: [PATCH 2/6] mm/memblock: make full utilization of numa info

In reply to Pingfan Liu
Thank you for the patch! Yet something to improve:

Hi Pingfan,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on linus/master]
[also build test ERROR on v5.0-rc4]
[cannot apply to next-20190222]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url: https://github.com/0day-ci/linux/commits/Pingfan-Liu/mm-numa-extract-the-code-of-building-node-fall-back-list/20190225-143613
config: i386-randconfig-a1-201908 (attached as .config)
compiler: gcc-4.9 (Debian 4.9.4-2) 4.9.4 reproduce:

        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

ld: mm/memblock.o: in function `memblock_build_node_order':

>> mm/memblock.c:1364: undefined reference to `build_node_order'

vim +1364 mm/memblock.c

1342
1343 /*
1344 * build_node_order() relies on cpumask_of_node(), hence arch should set up
1345 * cpumask before calling this func.
1346 */
1347 void __init memblock_build_node_order(void)
1348 {
1349 int nid, i;
1350 nodemask_t used_mask;
1351
1352 node_fallback = memblock_alloc(MAX_NUMNODES * sizeof(int *),
1353 sizeof(int *));
1354 for_each_online_node(nid) {
1355 node_fallback[nid] = memblock_alloc(
1356 num_online_nodes() * sizeof(int), sizeof(int));
1357 for (i = 0; i < num_online_nodes(); i++)
1358 node_fallback[nid][i] = NUMA_NO_NODE;
1359 }
1360
1361 for_each_online_node(nid) {
1362 nodes_clear(used_mask);
1363 node_set(nid, used_mask);

> 1364			build_node_order(node_fallback[nid], num_online_nodes(),

1365 nid, &used_mask);
1366 }
1367 }
1368

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#5 lore raw

Re: [PATCH 2/6] mm/memblock: make full utilization of numa info

In reply to Pingfan Liu
Quoted 4 lines · Pingfan Liu

+/*
+ * build_node_order() relies on cpumask_of_node(), hence arch should
+ * set up cpumask before calling this func.
+ */

Whenever I see comments like this, I wonder what happens if the arch doesn't do this? Do we just crash in early boot in wonderful new ways?
Or do we get a nice message telling us?

Quoted 11 lines
mm/memblock.c +11 -0
+void __init memblock_build_node_order(void)
+{
+	int nid, i;
+	nodemask_t used_mask;
+
+	node_fallback = memblock_alloc(MAX_NUMNODES * sizeof(int *),
+		sizeof(int *));
+	for_each_online_node(nid) {
+		node_fallback[nid] = memblock_alloc(
+			num_online_nodes() * sizeof(int), sizeof(int));
+		for (i = 0; i < num_online_nodes(); i++)

mm/memblock.c phys_addr_t __init memblock_phys_alloc_try_nid(phys_addr_t size, phys_addr_t ali · line 1367

Quoted 10 lines
mm/memblock.c +10 -0
+			node_fallback[nid][i] = NUMA_NO_NODE;
+	}
+
+	for_each_online_node(nid) {
+		nodes_clear(used_mask);
+		node_set(nid, used_mask);
+		build_node_order(node_fallback[nid], num_online_nodes(),
+			nid, &used_mask);
+	}
+}

This doesn't get used until patch 6 as far as I can tell. Was there a reason to define it here?

#6 lore raw

Re: [PATCH 2/6] mm/memblock: make full utilization of numa info

Quoted 11 lines · Dave Hansen
Quoted 4 lines · Pingfan Liu

+/*
+ * build_node_order() relies on cpumask_of_node(), hence arch should
+ * set up cpumask before calling this func.
+ */

Whenever I see comments like this, I wonder what happens if the arch doesn't do this? Do we just crash in early boot in wonderful new ways?
Or do we get a nice message telling us?

If doesn't do this, this function will crash. It is a shame but a little hard to work around, since this function is called at early boot stage, things like cpumask_of_node(cpu_to_node(cpu)) can not work reliably, and we lack of an abstract interface to get such information from all archs. So I leave this to arch's developer.

Quoted 16 lines
Quoted 16 lines
mm/memblock.c +16 -0
+void __init memblock_build_node_order(void)
+{
+     int nid, i;
+     nodemask_t used_mask;
+
+     node_fallback = memblock_alloc(MAX_NUMNODES * sizeof(int *),
+             sizeof(int *));
+     for_each_online_node(nid) {
+             node_fallback[nid] = memblock_alloc(
+                     num_online_nodes() * sizeof(int), sizeof(int));
+             for (i = 0; i < num_online_nodes(); i++)
+                     node_fallback[nid][i] = NUMA_NO_NODE;
+     }
+
+     for_each_online_node(nid) {
+             nodes_clear(used_mask);
Quoted 9 lines

mm/memblock.c phys_addr_t __init memblock_phys_alloc_try_nid(phys_addr_t size, phys_addr_t ali · line 1367

Quoted 5 lines
mm/memblock.c +5 -0
+             node_set(nid, used_mask);
+             build_node_order(node_fallback[nid], num_online_nodes(),
+                     nid, &used_mask);
+     }
+}

This doesn't get used until patch 6 as far as I can tell. Was there a reason to define it here?

Yes, it gets used until patch 6. Patch 6 has two groups of pre-requirements [1-2] and [3-5]. Do you think reorder the patches and moving [3-5] ahead of [1-2] is a better choice?

Thanks and regards,
Pingfan

#7 lore raw

Re: [PATCH 2/6] mm/memblock: make full utilization of numa info

Quoted 6 lines · Pingfan Liu
Quoted 3 lines

This doesn't get used until patch 6 as far as I can tell. Was there a reason to define it here?

Yes, it gets used until patch 6. Patch 6 has two groups of pre-requirements [1-2] and [3-5]. Do you think reorder the patches and moving [3-5] ahead of [1-2] is a better choice?

I'd rather that you just introduce the code along with its first user.

#8 lore raw

Re: [PATCH 2/6] mm/memblock: make full utilization of numa info

In reply to Pingfan Liu
Quoted 5 lines · Pingfan Liu

There are numa machines with memory-less node. When allocating memory for the memory-less node, memblock allocator falls back to 'Node 0' without fully utilizing the nearest node. This hurts the performance, especially for per cpu section. Suppressing this defect by building the full node fall back info for memblock allocator, like what we have done for page allocator.

Is it really necessary to build full node fallback info for memblock and then rebuild it again for the page allocator?

I think it should be possible to split parts of build_all_zonelists_init() that do not touch per-cpu areas into a separate function and call that function after topology detection. Then it would be possible to use local_memory_node() when calling memblock.

Quoted 142 lines
CC Thomas Gleixner <[email protected]>
CC Ingo Molnar <[email protected]>
CC Borislav Petkov <[email protected]>
CC "H. Peter Anvin" <[email protected]>
CC Dave Hansen <[email protected]>
CC Vlastimil Babka <[email protected]>
CC Mike Rapoport <[email protected]>
CC Andrew Morton <[email protected]>
CC Mel Gorman <[email protected]>
CC Joonsoo Kim <[email protected]>
CC Andy Lutomirski <[email protected]>
CC Andi Kleen <[email protected]>
CC Petr Tesarik <[email protected]>
CC Michal Hocko <[email protected]>
CC Stephen Rothwell <[email protected]>
CC Jonathan Corbet <[email protected]>
CC Nicholas Piggin <[email protected]>
CC Daniel Vacek <[email protected]>
---
 include/linux/memblock.h |  3 +++
 mm/memblock.c            | 68 ++++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 66 insertions(+), 5 deletions(-)
include/linux/memblock.h +3 -0
index 64c41cf..ee999c5 100644
@@ -342,6 +342,9 @@ void *memblock_alloc_try_nid_nopanic(phys_addr_t size, phys_addr_t align,
 void *memblock_alloc_try_nid(phys_addr_t size, phys_addr_t align,
 			     phys_addr_t min_addr, phys_addr_t max_addr,
 			     int nid);
+extern int build_node_order(int *node_oder_array, int sz,
+	int local_node, nodemask_t *used_mask);
+void memblock_build_node_order(void);
 
 static inline void * __init memblock_alloc(phys_addr_t size,  phys_addr_t align)
 {
mm/memblock.c +63 -5
index 022d4cb..cf78850 100644
@@ -1338,6 +1338,47 @@ phys_addr_t __init memblock_phys_alloc_try_nid(phys_addr_t size, phys_addr_t ali
 	return memblock_alloc_base(size, align, MEMBLOCK_ALLOC_ACCESSIBLE);
 }
 
+static int **node_fallback __initdata;
+
+/*
+ * build_node_order() relies on cpumask_of_node(), hence arch should set up
+ * cpumask before calling this func.
+ */
+void __init memblock_build_node_order(void)
+{
+	int nid, i;
+	nodemask_t used_mask;
+
+	node_fallback = memblock_alloc(MAX_NUMNODES * sizeof(int *),
+		sizeof(int *));
+	for_each_online_node(nid) {
+		node_fallback[nid] = memblock_alloc(
+			num_online_nodes() * sizeof(int), sizeof(int));
+		for (i = 0; i < num_online_nodes(); i++)
+			node_fallback[nid][i] = NUMA_NO_NODE;
+	}
+
+	for_each_online_node(nid) {
+		nodes_clear(used_mask);
+		node_set(nid, used_mask);
+		build_node_order(node_fallback[nid], num_online_nodes(),
+			nid, &used_mask);
+	}
+}
+
+static void __init memblock_free_node_order(void)
+{
+	int nid;
+
+	if (!node_fallback)
+		return;
+	for_each_online_node(nid)
+		memblock_free(__pa(node_fallback[nid]),
+			num_online_nodes() * sizeof(int));
+	memblock_free(__pa(node_fallback), MAX_NUMNODES * sizeof(int *));
+	node_fallback = NULL;
+}
+
 /**
  * memblock_alloc_internal - allocate boot memory block
  * @size: size of memory block to be allocated in bytes
@@ -1370,6 +1411,7 @@ static void * __init memblock_alloc_internal(
 {
 	phys_addr_t alloc;
 	void *ptr;
+	int node;
 	enum memblock_flags flags = choose_memblock_flags();
 
 	if (WARN_ONCE(nid == MAX_NUMNODES, "Usage of MAX_NUMNODES is deprecated. Use NUMA_NO_NODE instead\n"))
@@ -1397,11 +1439,26 @@ static void * __init memblock_alloc_internal(
 		goto done;
 
 	if (nid != NUMA_NO_NODE) {
-		alloc = memblock_find_in_range_node(size, align, min_addr,
-						    max_addr, NUMA_NO_NODE,
-						    flags);
-		if (alloc && !memblock_reserve(alloc, size))
-			goto done;
+		if (!node_fallback) {
+			alloc = memblock_find_in_range_node(size, align,
+					min_addr, max_addr,
+					NUMA_NO_NODE, flags);
+			if (alloc && !memblock_reserve(alloc, size))
+				goto done;
+		} else {
+			int i;
+			for (i = 0; i < num_online_nodes(); i++) {
+				node = node_fallback[nid][i];
+				/* fallback list has all memory nodes */
+				if (node == NUMA_NO_NODE)
+					break;
+				alloc = memblock_find_in_range_node(size,
+						align, min_addr, max_addr,
+						node, flags);
+				if (alloc && !memblock_reserve(alloc, size))
+					goto done;
+			}
+		}
 	}
 
 	if (min_addr) {
@@ -1969,6 +2026,7 @@ unsigned long __init memblock_free_all(void)
 
 	reset_all_zones_managed_pages();
 
+	memblock_free_node_order();
 	pages = free_low_memory_core_early();
 	totalram_pages_add(pages);
 
Signature · 1 line
2.7.4
Signature · 2 lines
Sincerely yours,
Mike.
#9 lore raw

Re: [PATCH 2/6] mm/memblock: make full utilization of numa info

Quoted 11 lines · Mike Rapoport
Quoted 5 lines · Pingfan Liu

There are numa machines with memory-less node. When allocating memory for the memory-less node, memblock allocator falls back to 'Node 0' without fully utilizing the nearest node. This hurts the performance, especially for per cpu section. Suppressing this defect by building the full node fall back info for memblock allocator, like what we have done for page allocator.

Is it really necessary to build full node fallback info for memblock and then rebuild it again for the page allocator?

Do you mean building the full node fallback info once, and share it by both memblock and page allocator? If it is, then node online/offline is the corner case to block this design.

Quoted 5 lines

I think it should be possible to split parts of build_all_zonelists_init() that do not touch per-cpu areas into a separate function and call that function after topology detection. Then it would be possible to use local_memory_node() when calling memblock.

Yes, this is one way but may be with higher pay of changing the code. I will try it.
Thank your for your suggestion.

Best regards,
Pingfan

Quoted 147 lines
Quoted 140 lines
CC Thomas Gleixner <[email protected]>
CC Ingo Molnar <[email protected]>
CC Borislav Petkov <[email protected]>
CC "H. Peter Anvin" <[email protected]>
CC Dave Hansen <[email protected]>
CC Vlastimil Babka <[email protected]>
CC Mike Rapoport <[email protected]>
CC Andrew Morton <[email protected]>
CC Mel Gorman <[email protected]>
CC Joonsoo Kim <[email protected]>
CC Andy Lutomirski <[email protected]>
CC Andi Kleen <[email protected]>
CC Petr Tesarik <[email protected]>
CC Michal Hocko <[email protected]>
CC Stephen Rothwell <[email protected]>
CC Jonathan Corbet <[email protected]>
CC Nicholas Piggin <[email protected]>
CC Daniel Vacek <[email protected]>
---
 include/linux/memblock.h |  3 +++
 mm/memblock.c            | 68 ++++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 66 insertions(+), 5 deletions(-)
include/linux/memblock.h +3 -0
index 64c41cf..ee999c5 100644
@@ -342,6 +342,9 @@ void *memblock_alloc_try_nid_nopanic(phys_addr_t size, phys_addr_t align,
 void *memblock_alloc_try_nid(phys_addr_t size, phys_addr_t align,
                           phys_addr_t min_addr, phys_addr_t max_addr,
                           int nid);
+extern int build_node_order(int *node_oder_array, int sz,
+     int local_node, nodemask_t *used_mask);
+void memblock_build_node_order(void);
 
 static inline void * __init memblock_alloc(phys_addr_t size,  phys_addr_t align)
 {
mm/memblock.c +63 -6
index 022d4cb..cf78850 100644
@@ -1338,6 +1338,47 @@ phys_addr_t __init memblock_phys_alloc_try_nid(phys_addr_t size, phys_addr_t ali
      return memblock_alloc_base(size, align, MEMBLOCK_ALLOC_ACCESSIBLE);
 }
 
+static int **node_fallback __initdata;
+
+/*
+ * build_node_order() relies on cpumask_of_node(), hence arch should set up
+ * cpumask before calling this func.
+ */
+void __init memblock_build_node_order(void)
+{
+     int nid, i;
+     nodemask_t used_mask;
+
+     node_fallback = memblock_alloc(MAX_NUMNODES * sizeof(int *),
+             sizeof(int *));
+     for_each_online_node(nid) {
+             node_fallback[nid] = memblock_alloc(
+                     num_online_nodes() * sizeof(int), sizeof(int));
+             for (i = 0; i < num_online_nodes(); i++)
+                     node_fallback[nid][i] = NUMA_NO_NODE;
+     }
+
+     for_each_online_node(nid) {
+             nodes_clear(used_mask);
+             node_set(nid, used_mask);
+             build_node_order(node_fallback[nid], num_online_nodes(),
+                     nid, &used_mask);
+     }
+}
+
+static void __init memblock_free_node_order(void)
+{
+     int nid;
+
+     if (!node_fallback)
+             return;
+     for_each_online_node(nid)
+             memblock_free(__pa(node_fallback[nid]),
+                     num_online_nodes() * sizeof(int));
+     memblock_free(__pa(node_fallback), MAX_NUMNODES * sizeof(int *));
+     node_fallback = NULL;
+}
+
 /**
  * memblock_alloc_internal - allocate boot memory block
  * @size: size of memory block to be allocated in bytes
@@ -1370,6 +1411,7 @@ static void * __init memblock_alloc_internal(
 {
      phys_addr_t alloc;
      void *ptr;
+     int node;
      enum memblock_flags flags = choose_memblock_flags();
 
      if (WARN_ONCE(nid == MAX_NUMNODES, "Usage of MAX_NUMNODES is deprecated. Use NUMA_NO_NODE instead\n"))
@@ -1397,11 +1439,26 @@ static void * __init memblock_alloc_internal(
              goto done;
 
      if (nid != NUMA_NO_NODE) {
-             alloc = memblock_find_in_range_node(size, align, min_addr,
-                                                 max_addr, NUMA_NO_NODE,
-                                                 flags);
-             if (alloc && !memblock_reserve(alloc, size))
-                     goto done;
+             if (!node_fallback) {
+                     alloc = memblock_find_in_range_node(size, align,
+                                     min_addr, max_addr,
+                                     NUMA_NO_NODE, flags);
+                     if (alloc && !memblock_reserve(alloc, size))
+                             goto done;
+             } else {
+                     int i;
+                     for (i = 0; i < num_online_nodes(); i++) {
+                             node = node_fallback[nid][i];
+                             /* fallback list has all memory nodes */
+                             if (node == NUMA_NO_NODE)
+                                     break;
+                             alloc = memblock_find_in_range_node(size,
+                                             align, min_addr, max_addr,
+                                             node, flags);
+                             if (alloc && !memblock_reserve(alloc, size))
+                                     goto done;
+                     }
+             }
      }
 
      if (min_addr) {
@@ -1969,6 +2026,7 @@ unsigned long __init memblock_free_all(void)
 
      reset_all_zones_managed_pages();
 
+     memblock_free_node_order();
      pages = free_low_memory_core_early();
      totalram_pages_add(pages);
 
--
Quoted 2 lines

2.7.4

--
Sincerely yours,
Mike.

#10 lore raw

PATCH 3/6 x86/numa: define numa_init_array() conditional on CONFIG_NUMA

In reply to Pingfan Liu

For non-NUMA, it turns out that numa_init_array() has no operations. Make separated definition for non-NUMA and NUMA, so later they can be combined into their counterpart init_cpu_to_node().

CC Thomas Gleixner <[email protected]>
CC Ingo Molnar <[email protected]>
CC Borislav Petkov <[email protected]>
CC "H. Peter Anvin" <[email protected]>
CC Dave Hansen <[email protected]>
CC Vlastimil Babka <[email protected]>
CC Mike Rapoport <[email protected]>
CC Andrew Morton <[email protected]>
CC Mel Gorman <[email protected]>
CC Joonsoo Kim <[email protected]>
CC Andy Lutomirski <[email protected]>
CC Andi Kleen <[email protected]>
CC Petr Tesarik <[email protected]>
CC Michal Hocko <[email protected]>
CC Stephen Rothwell <[email protected]>
CC Jonathan Corbet <[email protected]>
CC Nicholas Piggin <[email protected]>
CC Daniel Vacek <[email protected]>
---
 arch/x86/mm/numa.c | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/mm/numa.c +4 -0
index 1308f54..bfe6732 100644
@@ -599,6 +599,7 @@ static int __init numa_register_memblks(struct numa_meminfo *mi)
 	return 0;
 }
 
+#ifdef CONFIG_NUMA
 /*
  * There are unfortunately some poorly designed mainboards around that
  * only connect memory to a single CPU. This breaks the 1:1 cpu->node
@@ -618,6 +619,9 @@ static void __init numa_init_array(void)
 		rr = next_node_in(rr, node_online_map);
 	}
 }
+#else
+static void __init numa_init_array(void) {}
+#endif
 
 static int __init numa_init(int (*init_func)(void))
 {
Signature · 1 line
2.7.4
#11 lore raw

Re: [PATCH 3/6] x86/numa: define numa_init_array() conditional on CONFIG_NUMA

arch/x86/mm/numa.c static void __init numa_init_array(void) · line 624

Quoted 11 lines · Pingfan Liu
arch/x86/mm/numa.c +4 -0
+#ifdef CONFIG_NUMA
 /*
  * There are unfortunately some poorly designed mainboards around that
  * only connect memory to a single CPU. This breaks the 1:1 cpu->node
@@ -618,6 +619,9 @@ static void __init numa_init_array(void)
 		rr = next_node_in(rr, node_online_map);
 	}
 }
+#else
+static void __init numa_init_array(void) {}
+#endif

What functional effect does this #ifdef have?

Let's look at the code:

Quoted 4 lines
(quoted diff) +0 -0
 static void __init numa_init_array(void)
 {
        int rr, i;
Quoted 8 lines
        rr = first_node(node_online_map);
        for (i = 0; i < nr_cpu_ids; i++) {
                if (early_cpu_to_node(i) != NUMA_NO_NODE)
                        continue;
                numa_set_node(i, rr);
                rr = next_node_in(rr, node_online_map);
        }
}

and "play compiler" for a bit.

The first iteration will see early_cpu_to_node(i)==1 because:

static inline int early_cpu_to_node(int cpu)

{
        return 0;
}

if CONFIG_NUMA=n.

In other words, I'm not sure this patch does *anything*.

#12 lore raw

Re: [PATCH 3/6] x86/numa: define numa_init_array() conditional on CONFIG_NUMA

Quoted 35 lines · Dave Hansen

arch/x86/mm/numa.c static void __init numa_init_array(void) · line 624

Quoted 11 lines · Pingfan Liu
arch/x86/mm/numa.c +4 -0
+#ifdef CONFIG_NUMA
 /*
  * There are unfortunately some poorly designed mainboards around that
  * only connect memory to a single CPU. This breaks the 1:1 cpu->node
@@ -618,6 +619,9 @@ static void __init numa_init_array(void)
              rr = next_node_in(rr, node_online_map);
      }
 }
+#else
+static void __init numa_init_array(void) {}
+#endif

What functional effect does this #ifdef have?

Let's look at the code:

Quoted 4 lines
(quoted diff) +0 -0
 static void __init numa_init_array(void)
 {
        int rr, i;
Quoted 8 lines
        rr = first_node(node_online_map);
        for (i = 0; i < nr_cpu_ids; i++) {
                if (early_cpu_to_node(i) != NUMA_NO_NODE)
                        continue;
                numa_set_node(i, rr);
                rr = next_node_in(rr, node_online_map);
        }
}

and "play compiler" for a bit.

The first iteration will see early_cpu_to_node(i)==1 because:

Quoted 8 lines

static inline int early_cpu_to_node(int cpu)

{
        return 0;
}

if CONFIG_NUMA=n.

In other words, I'm not sure this patch does *anything*.

I had thought separating [3/6] and [4/6] can ease the review. And I will merge them in next version.

Thanks and regards,
Pingfan

#13 lore raw

PATCH 4/6 x86/numa: concentrate the code of setting cpu to node map

In reply to Pingfan Liu

Both numa_init_array() and init_cpu_to_node() aim at setting up the cpu to node map, so combining them. And the coming patch will set up node to cpumask map in the combined function.

CC Thomas Gleixner <[email protected]>
CC Ingo Molnar <[email protected]>
CC Borislav Petkov <[email protected]>
CC "H. Peter Anvin" <[email protected]>
CC Dave Hansen <[email protected]>
CC Vlastimil Babka <[email protected]>
CC Mike Rapoport <[email protected]>
CC Andrew Morton <[email protected]>
CC Mel Gorman <[email protected]>
CC Joonsoo Kim <[email protected]>
CC Andy Lutomirski <[email protected]>
CC Andi Kleen <[email protected]>
CC Petr Tesarik <[email protected]>
CC Michal Hocko <[email protected]>
CC Stephen Rothwell <[email protected]>
CC Jonathan Corbet <[email protected]>
CC Nicholas Piggin <[email protected]>
CC Daniel Vacek <[email protected]>
---
 arch/x86/mm/numa.c | 39 +++++++++++++--------------------------
 1 file changed, 13 insertions(+), 26 deletions(-)
arch/x86/mm/numa.c +13 -26
index bfe6732..c8dd7af 100644
@@ -599,30 +599,6 @@ static int __init numa_register_memblks(struct numa_meminfo *mi)
 	return 0;
 }
 
-#ifdef CONFIG_NUMA
-/*
- * There are unfortunately some poorly designed mainboards around that
- * only connect memory to a single CPU. This breaks the 1:1 cpu->node
- * mapping. To avoid this fill in the mapping for all possible CPUs,
- * as the number of CPUs is not known yet. We round robin the existing
- * nodes.
- */
-static void __init numa_init_array(void)
-{
-	int rr, i;
-
-	rr = first_node(node_online_map);
-	for (i = 0; i < nr_cpu_ids; i++) {
-		if (early_cpu_to_node(i) != NUMA_NO_NODE)
-			continue;
-		numa_set_node(i, rr);
-		rr = next_node_in(rr, node_online_map);
-	}
-}
-#else
-static void __init numa_init_array(void) {}
-#endif
-
 static int __init numa_init(int (*init_func)(void))
 {
 	int i;
@@ -675,7 +651,6 @@ static int __init numa_init(int (*init_func)(void))
 		if (!node_online(nid))
 			numa_clear_node(i);
 	}
-	numa_init_array();
 
 	return 0;
 }
@@ -758,14 +733,26 @@ void __init init_cpu_to_node(void)
 {
 	int cpu;
 	u16 *cpu_to_apicid = early_per_cpu_ptr(x86_cpu_to_apicid);
+	int rr;
 
 	BUG_ON(cpu_to_apicid == NULL);
+	rr = first_node(node_online_map);
 
 	for_each_possible_cpu(cpu) {
 		int node = numa_cpu_node(cpu);
 
-		if (node == NUMA_NO_NODE)
+		/*
+		 * There are unfortunately some poorly designed mainboards
+		 * around that only connect memory to a single CPU. This
+		 * breaks the 1:1 cpu->node mapping. To avoid this fill in
+		 * the mapping for all possible CPUs, as the number of CPUs
+		 * is not known yet. We round robin the existing nodes.
+		 */
+		if (node == NUMA_NO_NODE) {
+			numa_set_node(cpu, rr);
+			rr = next_node_in(rr, node_online_map);
 			continue;
+		}
 
 		if (!node_online(node))
 			init_memory_less_node(node);
Signature · 1 line
2.7.4
#14 lore raw

Re: [PATCH 4/6] x86/numa: concentrate the code of setting cpu to node map

Quoted 8 lines · Pingfan Liu
arch/x86/mm/numa.c +0 -4
-#else
-static void __init numa_init_array(void) {}
-#endif
-
 static int __init numa_init(int (*init_func)(void))
 {
 	int i;

arch/x86/mm/numa.c static int __init numa_init(int (*init_func)(void)) · line 656

Quoted 7 lines
arch/x86/mm/numa.c +0 -1
@@ -675,7 +651,6 @@ static int __init numa_init(int (*init_func)(void))
 		if (!node_online(nid))
 			numa_clear_node(i);
 	}
-	numa_init_array();
 
 	return 0;
 }

Huh, and now the (stub) code that was just added gets removed?

#15 lore raw

PATCH 5/6 x86/numa: push forward the setup of node to cpumask map

In reply to Pingfan Liu

At present the node to cpumask map is set up until the secondary cpu boot up. But it is too late for the purpose of building node fall back list at early boot stage. Considering that init_cpu_to_node() already owns cpu to node map, it is a good place to set up node to cpumask map too. So do it by calling numa_add_cpu(cpu) in init_cpu_to_node().

CC Thomas Gleixner <[email protected]>
CC Ingo Molnar <[email protected]>
CC Borislav Petkov <[email protected]>
CC "H. Peter Anvin" <[email protected]>
CC Dave Hansen <[email protected]>
CC Vlastimil Babka <[email protected]>
CC Mike Rapoport <[email protected]>
CC Andrew Morton <[email protected]>
CC Mel Gorman <[email protected]>
CC Joonsoo Kim <[email protected]>
CC Andy Lutomirski <[email protected]>
CC Andi Kleen <[email protected]>
CC Petr Tesarik <[email protected]>
CC Michal Hocko <[email protected]>
CC Stephen Rothwell <[email protected]>
CC Jonathan Corbet <[email protected]>
CC Nicholas Piggin <[email protected]>
CC Daniel Vacek <[email protected]>
---
 arch/x86/include/asm/topology.h | 4 ----
 arch/x86/kernel/setup_percpu.c  | 3 ---
 arch/x86/mm/numa.c              | 5 ++++-
 3 files changed, 4 insertions(+), 8 deletions(-)
arch/x86/include/asm/topology.h +0 -4
index 453cf38..fad77c7 100644
@@ -73,8 +73,6 @@ static inline const struct cpumask *cpumask_of_node(int node)
 }
 #endif
 
-extern void setup_node_to_cpumask_map(void);
-
 #define pcibus_to_node(bus) __pcibus_to_node(bus)
 
 extern int __node_distance(int, int);
@@ -96,8 +94,6 @@ static inline int early_cpu_to_node(int cpu)
 	return 0;
 }
 
-static inline void setup_node_to_cpumask_map(void) { }
-
 #endif
 
 #include <asm-generic/topology.h>
arch/x86/kernel/setup_percpu.c +0 -3
index e8796fc..206fa43 100644
@@ -283,9 +283,6 @@ void __init setup_per_cpu_areas(void)
 	early_per_cpu_ptr(x86_cpu_to_node_map) = NULL;
 #endif
 
-	/* Setup node to cpumask map */
-	setup_node_to_cpumask_map();
-
 	/* Setup cpu initialized, callin, callout masks */
 	setup_cpu_local_masks();
 
arch/x86/mm/numa.c +4 -1
index c8dd7af..8d73e2273 100644
@@ -110,7 +110,7 @@ void numa_clear_node(int cpu)
  * Note: cpumask_of_node() is not valid until after this is done.
  * (Use CONFIG_DEBUG_PER_CPU_MAPS to check this.)
  */
-void __init setup_node_to_cpumask_map(void)
+static void __init setup_node_to_cpumask_map(void)
 {
 	unsigned int node;
 
@@ -738,6 +738,7 @@ void __init init_cpu_to_node(void)
 	BUG_ON(cpu_to_apicid == NULL);
 	rr = first_node(node_online_map);
 
+	setup_node_to_cpumask_map();
 	for_each_possible_cpu(cpu) {
 		int node = numa_cpu_node(cpu);
 
@@ -750,6 +751,7 @@ void __init init_cpu_to_node(void)
 		 */
 		if (node == NUMA_NO_NODE) {
 			numa_set_node(cpu, rr);
+			numa_add_cpu(cpu);
 			rr = next_node_in(rr, node_online_map);
 			continue;
 		}
@@ -758,6 +760,7 @@ void __init init_cpu_to_node(void)
 			init_memory_less_node(node);
 
 		numa_set_node(cpu, node);
+		numa_add_cpu(cpu);
 	}
 }
 
Signature · 1 line
2.7.4
#16 lore raw

Re: [PATCH 5/6] x86/numa: push forward the setup of node to cpumask map

Quoted 5 lines · Pingfan Liu

At present the node to cpumask map is set up until the secondary cpu boot up. But it is too late for the purpose of building node fall back list at early boot stage. Considering that init_cpu_to_node() already owns cpu to node map, it is a good place to set up node to cpumask map too. So do it by calling numa_add_cpu(cpu) in init_cpu_to_node().

It sounds like you have carefully considered the ordering and dependencies here. However, none of that consideration has made it into the code.

Could you please add some comments to the new call-sites to explain why the *must* be where they are?

#17 lore raw

Re: [PATCH 5/6] x86/numa: push forward the setup of node to cpumask map

Quoted 8 lines · Dave Hansen
Quoted 5 lines · Pingfan Liu

At present the node to cpumask map is set up until the secondary cpu boot up. But it is too late for the purpose of building node fall back list at early boot stage. Considering that init_cpu_to_node() already owns cpu to node map, it is a good place to set up node to cpumask map too. So do it by calling numa_add_cpu(cpu) in init_cpu_to_node().

Quoted 6 lines

It sounds like you have carefully considered the ordering and dependencies here. However, none of that consideration has made it into the code.

Could you please add some comments to the new call-sites to explain why the *must* be where they are?

OK. How about: "building up node fallback list needs cpumask info, so filling cpumask info here"
Thanks for your kindly review.

Regards,
Pingfan

#18 lore raw

PATCH 6/6 x86/numa: build node fallback info after setting up node to cpumask map

In reply to Pingfan Liu

After the previous patches, on x86, it is safe to call memblock_build_node_order() after init_cpu_to_node(), which has set up node to cpumask map. So calling memblock_build_node_order() to feed memblock with numa node fall back info.

CC Thomas Gleixner <[email protected]>
CC Ingo Molnar <[email protected]>
CC Borislav Petkov <[email protected]>
CC "H. Peter Anvin" <[email protected]>
CC Dave Hansen <[email protected]>
CC Vlastimil Babka <[email protected]>
CC Mike Rapoport <[email protected]>
CC Andrew Morton <[email protected]>
CC Mel Gorman <[email protected]>
CC Joonsoo Kim <[email protected]>
CC Andy Lutomirski <[email protected]>
CC Andi Kleen <[email protected]>
CC Petr Tesarik <[email protected]>
CC Michal Hocko <[email protected]>
CC Stephen Rothwell <[email protected]>
CC Jonathan Corbet <[email protected]>
CC Nicholas Piggin <[email protected]>
CC Daniel Vacek <[email protected]>
---
 arch/x86/kernel/setup.c | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/kernel/setup.c +2 -0
index 3d872a5..3ec1a6e 100644
@@ -1245,6 +1245,8 @@ void __init setup_arch(char **cmdline_p)
 	prefill_possible_map();
 
 	init_cpu_to_node();
+	/* node to cpumask map is ready */
+	memblock_build_node_order();
 
 	io_apic_init_mappings();
 
Signature · 1 line
2.7.4
#19 lore raw
In reply to Pingfan Liu
Quoted 10 lines · Pingfan Liu

There are NUMA machines with memory-less node. At present page allocator builds the full fallback info by build_zonelists(). But memblock allocator does not utilize this info. And for memory-less node, memblock allocator just falls back "node 0", without utilizing the nearest node. Unfortunately, the percpu section is allocated by memblock, which is accessed frequently after bootup.

This series aims to improve the performance of per cpu section on memory-less node by feeding node's fallback info to memblock allocator on x86, like we do for page allocator. On other archs, it requires independent effort to setup node to cpumask map ahead.

Do you have any numbers to tell us how much does this improve the situation?

Signature · 2 lines
Michal Hocko
SUSE Labs
#20 lore raw
Quoted 8 lines · Michal Hocko
Quoted 6 lines · Pingfan Liu

There are NUMA machines with memory-less node. At present page allocator builds the full fallback info by build_zonelists(). But memblock allocator does not utilize this info. And for memory-less node, memblock allocator just falls back "node 0", without utilizing the nearest node. Unfortunately, the percpu section is allocated by memblock, which is accessed frequently after bootup.

Quoted 7 lines
Quoted 4 lines

This series aims to improve the performance of per cpu section on memory-less node by feeding node's fallback info to memblock allocator on x86, like we do for page allocator. On other archs, it requires independent effort to setup node to cpumask map ahead.

Do you have any numbers to tell us how much does this improve the situation?

Not yet. At present just based on the fact that we prefer to allocate per cpu area on local node.

Thanks,
Pingfan

#21 lore raw
Quoted 14 lines · Pingfan Liu
Quoted 13 lines · Michal Hocko
Quoted 10 lines · Pingfan Liu

There are NUMA machines with memory-less node. At present page allocator builds the full fallback info by build_zonelists(). But memblock allocator does not utilize this info. And for memory-less node, memblock allocator just falls back "node 0", without utilizing the nearest node. Unfortunately, the percpu section is allocated by memblock, which is accessed frequently after bootup.

This series aims to improve the performance of per cpu section on memory-less node by feeding node's fallback info to memblock allocator on x86, like we do for page allocator. On other archs, it requires independent effort to setup node to cpumask map ahead.

Quoted 5 lines
Quoted 2 lines

Do you have any numbers to tell us how much does this improve the situation?

Not yet. At present just based on the fact that we prefer to allocate per cpu area on local node.

Yes, we _usually_ do. But the additional complexity should be worth it. And if we find out that the final improvement is not all that great and considering that memory-less setups are crippled anyway then it might turn out we just do not care all that much.

Signature · 2 lines
Michal Hocko
SUSE Labs
#22 lore raw
Quoted 19 lines · Michal Hocko
Quoted 17 lines · Pingfan Liu
Quoted 15 lines · Michal Hocko
Quoted 10 lines · Pingfan Liu

There are NUMA machines with memory-less node. At present page allocator builds the full fallback info by build_zonelists(). But memblock allocator does not utilize this info. And for memory-less node, memblock allocator just falls back "node 0", without utilizing the nearest node. Unfortunately, the percpu section is allocated by memblock, which is accessed frequently after bootup.

This series aims to improve the performance of per cpu section on memory-less node by feeding node's fallback info to memblock allocator on x86, like we do for page allocator. On other archs, it requires independent effort to setup node to cpumask map ahead.

Do you have any numbers to tell us how much does this improve the situation?

Quoted 8 lines
Quoted 2 lines

Not yet. At present just based on the fact that we prefer to allocate per cpu area on local node.

Yes, we _usually_ do. But the additional complexity should be worth it. And if we find out that the final improvement is not all that great and considering that memory-less setups are crippled anyway then it might turn out we just do not care all that much.
--

I had finished some test on a "Dell Inc. PowerEdge R7425/02MJ3T" machine, which owns 8 numa node. and the topology is:

L1d cache:           32K
L1i cache:           64K
L2 cache:            512K
L3 cache:            4096K
NUMA node0 CPU(s):   0,8,16,24
NUMA node1 CPU(s):   2,10,18,26
NUMA node2 CPU(s):   4,12,20,28
NUMA node3 CPU(s):   6,14,22,30
NUMA node4 CPU(s):   1,9,17,25
NUMA node5 CPU(s):   3,11,19,27
NUMA node6 CPU(s):   5,13,21,29
NUMA node7 CPU(s):   7,15,23,31

Here is the basic info about the NUMA machine. cpu 0 and 16 share the same L3 cache. Only node 1 and 5 own memory. Using local node as baseline, the memory write performance suffer 25% drop to nearest node (i.e. writing data from node 0 to 1), and 78% drop to farthest node (i.e. writing from 0 to 5).

I used a user space test case to get the performance difference between the nearest node and the farthest. The case pins two tasks on cpu 0 and 16. The case used two memory chunks, A which emulates a small footprint of per cpu section, and B which emulates a large footprint. Chunk B is always allocated on nearest node, while chunk A switch between nearest node and the farthest to render comparable result. To emulate around 2.5% access to per cpu area, the case composes two groups of writing, 1 time to memory chunk A, then 40 times to chunk B.

On the nearest node, I used 4MB foot print, which is the same size as L3 cache. And varying foot print from 2K -> 4K ->8K to emulate the access to the per cpu section. For 2K and 4K, perf result can not tell the difference exactly, due to the difference is smaller than the variance. For 8K: 1.8% improvement, then the larger footprint, the higher improvement in performance. But 8K means that a module allocates 4K/per cpu in the section. This is not in practice.

So the changes may be not need.

Regards,
Pingfan