Skip to contentSkip to conversation

powerpc/numa: Return the first online node instead of 0

PATCH In review Aneesh Kumar K.V 2 participants 6 messages Jun 23, 2022 – Jun 27, 2022 PowerPC

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 powerpc/numa: Return the first online node instead of 0

If early cpu to node mapping finds an invalid node id, return the first online node instead of node 0.

With commit e75130f20b1f ("powerpc/numa: Offline memoryless cpuless node 0") the kernel marks node 0 offline in certain scenarios.

---
 arch/powerpc/include/asm/topology.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/include/asm/topology.h +1 -1
index 8a4d4f4d9749..704088b1d53c 100644
@@ -60,7 +60,7 @@ static inline int early_cpu_to_node(int cpu)
 	 * Fall back to node 0 if nid is unset (it should be, except bugs).
 	 * This allows callers to safely do NODE_DATA(early_cpu_to_node(cpu)).
 	 */
-	return (nid < 0) ? 0 : nid;
+	return (nid < 0) ? first_online_node : nid;
 }
 
 int of_drconf_to_nid_single(struct drmem_lmb *lmb);
Signature · 1 line
2.36.1
#1 lore raw

PATCH 2/2 powerpc/numa: Return the first online node if device tree mapping returns a not online node

While building the cpu_to_node map make sure we always use the online node to build the mapping table. In general this should not be an issue because the kernel use similar lookup mechanism (vphn_get_nid()) to mark nodes online (setup_node_data()). Hence NUMA nodes we find during lookup in numa_setup_cpu() will always be found online.

To keep logic simpler/correct, make sure that if the hypervisor or device tree returned a not online node, don't use that to build the map table. Instead, use the first_online_node.

---
 arch/powerpc/mm/numa.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/mm/numa.c +1 -1
index 0801b2ce9b7d..f387b9eb9dc9 100644
@@ -741,7 +741,7 @@ static int numa_setup_cpu(unsigned long lcpu)
 	of_node_put(cpu);
 
 out_present:
-	if (nid < 0 || !node_possible(nid))
+	if (nid < 0 || !node_online(nid))
 		nid = first_online_node;
 
 	/*
Signature · 1 line
2.36.1
#4 lore raw
In reply to Aneesh Kumar K.V
Quoted 16 lines · Aneesh Kumar K.V

If early cpu to node mapping finds an invalid node id, return the first online node instead of node 0.

With commit e75130f20b1f ("powerpc/numa: Offline memoryless cpuless node 0") the kernel marks node 0 offline in certain scenarios.

---
 arch/powerpc/include/asm/topology.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/include/asm/topology.h +0 -0
index 8a4d4f4d9749..704088b1d53c 100644

arch/powerpc/include/asm/topology.h static inline int early_cpu_to_node(int cpu) · line 63

Quoted 5 lines
arch/powerpc/include/asm/topology.h +1 -1
@@ -60,7 +60,7 @@ static inline int early_cpu_to_node(int cpu)
 	 * Fall back to node 0 if nid is unset (it should be, except bugs).
 	 * This allows callers to safely do NODE_DATA(early_cpu_to_node(cpu)).
 	 */
-	return (nid < 0) ? 0 : nid;
+	return (nid < 0) ? first_online_node : nid;

Looks good but just two queries.

1. Is there a possibility of early_cpu_to_node() being called before any node is online?

2. first_online_node is actually not a variable, it returns the lowest online node. Right? If lets a early_cpu_to_node() for the same CPU across a node online/offline may end up giving two different nids. Right?

Quoted 6 lines
arch/powerpc/include/asm/topology.h +0 -0
 }
 
 int of_drconf_to_nid_single(struct drmem_lmb *lmb);
Signature · 1 line
2.36.1
Signature · 2 lines
Thanks and Regards
Srikar Dronamraju
#2 lore raw

Re: [PATCH 2/2] powerpc/numa: Return the first online node if device tree mapping returns a not online node

In reply to Aneesh Kumar K.V
Quoted 9 lines · Aneesh Kumar K.V

While building the cpu_to_node map make sure we always use the online node to build the mapping table. In general this should not be an issue because the kernel use similar lookup mechanism (vphn_get_nid()) to mark nodes online (setup_node_data()). Hence NUMA nodes we find during lookup in numa_setup_cpu() will always be found online.

To keep logic simpler/correct, make sure that if the hypervisor or device tree returned a not online node, don't use that to build the map table. Instead, use the first_online_node.

Why should the returned nid be already online. Are we facing any problem with the current code?

Since general idea is to keep cpu_to_node constant, By assigining it the first_online_node, we may be ending up assigning a wrong node, resulting a performance penalty later on. i.e CPU may actually belong to node 4 but we assigned it to node 1. Also the coregroup id is derived from associavity array. If there is a mismatch between the coregroup id and nid, scheduler will crib.

Quoted 22 lines
---
 arch/powerpc/mm/numa.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/mm/numa.c +1 -1
index 0801b2ce9b7d..f387b9eb9dc9 100644
@@ -741,7 +741,7 @@ static int numa_setup_cpu(unsigned long lcpu)
 	of_node_put(cpu);
 
 out_present:
-	if (nid < 0 || !node_possible(nid))
+	if (nid < 0 || !node_online(nid))
 		nid = first_online_node;
 
 	/*
Signature · 1 line
2.36.1
Signature · 2 lines
Thanks and Regards
Srikar Dronamraju
#3 lore raw

Re: [PATCH 2/2] powerpc/numa: Return the first online node if device tree mapping returns a not online node

Quoted 8 lines · Srikar Dronamraju
Quoted 6 lines · Aneesh Kumar K.V

While building the cpu_to_node map make sure we always use the online node to build the mapping table. In general this should not be an issue because the kernel use similar lookup mechanism (vphn_get_nid()) to mark nodes online (setup_node_data()). Hence NUMA nodes we find during lookup in numa_setup_cpu() will always be found online.

Quoted 6 lines
Quoted 3 lines

To keep logic simpler/correct, make sure that if the hypervisor or device tree returned a not online node, don't use that to build the map table. Instead, use the first_online_node.

Why should the returned nid be already online. Are we facing any problem with the current code?

There is no issue with the current upstream code. The problem was detected in a distro kernel where we had partial backport. We didn't initialize NUMA node details correctly. So only a subset of actual NUMA nodes got marked online. This resulted in a crash at completely independent part of the kernel because the kernel try to derefer NODE_DATA() which was not initialized.

Quoted 7 lines

Since general idea is to keep cpu_to_node constant, By assigining it the first_online_node, we may be ending up assigning a wrong node, resulting a performance penalty later on. i.e CPU may actually belong to node 4 but we assigned it to node 1. Also the coregroup id is derived from associavity array. If there is a mismatch between the coregroup id and nid, scheduler will crib.

The changed code will never be executed unless something broke between the NUMA node init (setup_node_data() and numa_setup_cpu()). The reason the code change was made was to keep it consistent with the fact that we only initialize NODE_DATA for online NUMA nodes and hence don't return a possible NUMA node value in numa_setup_cpu().

Quoted 27 lines
Quoted 23 lines
---
 arch/powerpc/mm/numa.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/mm/numa.c +1 -1
index 0801b2ce9b7d..f387b9eb9dc9 100644
@@ -741,7 +741,7 @@ static int numa_setup_cpu(unsigned long lcpu)
 	of_node_put(cpu);
 
 out_present:
-	if (nid < 0 || !node_possible(nid))
+	if (nid < 0 || !node_online(nid))
 		nid = first_online_node;
 
 	/*
Signature · 1 line
2.36.1
Signature · 2 lines
Thanks and Regards
Srikar Dronamraju
#5 lore raw
In reply to Srikar Dronamraju
Quoted 26 lines · Srikar Dronamraju
Quoted 16 lines · Aneesh Kumar K.V

If early cpu to node mapping finds an invalid node id, return the first online node instead of node 0.

With commit e75130f20b1f ("powerpc/numa: Offline memoryless cpuless node 0") the kernel marks node 0 offline in certain scenarios.

---
 arch/powerpc/include/asm/topology.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/include/asm/topology.h +0 -0
index 8a4d4f4d9749..704088b1d53c 100644

arch/powerpc/include/asm/topology.h static inline int early_cpu_to_node(int cpu) · line 63

Quoted 5 lines
arch/powerpc/include/asm/topology.h +1 -1
@@ -60,7 +60,7 @@ static inline int early_cpu_to_node(int cpu)
 	 * Fall back to node 0 if nid is unset (it should be, except bugs).
 	 * This allows callers to safely do NODE_DATA(early_cpu_to_node(cpu)).
 	 */
-	return (nid < 0) ? 0 : nid;
+	return (nid < 0) ? first_online_node : nid;

Looks good but just two queries.

Quoted 2 lines

1. Is there a possibility of early_cpu_to_node() being called before any node is online?

The kernel operate with node 0 online most of the early boot and mark it offline in mem_topology_setup() just before parse_numa_properties(). So we should find some nodes set online.

Quoted 5 lines

2. first_online_node is actually not a variable, it returns the lowest online node. Right? If lets a early_cpu_to_node() for the same CPU across a node online/offline may end up giving two different nids. Right?

The change is specific to a case where we find uninitialized numa_cpu_lookup_table. ie, the firmware didn't specify the mapping for the cpu. I do agree that for such cpus the node mapping can change because of the above. I am not sure whether this can cause any issue in practice. But returning node 0 which can be marked offline can result in crashes?

Quoted 11 lines
Quoted 7 lines
arch/powerpc/include/asm/topology.h +0 -0
 }
 
 int of_drconf_to_nid_single(struct drmem_lmb *lmb);
Signature · 1 line
2.36.1
Signature · 2 lines
Thanks and Regards
Srikar Dronamraju