powerpc/numa: Return the first online node instead of 0
PATCH 1/2 powerpc/numa: Return the first online node instead of 0
If early cpu to node mapping finds an invalid node id, return the first online node instead of node 0.
With commit e75130f20b1f ("powerpc/numa: Offline memoryless cpuless node 0") the kernel marks node 0 offline in certain scenarios.
--- arch/powerpc/include/asm/topology.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/include/asm/topology.h +1 -1
@@ -60,7 +60,7 @@ static inline int early_cpu_to_node(int cpu)* Fall back to node 0 if nid is unset (it should be, except bugs). * This allows callers to safely do NODE_DATA(early_cpu_to_node(cpu)). */ -return (nid <0 ) ?0 : nid; +return (nid <0 ) ? first_online_node : nid; }int of_drconf_to_nid_single (struct drmem_lmb *lmb);
Signature · 1 line
2.36.1
PATCH 2/2 powerpc/numa: Return the first online node if device tree mapping returns a not online node
While building the cpu_to_node map make sure we always use the online node to build the mapping table. In general this should not be an issue because the kernel use similar lookup mechanism (vphn_get_nid()) to mark nodes online (setup_node_data()). Hence NUMA nodes we find during lookup in numa_setup_cpu() will always be found online.
To keep logic simpler/correct, make sure that if the hypervisor or device tree returned a not online node, don't use that to build the map table. Instead, use the first_online_node.
--- arch/powerpc/mm/numa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/mm/numa.c +1 -1
@@ -741,7 +741,7 @@ static int numa_setup_cpu(unsigned long lcpu)of_node_put (cpu); out_present: -if (nid <0 || !node_possible (nid)) +if (nid <0 || !node_online (nid)) nid = first_online_node;/*
Signature · 1 line
2.36.1
Quoted 16 lines · Aneesh Kumar K.V
If early cpu to node mapping finds an invalid node id, return the first online node instead of node 0.
With commit
e75130f20b1f("powerpc/numa: Offline memoryless cpuless node 0") the kernel marks node 0 offline in certain scenarios.Signed-off-by Aneesh Kumar K.V <[email protected]>--- arch/powerpc/include/asm/topology.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/powerpc/include/asm/topology.h +0
-0
arch/powerpc/include/asm/topology.h static inline int early_cpu_to_node(int cpu) · line 63
Quoted 5 lines
arch/powerpc/include/asm/topology.h +1
-1@@ -60,7 +60,7 @@ static inline int early_cpu_to_node(int cpu)* Fall back to node 0 if nid is unset (it should be, except bugs). * This allows callers to safely do NODE_DATA(early_cpu_to_node(cpu)). */ -return (nid <0 ) ?0 : nid; +return (nid <0 ) ? first_online_node : nid;
Looks good but just two queries.
1. Is there a possibility of early_cpu_to_node() being called before any node is online?
2. first_online_node is actually not a variable, it returns the lowest online node. Right? If lets a early_cpu_to_node() for the same CPU across a node online/offline may end up giving two different nids. Right?
Quoted 6 lines
arch/powerpc/include/asm/topology.h +0
-0}int of_drconf_to_nid_single (struct drmem_lmb *lmb);Signature · 1 line
2.36.1
Signature · 2 lines
Thanks and Regards Srikar Dronamraju
Re: [PATCH 2/2] powerpc/numa: Return the first online node if device tree mapping returns a not online node
In reply to Aneesh Kumar K.VQuoted 9 lines · Aneesh Kumar K.V
While building the
cpu_to_nodemap make sure we always use the online node to build the mapping table. In general this should not be an issue because the kernel use similar lookup mechanism (vphn_get_nid()) to mark nodes online (setup_node_data()). Hence NUMA nodes we find during lookup innuma_setup_cpu()will always be found online.To keep logic simpler/correct, make sure that if the hypervisor or device tree returned a not online node, don't use that to build the map table. Instead, use the
first_online_node.
Why should the returned nid be already online. Are we facing any problem with the current code?
Since general idea is to keep cpu_to_node constant, By assigining it the first_online_node, we may be ending up assigning a wrong node, resulting a performance penalty later on. i.e CPU may actually belong to node 4 but we assigned it to node 1. Also the coregroup id is derived from associavity array. If there is a mismatch between the coregroup id and nid, scheduler will crib.
Quoted 22 lines
Signed-off-by Aneesh Kumar K.V <[email protected]>--- arch/powerpc/mm/numa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/powerpc/mm/numa.c +1
-1@@ -741,7 +741,7 @@ static int numa_setup_cpu(unsigned long lcpu)of_node_put (cpu); out_present: -if (nid <0 || !node_possible (nid)) +if (nid <0 || !node_online (nid)) nid = first_online_node;/* Signature · 1 line
2.36.1
Signature · 2 lines
Thanks and Regards Srikar Dronamraju
Re: [PATCH 2/2] powerpc/numa: Return the first online node if device tree mapping returns a not online node
Quoted 8 lines · Srikar Dronamraju
Quoted 6 lines · Aneesh Kumar K.V
While building the
cpu_to_nodemap make sure we always use the online node to build the mapping table. In general this should not be an issue because the kernel use similar lookup mechanism (vphn_get_nid()) to mark nodes online (setup_node_data()). Hence NUMA nodes we find during lookup innuma_setup_cpu()will always be found online.
Quoted 6 lines
Quoted 3 lines
To keep logic simpler/correct, make sure that if the hypervisor or device tree returned a not online node, don't use that to build the map table. Instead, use the
first_online_node.Why should the returned nid be already online. Are we facing any problem with the current code?
There is no issue with the current upstream code. The problem was detected in a distro kernel where we had partial backport. We didn't initialize NUMA node details correctly. So only a subset of actual NUMA nodes got marked online. This resulted in a crash at completely independent part of the kernel because the kernel try to derefer NODE_DATA() which was not initialized.
Quoted 7 lines
Since general idea is to keep
cpu_to_nodeconstant, By assigining it thefirst_online_node, we may be ending up assigning a wrong node, resulting a performance penalty later on. i.e CPU may actually belong to node 4 but we assigned it to node 1. Also the coregroup id is derived from associavity array. If there is a mismatch between the coregroup id and nid, scheduler will crib.
The changed code will never be executed unless something broke between the NUMA node init (setup_node_data() and numa_setup_cpu()). The reason the code change was made was to keep it consistent with the fact that we only initialize NODE_DATA for online NUMA nodes and hence don't return a possible NUMA node value in numa_setup_cpu().
Quoted 27 lines
Quoted 23 lines
Signed-off-by Aneesh Kumar K.V <[email protected]>--- arch/powerpc/mm/numa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/powerpc/mm/numa.c +1
-1@@ -741,7 +741,7 @@ static int numa_setup_cpu(unsigned long lcpu)of_node_put (cpu); out_present: -if (nid <0 || !node_possible (nid)) +if (nid <0 || !node_online (nid)) nid = first_online_node;/* Signature · 1 line
2.36.1Signature · 2 lines
Thanks and Regards Srikar Dronamraju
Quoted 26 lines · Srikar Dronamraju
Quoted 16 lines · Aneesh Kumar K.V
If early cpu to node mapping finds an invalid node id, return the first online node instead of node 0.
With commit
e75130f20b1f("powerpc/numa: Offline memoryless cpuless node 0") the kernel marks node 0 offline in certain scenarios.Signed-off-by Aneesh Kumar K.V <[email protected]>--- arch/powerpc/include/asm/topology.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/powerpc/include/asm/topology.h +0
-0arch/powerpc/include/asm/topology.h static inline int early_cpu_to_node(int cpu) · line 63
Quoted 5 lines
arch/powerpc/include/asm/topology.h +1
-1@@ -60,7 +60,7 @@ static inline int early_cpu_to_node(int cpu)* Fall back to node 0 if nid is unset (it should be, except bugs). * This allows callers to safely do NODE_DATA(early_cpu_to_node(cpu)). */ -return (nid <0 ) ?0 : nid; +return (nid <0 ) ? first_online_node : nid;Looks good but just two queries.
Quoted 2 lines
1. Is there a possibility of
early_cpu_to_node()being called before any node is online?
The kernel operate with node 0 online most of the early boot and mark it offline in mem_topology_setup() just before parse_numa_properties(). So we should find some nodes set online.
Quoted 5 lines
2.
first_online_nodeis actually not a variable, it returns the lowest online node. Right? If lets aearly_cpu_to_node()for the same CPU across a node online/offline may end up giving two different nids. Right?
The change is specific to a case where we find uninitialized numa_cpu_lookup_table. ie, the firmware didn't specify the mapping for the cpu. I do agree that for such cpus the node mapping can change because of the above. I am not sure whether this can cause any issue in practice. But returning node 0 which can be marked offline can result in crashes?
Quoted 11 lines
Quoted 7 lines
arch/powerpc/include/asm/topology.h +0
-0}int of_drconf_to_nid_single (struct drmem_lmb *lmb);Signature · 1 line
2.36.1Signature · 2 lines
Thanks and Regards Srikar Dronamraju