ppc64/numa: consider the max numa node for migratable LPAR
PATCH v2 1/1 ppc64/numa: consider the max numa node for migratable LPAR
When a LPAR is migratable, we should consider the maximum possible NUMA node instead the number of NUMA node from the actual system.
The DT property 'ibm,current-associativity-domains' is defining the maximum number of nodes the LPAR can see when running on that box. But if the LPAR is being migrated on another box, it may seen up to the nodes defined by 'ibm,max-associativity-domains'. So if a LPAR is migratable, that value should be used.
Unfortunately, there is no easy way to know if a LPAR is migratable or not. The hypervisor is exporting the property 'ibm,migratable-partition' in the case it set to migrate partition, but that would not mean that the current partition is migratable.
Without this patch, when a LPAR is started on a 2 nodes box and then migrated to a 3 nodes box, the hypervisor may spread the LPAR's CPUs on the 3rd node. In that case if a CPU from that 3rd node is added to the LPAR, it will be wrongly assigned to the node because the kernel has been set to use up to 2 nodes (the configuration of the departure node). With this patch applies, the CPU is correctly added to the 3rd node.
---
V2: Address Srikar's comments
- Fix the commit message
- Use pr_info instead printk(KERN_INFO..)
--- arch/powerpc/mm/numa.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
arch/powerpc/mm/numa.c +10 -3
@@ -893,7 +893,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)static void __init find_possible_nodes (void ) {struct device_node *rtas; -const __be32 *domains; +const __be32 *domains =NULL ;int prop_length, max_nodes;u32 i;
@@ -909,9 +909,14 @@ static void __init find_possible_nodes(void)* it doesn't exist, then fallback on ibm,max-associativity-domains. * Current denotes what the platform can support compared to max * which denotes what the Hypervisor can support. +* +* If the LPAR is migratable, new nodes might be activated after a LPM, +* so we should consider the max number in that case. */ - domains =of_get_property (rtas,"ibm,current-associativity-domains" , - &prop_length); +if (!of_get_property (of_root,"ibm,migratable-partition" ,NULL )) + domains =of_get_property (rtas, +"ibm,current-associativity-domains" , + &prop_length);if (!domains) { domains =of_get_property (rtas,"ibm,max-associativity-domains" , &prop_length);
@@ -920,6 +925,8 @@ static void __init find_possible_nodes(void) } max_nodes =of_read_number (&domains[min_common_depth],1 ); +pr_info ("Partition configured for %d NUMA nodes.\n" , max_nodes); +for (i =0 ; i < max_nodes; i++) {if (!node_possible (i))node_set (i, node_possible_map);
Signature · 1 line
2.31.1
Hi Michael,
Do you mind taking this patch of 5.14?
Thanks,
Laurent.
Le 11/05/2021 à 09:31, Laurent Dufour a écrit :
Quoted 72 lines
When a LPAR is migratable, we should consider the maximum possible NUMA node instead the number of NUMA node from the actual system.
The DT property 'ibm,current-associativity-domains' is defining the maximum number of nodes the LPAR can see when running on that box. But if the LPAR is being migrated on another box, it may seen up to the nodes defined by 'ibm,max-associativity-domains'. So if a LPAR is migratable, that value should be used.
Unfortunately, there is no easy way to know if a LPAR is migratable or not. The hypervisor is exporting the property 'ibm,migratable-partition' in the case it set to migrate partition, but that would not mean that the current partition is migratable.
Without this patch, when a LPAR is started on a 2 nodes box and then migrated to a 3 nodes box, the hypervisor may spread the LPAR's CPUs on the 3rd node. In that case if a CPU from that 3rd node is added to the LPAR, it will be wrongly assigned to the node because the kernel has been set to use up to 2 nodes (the configuration of the departure node). With this patch applies, the CPU is correctly added to the 3rd node.
Fixes f9f130ff2ec9 ("powerpc/numa: Detect support for coregroup")Reviewed-by Srikar Dronamraju <[email protected]>Signed-off-by Laurent Dufour <[email protected]>---
V2: Address Srikar's comments
- Fix the commit message
- Usepr_infoinsteadprintk(KERN_INFO..)--- arch/powerpc/mm/numa.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)arch/powerpc/mm/numa.c +10
-3@@ -893,7 +893,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)static void __init find_possible_nodes (void ) {struct device_node *rtas; -const __be32 *domains; +const __be32 *domains =NULL ;int prop_length, max_nodes;u32 i;@@ -909,9 +909,14 @@ static void __init find_possible_nodes(void)* it doesn't exist, then fallback on ibm,max-associativity-domains. * Current denotes what the platform can support compared to max * which denotes what the Hypervisor can support. +* +* If the LPAR is migratable, new nodes might be activated after a LPM, +* so we should consider the max number in that case. */ - domains =of_get_property (rtas,"ibm,current-associativity-domains" , - &prop_length); +if (!of_get_property (of_root,"ibm,migratable-partition" ,NULL )) + domains =of_get_property (rtas, +"ibm,current-associativity-domains" , + &prop_length);if (!domains) { domains =of_get_property (rtas,"ibm,max-associativity-domains" , &prop_length);@@ -920,6 +925,8 @@ static void __init find_possible_nodes(void) } max_nodes =of_read_number (&domains[min_common_depth],1 ); +pr_info ("Partition configured for %d NUMA nodes.\n" , max_nodes); +for (i =0 ; i < max_nodes; i++) {if (!node_possible (i))node_set (i, node_possible_map);
Hi Michael,
Is there a way to get that patch in 5.14?
Thanks,
Laurent.
Le 11/05/2021 à 09:31, Laurent Dufour a écrit :
Quoted 72 lines
When a LPAR is migratable, we should consider the maximum possible NUMA node instead the number of NUMA node from the actual system.
The DT property 'ibm,current-associativity-domains' is defining the maximum number of nodes the LPAR can see when running on that box. But if the LPAR is being migrated on another box, it may seen up to the nodes defined by 'ibm,max-associativity-domains'. So if a LPAR is migratable, that value should be used.
Unfortunately, there is no easy way to know if a LPAR is migratable or not. The hypervisor is exporting the property 'ibm,migratable-partition' in the case it set to migrate partition, but that would not mean that the current partition is migratable.
Without this patch, when a LPAR is started on a 2 nodes box and then migrated to a 3 nodes box, the hypervisor may spread the LPAR's CPUs on the 3rd node. In that case if a CPU from that 3rd node is added to the LPAR, it will be wrongly assigned to the node because the kernel has been set to use up to 2 nodes (the configuration of the departure node). With this patch applies, the CPU is correctly added to the 3rd node.
Fixes f9f130ff2ec9 ("powerpc/numa: Detect support for coregroup")Reviewed-by Srikar Dronamraju <[email protected]>Signed-off-by Laurent Dufour <[email protected]>---
V2: Address Srikar's comments
- Fix the commit message
- Usepr_infoinsteadprintk(KERN_INFO..)--- arch/powerpc/mm/numa.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)arch/powerpc/mm/numa.c +10
-3@@ -893,7 +893,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)static void __init find_possible_nodes (void ) {struct device_node *rtas; -const __be32 *domains; +const __be32 *domains =NULL ;int prop_length, max_nodes;u32 i;@@ -909,9 +909,14 @@ static void __init find_possible_nodes(void)* it doesn't exist, then fallback on ibm,max-associativity-domains. * Current denotes what the platform can support compared to max * which denotes what the Hypervisor can support. +* +* If the LPAR is migratable, new nodes might be activated after a LPM, +* so we should consider the max number in that case. */ - domains =of_get_property (rtas,"ibm,current-associativity-domains" , - &prop_length); +if (!of_get_property (of_root,"ibm,migratable-partition" ,NULL )) + domains =of_get_property (rtas, +"ibm,current-associativity-domains" , + &prop_length);if (!domains) { domains =of_get_property (rtas,"ibm,max-associativity-domains" , &prop_length);@@ -920,6 +925,8 @@ static void __init find_possible_nodes(void) } max_nodes =of_read_number (&domains[min_common_depth],1 ); +pr_info ("Partition configured for %d NUMA nodes.\n" , max_nodes); +for (i =0 ; i < max_nodes; i++) {if (!node_possible (i))node_set (i, node_possible_map);
Michael Ellerman
AppliedQuoted 10 lines · Laurent Dufour
When a LPAR is migratable, we should consider the maximum possible NUMA node instead the number of NUMA node from the actual system.
The DT property 'ibm,current-associativity-domains' is defining the maximum number of nodes the LPAR can see when running on that box. But if the LPAR is being migrated on another box, it may seen up to the nodes defined by 'ibm,max-associativity-domains'. So if a LPAR is migratable, that value should be used.
[...]
Applied to powerpc/next.
[1/1] ppc64/numa: consider the max numa node for migratable LPAR
https://git.kernel.org/powerpc/c/9c7248bb8de31f51c693bfa6a6ea53b1c07e0fa8
cheers