sched/topology: Optimize topology_span_sane()
Subsystems (2)
cover sched/topology: Optimize topology_span_sane()
A soft lockup is being detected in build_sched_domains() on 32 socket Sapphire Rapids systems with 3840 processors.
topology_span_sane(), called by build_sched_domains(), checks that each processor's non-NUMA scheduling domains are completely equal or completely disjoint. If a non-NUMA scheduling domain partially overlaps another, scheduling groups can break.
This series adds for_each_cpu_from() as a generic cpumask macro to optimize topology_span_sane() by removing duplicate comparisons. The total number of comparisons is reduced from N * (N - 1) to
N * (N - 1) / 2 (per non-NUMA scheduling domain level), decreasing the boot time by approximately 20 seconds and preventing the soft lockup on the mentioned systems.
RESEND because Valentin Schneider reported that PATCH 2/2 wasn't delivered to all recipients.
Kyle Meyer (2):
cpumask: Add for_each_cpu_from()
sched/topology: Optimize topology_span_sane()
include/linux/cpumask.h | 10 ++++++++++ kernel/sched/topology.c | 6 ++---- 2 files changed, 12 insertions(+), 4 deletions(-)
Signature · 1 line
2.44.0
PATCH RESEND 1/2 cpumask: Add for_each_cpu_from()
Add for_each_cpu_from() as a generic cpumask macro.
for_each_cpu_from() is the same as for_each_cpu(), except it starts at @cpu instead of zero.
--- include/linux/cpumask.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
include/linux/cpumask.h +10 -0
@@ -368,6 +368,16 @@ unsigned int __pure cpumask_next_wrap(int n, const struct cpumask *mask, int sta#define for_each_cpu_or (cpu, mask1, mask2) \ for_each_or_bit(cpu, cpumask_bits(mask1), cpumask_bits(mask2), small_cpumask_bits) +/** +* for_each_cpu_from - iterate over every cpu present in @mask, starting at @cpu +* @cpu: the (optionally unsigned) integer iterator +* @mask: the cpumask pointer +* +* After the loop, cpu is >= nr_cpu_ids. +*/ +#define for_each_cpu_from (cpu, mask) \ + for_each_set_bit_from(cpu, cpumask_bits(mask), small_cpumask_bits) +/** * cpumask_any_but - return a "random" in a cpumask, but not this one. * @mask: the cpumask to search
Signature · 1 line
2.44.0
Vincent Guittot
maintainerRe: [PATCH 1/2 RESEND] cpumask: Add for_each_cpu_from()
Quoted 17 lines · Kyle Meyer
Add
for_each_cpu_from()as a generic cpumask macro.
for_each_cpu_from()is the same asfor_each_cpu(), except it starts at @cpu instead of zero.Signed-off-by Kyle Meyer <[email protected]>Acked-by Yury Norov <[email protected]>--- include/linux/cpumask.h | 10 ++++++++++ 1 file changed, 10 insertions(+)include/linux/cpumask.h +0
-0
include/linux/cpumask.h unsigned int __pure cpumask_next_wrap(int n, const struct cpumask *mask, int sta · line 372
Quoted 5 lines
include/linux/cpumask.h +2
-0@@ -368,6 +368,16 @@ unsigned int __pure cpumask_next_wrap(int n, const struct cpumask *mask, int sta#define for_each_cpu_or (cpu, mask1, mask2) \ for_each_or_bit(cpu, cpumask_bits(mask1), cpumask_bits(mask2), small_cpumask_bits) +/** +* for_each_cpu_from - iterate over every cpu present in @mask, starting at @cpu
So I was confused why you were not using for_each_cpu_wrap while reading the description which has the same comment :
"
* for_each_cpu_wrap - iterate over every cpu in a mask, starting at a
specified location
"
Could you clarify that it's not "every cpu present in @mask" but only those after @cpu ?
Quoted 14 lines
include/linux/cpumask.h +8
-1+* @cpu: the (optionally unsigned) integer iterator +* @mask: the cpumask pointer +* +* After the loop, cpu is >= nr_cpu_ids. +*/ +#define for_each_cpu_from (cpu, mask) \ + for_each_set_bit_from(cpu, cpumask_bits(mask), small_cpumask_bits) +/** * cpumask_any_but - return a "random" in a cpumask, but not this one. * @mask: the cpumask to search -- 2.44.0
PATCH RESEND 2/2 sched/topology: Optimize topology_span_sane()
In reply to Kyle MeyerOptimize topology_span_sane() by removing duplicate comparisons.
The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 (per non-NUMA scheduling domain level).
--- kernel/sched/topology.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
kernel/sched/topology.c +2 -4
@@ -2347,7 +2347,7 @@ static struct sched_domain *build_sched_domain(struct sched_domain_topology_levestatic bool topology_span_sane (struct sched_domain_topology_level *tl,const struct cpumask *cpu_map,int cpu) { -int i; +int i = cpu +1 ;/* NUMA levels are allowed to overlap */ if (tl->flags &SDTL_OVERLAP )
@@ -2359,9 +2359,7 @@ static bool topology_span_sane(struct sched_domain_topology_level *tl,* breaking the sched_group lists - i.e. a later get_group() pass * breaks the linking done for an earlier span. */ -for_each_cpu (i, cpu_map) { -if (i == cpu) -continue ; +for_each_cpu_from (i, cpu_map) {/* * We should 'and' all those masks with 'cpu_map' to exactly * match the topology we're about to build, but that can only
Signature · 1 line
2.44.0
Quoted 4 lines · Kyle Meyer
Optimize
topology_span_sane()by removing duplicate comparisons.The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 (per non-NUMA scheduling domain level).
...
kernel/sched/topology.c static bool topology_span_sane(struct sched_domain_topology_level *tl, · line 2362
Quoted 4 lines
kernel/sched/topology.c +1
-3-for_each_cpu (i, cpu_map) { -if (i == cpu) -continue ; +for_each_cpu_from (i, cpu_map) {
Hmm... I'm not familiar with the for_each_cpu*(), but from the above it seems only a single comparison? Or i.o.w. can i ever repeat the value? And what about i < cpu cases?
Signature · 2 lines
With Best Regards, Andy Shevchenko
Quoted 8 lines · Andy Shevchenko
Quoted 4 lines · Kyle Meyer
Optimize
topology_span_sane()by removing duplicate comparisons.The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 (per non-NUMA scheduling domain level).
...
Quoted 7 lines
kernel/sched/topology.c static bool topology_span_sane(struct sched_domain_topology_level *tl, · line 2362
Quoted 4 lines
kernel/sched/topology.c +1
-3-for_each_cpu (i, cpu_map) { -if (i == cpu) -continue ; +for_each_cpu_from (i, cpu_map) {Hmm... I'm not familiar with the for_each_cpu*(), but from the above it seems only a single comparison? Or i.o.w. can i ever repeat the value?
for_each_cpu() is a macro around for_each_set_bit() which iterates over each set bit in a bitmap starting at zero.
for_each_cpu_from() is a macro around for_each_set_bit_from() which iterates over each set bit in a bitmap starting at the specified bit.
The above (topology_span_sane()) currently does a "single comparison" for each CPU in cpu_map, but it's called for each CPU in cpu_map and for each scheduling domain level (please see build_sched_domains() in kernel/sched/topology.c).
Quoted 1 line
And what about i < cpu cases?
Those values have already been passed to topology_span_sane(). This patch uses for_each_cpu_from() starting at cpu + 1 to prevent those duplicate comparisons.
Thanks,
Kyle Meyer
Quoted 6 lines · Kyle Meyer
Quoted 5 lines · Andy Shevchenko
Quoted 4 lines · Kyle Meyer
Optimize
topology_span_sane()by removing duplicate comparisons.The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 (per non-NUMA scheduling domain level).
...
Quoted 14 lines
Quoted 7 lines
kernel/sched/topology.c static bool topology_span_sane(struct sched_domain_topology_level *tl, · line 2362
Quoted 4 lines
kernel/sched/topology.c +1
-3-for_each_cpu (i, cpu_map) { -if (i == cpu) -continue ; +for_each_cpu_from (i, cpu_map) {Hmm... I'm not familiar with the for_each_cpu*(), but from the above it seems only a single comparison? Or i.o.w. can i ever repeat the value?
for_each_cpu()is a macro aroundfor_each_set_bit()which iterates over each set bit in a bitmap starting at zero.
for_each_cpu_from()is a macro aroundfor_each_set_bit_from()which iterates over each set bit in a bitmap starting at the specified bit.
Quoted 8 lines
The above (
topology_span_sane()) currently does a "single comparison" for each CPU incpu_map, but it's called for each CPU incpu_mapand for each scheduling domain level (please seebuild_sched_domains()inkernel/sched/topology.c).Quoted 1 line
And what about i < cpu cases?
Those values have already been passed to
topology_span_sane(). This patch usesfor_each_cpu_from()starting at cpu + 1 to prevent those duplicate comparisons.
So, it appears to me that commit message has a room to improve / elaborate based on what you explained to me above.
Thanks!
Signature · 2 lines
With Best Regards, Andy Shevchenko
Quoted 8 lines · Kyle Meyer
Optimize
topology_span_sane()by removing duplicate comparisons.The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 (per non-NUMA scheduling domain level).
Signed-off-by Kyle Meyer <[email protected]>Reviewed-by Yury Norov <[email protected]>
Quoted 31 lines
--- kernel/sched/topology.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)kernel/sched/topology.c +2
-5@@ -2347,7 +2347,7 @@ static struct sched_domain *build_sched_domain(struct sched_domain_topology_levestatic bool topology_span_sane (struct sched_domain_topology_level *tl,const struct cpumask *cpu_map,int cpu) { -int i; +int i = cpu +1 ;/* NUMA levels are allowed to overlap */ if (tl->flags &SDTL_OVERLAP )@@ -2359,9 +2359,7 @@ static bool topology_span_sane(struct sched_domain_topology_level *tl,* breaking the sched_group lists - i.e. a later get_group() pass * breaks the linking done for an earlier span. */ -for_each_cpu (i, cpu_map) { -if (i == cpu) -continue ; +for_each_cpu_from (i, cpu_map) {/* * We should 'and' all those masks with 'cpu_map' to exactly * match the topology we're about to build, but that can only -- 2.44.0
Yury Norov
maintainerQuoted 15 lines · Kyle Meyer
A soft lockup is being detected in
build_sched_domains()on 32 socket Sapphire Rapids systems with 3840 processors.
topology_span_sane(), called bybuild_sched_domains(), checks that each processor's non-NUMA scheduling domains are completely equal or completely disjoint. If a non-NUMA scheduling domain partially overlaps another, scheduling groups can break.This series adds
for_each_cpu_from()as a generic cpumask macro to optimizetopology_span_sane()by removing duplicate comparisons. The total number of comparisons is reduced from N * (N - 1) to
N * (N - 1) / 2 (per non-NUMA scheduling domain level), decreasing the boot time by approximately 20 seconds and preventing the soft lockup on the mentioned systems.
Quoted 2 lines
RESEND because Valentin Schneider reported that PATCH 2/2 wasn't delivered to all recipients.
Hmm... And this one doesn't have #1, at least for me. Can you please resend again, so we all will have the series as a whole?
Also, can you rephrase this from patch #1 because it confuses people:
Quoted 1 line
+ *
for_each_cpu_from- iterate over every cpu present in @mask, starting at @cpu
Maybe: iterate over CPUs present in a @mask greater than a @cpu? Or similar.
Thanks,
Yury
Quoted 10 lines
Kyle Meyer (2):
cpumask: Addfor_each_cpu_from()
sched/topology: Optimizetopology_span_sane()include/linux/cpumask.h | 10 ++++++++++ kernel/sched/topology.c | 6 ++---- 2 files changed, 12 insertions(+), 4 deletions(-)Signature · 1 line
2.44.0