Skip to contentSkip to conversation

sched/topology: Optimize topology_span_sane()

PATCH In review Kyle Meyer 4 participants 9 messages Apr 10, 2024 – Apr 15, 2024

Subsystems (2)
lore
Share & tools
lore

Kyle Meyer

author
#0 lore raw

v2 cover sched/topology: Optimize topology_span_sane()

A soft lockup is being detected in build_sched_domains() on 32 socket Sapphire Rapids systems with 3840 processors.

topology_span_sane(), called by build_sched_domains(), checks that each processor's non-NUMA scheduling domains are completely equal or completely disjoint. If a non-NUMA scheduling domain partially overlaps another, scheduling groups can break.

This series adds for_each_cpu_from() as a generic cpumask macro to optimize topology_span_sane() by removing duplicate comparisons. The total number of comparisons is reduced from N * (N - 1) to
N * (N - 1) / 2 on each non-NUMA scheduling domain level, decreasing the boot time by approximately 20 seconds and preventing the soft lockup on the mentioned systems.

Changes in v2:
* 1/2: Change for_each_cpu()'s description.
* 2/2: Add more information to the commit message.
* https://lore.kernel.org/linux-kernel/[email protected]/T/

Kyle Meyer (2):
cpumask: Add for_each_cpu_from()
sched/topology: Optimize topology_span_sane()

 include/linux/cpumask.h | 10 ++++++++++
 kernel/sched/topology.c |  6 ++----
 2 files changed, 12 insertions(+), 4 deletions(-)
Signature · 1 line
2.44.0
author
#1 lore raw

PATCH v2 1/2 cpumask: Add for_each_cpu_from()

Add for_each_cpu_from() as a generic cpumask macro.

for_each_cpu_from() is the same as for_each_cpu(), except it starts at @cpu instead of zero.

---
 include/linux/cpumask.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
include/linux/cpumask.h +10 -0
index 1c29947db848..d75060fbd058 100644
@@ -368,6 +368,16 @@ unsigned int __pure cpumask_next_wrap(int n, const struct cpumask *mask, int sta
 #define for_each_cpu_or(cpu, mask1, mask2)				\
 	for_each_or_bit(cpu, cpumask_bits(mask1), cpumask_bits(mask2), small_cpumask_bits)
 
+/**
+ * for_each_cpu_from - iterate over CPUs present in @mask, from @cpu to the end of @mask.
+ * @cpu: the (optionally unsigned) integer iterator
+ * @mask: the cpumask pointer
+ *
+ * After the loop, cpu is >= nr_cpu_ids.
+ */
+#define for_each_cpu_from(cpu, mask)				\
+	for_each_set_bit_from(cpu, cpumask_bits(mask), small_cpumask_bits)
+
 /**
  * cpumask_any_but - return a "random" in a cpumask, but not this one.
  * @mask: the cpumask to search
Signature · 1 line
2.44.0
author
#3 lore raw

PATCH v2 2/2 sched/topology: Optimize topology_span_sane()

In reply to Kyle Meyer

Optimize topology_span_sane() by removing duplicate comparisons.

Since topology_span_sane() is called inside of for_each_cpu(), each pervious CPU has already been compared against every other CPU. The current CPU only needs to be compared against higher-numbered CPUs.

The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 on each non-NUMA scheduling domain level.

---
 kernel/sched/topology.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
kernel/sched/topology.c +2 -4
index 99ea5986038c..b6bcafc09969 100644
@@ -2347,7 +2347,7 @@ static struct sched_domain *build_sched_domain(struct sched_domain_topology_leve
 static bool topology_span_sane(struct sched_domain_topology_level *tl,
 			      const struct cpumask *cpu_map, int cpu)
 {
-	int i;
+	int i = cpu + 1;
 
 	/* NUMA levels are allowed to overlap */
 	if (tl->flags & SDTL_OVERLAP)
@@ -2359,9 +2359,7 @@ static bool topology_span_sane(struct sched_domain_topology_level *tl,
 	 * breaking the sched_group lists - i.e. a later get_group() pass
 	 * breaks the linking done for an earlier span.
 	 */
-	for_each_cpu(i, cpu_map) {
-		if (i == cpu)
-			continue;
+	for_each_cpu_from(i, cpu_map) {
 		/*
 		 * We should 'and' all those masks with 'cpu_map' to exactly
 		 * match the topology we're about to build, but that can only
Signature · 1 line
2.44.0
#4 lore raw
Quoted 4 lines · Kyle Meyer

Optimize topology_span_sane() by removing duplicate comparisons.

Since topology_span_sane() is called inside of for_each_cpu(), each pervious CPU has already been compared against every other CPU. The

previous

Quoted 4 lines

current CPU only needs to be compared against higher-numbered CPUs.

The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 on each non-NUMA scheduling domain level.

Thank you, now it makes sense.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
author
#5 lore raw
Quoted 7 lines · Andy Shevchenko
Quoted 4 lines · Kyle Meyer

Optimize topology_span_sane() by removing duplicate comparisons.

Since topology_span_sane() is called inside of for_each_cpu(), each pervious CPU has already been compared against every other CPU. The

previous

Thank you for pointing that out. Should I send an updated version or can a maintainer correct my mistake?

Quoted 6 lines
Quoted 4 lines

current CPU only needs to be compared against higher-numbered CPUs.

The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 on each non-NUMA scheduling domain level.

Thank you, now it makes sense.

Thanks,
Kyle Meyer

#6 lore raw
Quoted 2 lines · Kyle Meyer
Quoted 1 line · Andy Shevchenko

On Wed, Apr 10, 2024 at 04:33:11PM -0500, Kyle Meyer wrote:

...

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Since topology_span_sane() is called inside of for_each_cpu(), each pervious CPU has already been compared against every other CPU. The

previous

Thank you for pointing that out. Should I send an updated version or can a maintainer correct my mistake?

Depends on the maintainer. I'm not the one here, don't expect answer from me.

Quoted 1 line
Quoted 1 line
Quoted 1 line

current CPU only needs to be compared against higher-numbered CPUs.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
maintainer
#7 lore raw
Quoted 9 lines · Andy Shevchenko
Quoted 2 lines · Kyle Meyer
Quoted 1 line · Andy Shevchenko

On Wed, Apr 10, 2024 at 04:33:11PM -0500, Kyle Meyer wrote:

...

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

Since topology_span_sane() is called inside of for_each_cpu(), each pervious CPU has already been compared against every other CPU. The

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

previous

Thank you for pointing that out. Should I send an updated version or can a maintainer correct my mistake?

Depends on the maintainer. I'm not the one here, don't expect answer from me.

I like this rework, and I'll take it in bitmap-for-next for testing.

I see Vincent already acked it, so I can also move it upstream, if no objections from Peter, Juri and Ingo.

Thanks,
Yury

reviewer Reviewed-by on v2 2/2
#8 lore raw
In reply to Kyle Meyer
Quoted 6 lines · Kyle Meyer

Optimize topology_span_sane() by removing duplicate comparisons.

Since topology_span_sane() is called inside of for_each_cpu(), each pervious CPU has already been compared against every other CPU. The current CPU only needs to be compared against higher-numbered CPUs.

Quoted 6 lines

The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 on each non-NUMA scheduling domain level.