Skip to contentSkip to conversation

sched/topology: Optimize topology_span_sane()

PATCH RESEND Superseded Kyle Meyer 4 participants 9 messages Apr 9, 2024 – Apr 10, 2024

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Kyle Meyer

author
#0 lore raw

cover sched/topology: Optimize topology_span_sane()

A soft lockup is being detected in build_sched_domains() on 32 socket Sapphire Rapids systems with 3840 processors.

topology_span_sane(), called by build_sched_domains(), checks that each processor's non-NUMA scheduling domains are completely equal or completely disjoint. If a non-NUMA scheduling domain partially overlaps another, scheduling groups can break.

This series adds for_each_cpu_from() as a generic cpumask macro to optimize topology_span_sane() by removing duplicate comparisons. The total number of comparisons is reduced from N * (N - 1) to
N * (N - 1) / 2 (per non-NUMA scheduling domain level), decreasing the boot time by approximately 20 seconds and preventing the soft lockup on the mentioned systems.

RESEND because Valentin Schneider reported that PATCH 2/2 wasn't delivered to all recipients.

Kyle Meyer (2):
cpumask: Add for_each_cpu_from()
sched/topology: Optimize topology_span_sane()

 include/linux/cpumask.h | 10 ++++++++++
 kernel/sched/topology.c |  6 ++----
 2 files changed, 12 insertions(+), 4 deletions(-)
Signature · 1 line
2.44.0
author
#1 lore raw

PATCH RESEND 1/2 cpumask: Add for_each_cpu_from()

Add for_each_cpu_from() as a generic cpumask macro.

for_each_cpu_from() is the same as for_each_cpu(), except it starts at @cpu instead of zero.

---
 include/linux/cpumask.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
include/linux/cpumask.h +10 -0
index 1c29947db848..655211db38ff 100644
@@ -368,6 +368,16 @@ unsigned int __pure cpumask_next_wrap(int n, const struct cpumask *mask, int sta
 #define for_each_cpu_or(cpu, mask1, mask2)				\
 	for_each_or_bit(cpu, cpumask_bits(mask1), cpumask_bits(mask2), small_cpumask_bits)
 
+/**
+ * for_each_cpu_from - iterate over every cpu present in @mask, starting at @cpu
+ * @cpu: the (optionally unsigned) integer iterator
+ * @mask: the cpumask pointer
+ *
+ * After the loop, cpu is >= nr_cpu_ids.
+ */
+#define for_each_cpu_from(cpu, mask)				\
+	for_each_set_bit_from(cpu, cpumask_bits(mask), small_cpumask_bits)
+
 /**
  * cpumask_any_but - return a "random" in a cpumask, but not this one.
  * @mask: the cpumask to search
Signature · 1 line
2.44.0
maintainer
#2 lore raw

Re: [PATCH 1/2 RESEND] cpumask: Add for_each_cpu_from()

Quoted 17 lines · Kyle Meyer

Add for_each_cpu_from() as a generic cpumask macro.

for_each_cpu_from() is the same as for_each_cpu(), except it starts at @cpu instead of zero.

---
 include/linux/cpumask.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
include/linux/cpumask.h +0 -0
index 1c29947db848..655211db38ff 100644

include/linux/cpumask.h unsigned int __pure cpumask_next_wrap(int n, const struct cpumask *mask, int sta · line 372

Quoted 5 lines
include/linux/cpumask.h +2 -0
@@ -368,6 +368,16 @@ unsigned int __pure cpumask_next_wrap(int n, const struct cpumask *mask, int sta
 #define for_each_cpu_or(cpu, mask1, mask2)                             \
        for_each_or_bit(cpu, cpumask_bits(mask1), cpumask_bits(mask2), small_cpumask_bits)
 
+/**
+ * for_each_cpu_from - iterate over every cpu present in @mask, starting at @cpu

So I was confused why you were not using for_each_cpu_wrap while reading the description which has the same comment :
"
* for_each_cpu_wrap - iterate over every cpu in a mask, starting at a
specified location
"
Could you clarify that it's not "every cpu present in @mask" but only those after @cpu ?

Quoted 14 lines
include/linux/cpumask.h +8 -1
+ * @cpu: the (optionally unsigned) integer iterator
+ * @mask: the cpumask pointer
+ *
+ * After the loop, cpu is >= nr_cpu_ids.
+ */
+#define for_each_cpu_from(cpu, mask)                           \
+       for_each_set_bit_from(cpu, cpumask_bits(mask), small_cpumask_bits)
+
 /**
  * cpumask_any_but - return a "random" in a cpumask, but not this one.
  * @mask: the cpumask to search
--

2.44.0

author
#3 lore raw

PATCH RESEND 2/2 sched/topology: Optimize topology_span_sane()

In reply to Kyle Meyer

Optimize topology_span_sane() by removing duplicate comparisons.

The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 (per non-NUMA scheduling domain level).

---
 kernel/sched/topology.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
kernel/sched/topology.c +2 -4
index 99ea5986038c..b6bcafc09969 100644
@@ -2347,7 +2347,7 @@ static struct sched_domain *build_sched_domain(struct sched_domain_topology_leve
 static bool topology_span_sane(struct sched_domain_topology_level *tl,
 			      const struct cpumask *cpu_map, int cpu)
 {
-	int i;
+	int i = cpu + 1;
 
 	/* NUMA levels are allowed to overlap */
 	if (tl->flags & SDTL_OVERLAP)
@@ -2359,9 +2359,7 @@ static bool topology_span_sane(struct sched_domain_topology_level *tl,
 	 * breaking the sched_group lists - i.e. a later get_group() pass
 	 * breaks the linking done for an earlier span.
 	 */
-	for_each_cpu(i, cpu_map) {
-		if (i == cpu)
-			continue;
+	for_each_cpu_from(i, cpu_map) {
 		/*
 		 * We should 'and' all those masks with 'cpu_map' to exactly
 		 * match the topology we're about to build, but that can only
Signature · 1 line
2.44.0
#4 lore raw
Quoted 4 lines · Kyle Meyer

Optimize topology_span_sane() by removing duplicate comparisons.

The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 (per non-NUMA scheduling domain level).

...

kernel/sched/topology.c static bool topology_span_sane(struct sched_domain_topology_level *tl, · line 2362

Quoted 4 lines
kernel/sched/topology.c +1 -3
-	for_each_cpu(i, cpu_map) {
-		if (i == cpu)
-			continue;
+	for_each_cpu_from(i, cpu_map) {

Hmm... I'm not familiar with the for_each_cpu*(), but from the above it seems only a single comparison? Or i.o.w. can i ever repeat the value? And what about i < cpu cases?

Signature · 2 lines
With Best Regards,
Andy Shevchenko
author
#5 lore raw
Quoted 8 lines · Andy Shevchenko
Quoted 4 lines · Kyle Meyer

Optimize topology_span_sane() by removing duplicate comparisons.

The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 (per non-NUMA scheduling domain level).

...

Quoted 7 lines

kernel/sched/topology.c static bool topology_span_sane(struct sched_domain_topology_level *tl, · line 2362

Quoted 4 lines
kernel/sched/topology.c +1 -3
-	for_each_cpu(i, cpu_map) {
-		if (i == cpu)
-			continue;
+	for_each_cpu_from(i, cpu_map) {

Hmm... I'm not familiar with the for_each_cpu*(), but from the above it seems only a single comparison? Or i.o.w. can i ever repeat the value?

for_each_cpu() is a macro around for_each_set_bit() which iterates over each set bit in a bitmap starting at zero.

for_each_cpu_from() is a macro around for_each_set_bit_from() which iterates over each set bit in a bitmap starting at the specified bit.

The above (topology_span_sane()) currently does a "single comparison" for each CPU in cpu_map, but it's called for each CPU in cpu_map and for each scheduling domain level (please see build_sched_domains() in kernel/sched/topology.c).

Quoted 1 line

And what about i < cpu cases?

Those values have already been passed to topology_span_sane(). This patch uses for_each_cpu_from() starting at cpu + 1 to prevent those duplicate comparisons.

Thanks,
Kyle Meyer

#6 lore raw
Quoted 6 lines · Kyle Meyer
Quoted 5 lines · Andy Shevchenko
Quoted 4 lines · Kyle Meyer

Optimize topology_span_sane() by removing duplicate comparisons.

The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 (per non-NUMA scheduling domain level).

...

Quoted 14 lines
Quoted 7 lines

kernel/sched/topology.c static bool topology_span_sane(struct sched_domain_topology_level *tl, · line 2362

Quoted 4 lines
kernel/sched/topology.c +1 -3
-	for_each_cpu(i, cpu_map) {
-		if (i == cpu)
-			continue;
+	for_each_cpu_from(i, cpu_map) {

Hmm... I'm not familiar with the for_each_cpu*(), but from the above it seems only a single comparison? Or i.o.w. can i ever repeat the value?

for_each_cpu() is a macro around for_each_set_bit() which iterates over each set bit in a bitmap starting at zero.

for_each_cpu_from() is a macro around for_each_set_bit_from() which iterates over each set bit in a bitmap starting at the specified bit.

Quoted 8 lines

The above (topology_span_sane()) currently does a "single comparison" for each CPU in cpu_map, but it's called for each CPU in cpu_map and for each scheduling domain level (please see build_sched_domains() in kernel/sched/topology.c).

Quoted 1 line

And what about i < cpu cases?

Those values have already been passed to topology_span_sane(). This patch uses for_each_cpu_from() starting at cpu + 1 to prevent those duplicate comparisons.

So, it appears to me that commit message has a room to improve / elaborate based on what you explained to me above.

Thanks!

Signature · 2 lines
With Best Regards,
Andy Shevchenko
maintainer Acked-by on v1 2/2
#7 lore raw
In reply to Kyle Meyer
Quoted 8 lines · Kyle Meyer

Optimize topology_span_sane() by removing duplicate comparisons.

The total number of comparisons is reduced from N * (N - 1) to N * (N - 1) / 2 (per non-NUMA scheduling domain level).

Quoted 31 lines
---
 kernel/sched/topology.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
kernel/sched/topology.c +2 -5
index 99ea5986038c..b6bcafc09969 100644
@@ -2347,7 +2347,7 @@ static struct sched_domain *build_sched_domain(struct sched_domain_topology_leve
 static bool topology_span_sane(struct sched_domain_topology_level *tl,
                              const struct cpumask *cpu_map, int cpu)
 {
-       int i;
+       int i = cpu + 1;
 
        /* NUMA levels are allowed to overlap */
        if (tl->flags & SDTL_OVERLAP)
@@ -2359,9 +2359,7 @@ static bool topology_span_sane(struct sched_domain_topology_level *tl,
         * breaking the sched_group lists - i.e. a later get_group() pass
         * breaks the linking done for an earlier span.
         */
-       for_each_cpu(i, cpu_map) {
-               if (i == cpu)
-                       continue;
+       for_each_cpu_from(i, cpu_map) {
                /*
                 * We should 'and' all those masks with 'cpu_map' to exactly
                 * match the topology we're about to build, but that can only
--

2.44.0

maintainer
#8 lore raw
In reply to Kyle Meyer
Quoted 15 lines · Kyle Meyer

A soft lockup is being detected in build_sched_domains() on 32 socket Sapphire Rapids systems with 3840 processors.

topology_span_sane(), called by build_sched_domains(), checks that each processor's non-NUMA scheduling domains are completely equal or completely disjoint. If a non-NUMA scheduling domain partially overlaps another, scheduling groups can break.

This series adds for_each_cpu_from() as a generic cpumask macro to optimize topology_span_sane() by removing duplicate comparisons. The total number of comparisons is reduced from N * (N - 1) to
N * (N - 1) / 2 (per non-NUMA scheduling domain level), decreasing the boot time by approximately 20 seconds and preventing the soft lockup on the mentioned systems.

Quoted 2 lines

RESEND because Valentin Schneider reported that PATCH 2/2 wasn't delivered to all recipients.

Hmm... And this one doesn't have #1, at least for me. Can you please resend again, so we all will have the series as a whole?

Also, can you rephrase this from patch #1 because it confuses people:

Quoted 1 line

+ * for_each_cpu_from - iterate over every cpu present in @mask, starting at @cpu

Maybe: iterate over CPUs present in a @mask greater than a @cpu? Or similar.

Thanks,
Yury

Quoted 10 lines

Kyle Meyer (2):
cpumask: Add for_each_cpu_from()
sched/topology: Optimize topology_span_sane()

 include/linux/cpumask.h | 10 ++++++++++
 kernel/sched/topology.c |  6 ++----
 2 files changed, 12 insertions(+), 4 deletions(-)
Signature · 1 line
2.44.0