Skip to contentSkip to conversation

sched/fair: reduce false sharing on sched_balance_running

PATCH In review Eric Dumazet 4 participants 9 messages Apr 23, 2025 – Apr 26, 2025 Scheduler

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched/fair: reduce false sharing on sched_balance_running

rebalance_domains() can attempt to change sched_balance_running more than 350,000 times per second on our servers.

If sched_clock_irqtime and sched_balance_running share the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.

This patch only acquires sched_balance_running when sd->last_balance is old enough.

It also moves sched_balance_running into a dedicated cache line on SMP.

Cc Yafang Shao <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Josh Don <[email protected]>
---
 kernel/sched/fair.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)
kernel/sched/fair.c +14 -14
index e43993a4e5807eaffcacaf761c289e8adb354cfd..460008d0dc459b3ca60209565e89c419ea32a4e2 100644
@@ -12144,7 +12144,7 @@ static int active_load_balance_cpu_stop(void *data)
  *   execution, as non-SD_SERIALIZE domains will still be
  *   load-balanced in parallel.
  */
-static atomic_t sched_balance_running = ATOMIC_INIT(0);
+static __cacheline_aligned_in_smp atomic_t sched_balance_running = ATOMIC_INIT(0);
 
 /*
  * Scale the max sched_balance_rq interval with the number of CPUs in the system.
@@ -12220,25 +12220,25 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle)
 
 		interval = get_sd_balance_interval(sd, busy);
 
+		if (!time_after_eq(jiffies, sd->last_balance + interval))
+			goto out;
+
 		need_serialize = sd->flags & SD_SERIALIZE;
 		if (need_serialize) {
 			if (atomic_cmpxchg_acquire(&sched_balance_running, 0, 1))
 				goto out;
 		}
-
-		if (time_after_eq(jiffies, sd->last_balance + interval)) {
-			if (sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {
-				/*
-				 * The LBF_DST_PINNED logic could have changed
-				 * env->dst_cpu, so we can't know our idle
-				 * state even if we migrated tasks. Update it.
-				 */
-				idle = idle_cpu(cpu);
-				busy = !idle && !sched_idle_cpu(cpu);
-			}
-			sd->last_balance = jiffies;
-			interval = get_sd_balance_interval(sd, busy);
+		if (sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {
+			/*
+			 * The LBF_DST_PINNED logic could have changed
+			 * env->dst_cpu, so we can't know our idle
+			 * state even if we migrated tasks. Update it.
+			 */
+			idle = idle_cpu(cpu);
+			busy = !idle && !sched_idle_cpu(cpu);
 		}
+		sd->last_balance = jiffies;
+		interval = get_sd_balance_interval(sd, busy);
 		if (need_serialize)
 			atomic_set_release(&sched_balance_running, 0);
 out:
Signature · 1 line
2.49.0.805.g082f7c87e0-goog
#1 lore raw
Quoted 7 lines · Eric Dumazet

rebalance_domains() can attempt to change sched_balance_running more than 350,000 times per second on our servers.

If sched_clock_irqtime and sched_balance_running share the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.

CONFIG_IRQ_TIME_ACCOUNTING is enabled on your systems, right? Have you observed any impact on task CPU utilization measurements due to this configuration? [0]

If cache misses on sched_clock_irqtime are indeed the bottleneck, why not align it to improve performance?

[0]. https://lore.kernel.org/all/[email protected]/

Quoted 69 lines

This patch only acquires sched_balance_running when sd->last_balance is old enough.

It also moves sched_balance_running into a dedicated cache line on SMP.

Cc Yafang Shao <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Josh Don <[email protected]>
---
 kernel/sched/fair.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)
kernel/sched/fair.c +14 -15
index e43993a4e5807eaffcacaf761c289e8adb354cfd..460008d0dc459b3ca60209565e89c419ea32a4e2 100644
@@ -12144,7 +12144,7 @@ static int active_load_balance_cpu_stop(void *data)
  *   execution, as non-SD_SERIALIZE domains will still be
  *   load-balanced in parallel.
  */
-static atomic_t sched_balance_running = ATOMIC_INIT(0);
+static __cacheline_aligned_in_smp atomic_t sched_balance_running = ATOMIC_INIT(0);
 
 /*
  * Scale the max sched_balance_rq interval with the number of CPUs in the system.
@@ -12220,25 +12220,25 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle)
 
                interval = get_sd_balance_interval(sd, busy);
 
+               if (!time_after_eq(jiffies, sd->last_balance + interval))
+                       goto out;
+
                need_serialize = sd->flags & SD_SERIALIZE;
                if (need_serialize) {
                        if (atomic_cmpxchg_acquire(&sched_balance_running, 0, 1))
                                goto out;
                }
-
-               if (time_after_eq(jiffies, sd->last_balance + interval)) {
-                       if (sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {
-                               /*
-                                * The LBF_DST_PINNED logic could have changed
-                                * env->dst_cpu, so we can't know our idle
-                                * state even if we migrated tasks. Update it.
-                                */
-                               idle = idle_cpu(cpu);
-                               busy = !idle && !sched_idle_cpu(cpu);
-                       }
-                       sd->last_balance = jiffies;
-                       interval = get_sd_balance_interval(sd, busy);
+               if (sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {
+                       /*
+                        * The LBF_DST_PINNED logic could have changed
+                        * env->dst_cpu, so we can't know our idle
+                        * state even if we migrated tasks. Update it.
+                        */
+                       idle = idle_cpu(cpu);
+                       busy = !idle && !sched_idle_cpu(cpu);
                }
+               sd->last_balance = jiffies;
+               interval = get_sd_balance_interval(sd, busy);
                if (need_serialize)
                        atomic_set_release(&sched_balance_running, 0);
 out:
--

2.49.0.805.g082f7c87e0-goog

Signature · 2 lines
Regards
Yafang
#2 lore raw
Quoted 10 lines · Yafang Shao
Quoted 7 lines · Eric Dumazet

rebalance_domains() can attempt to change sched_balance_running more than 350,000 times per second on our servers.

If sched_clock_irqtime and sched_balance_running share the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.

Quoted 6 lines

CONFIG_IRQ_TIME_ACCOUNTING is enabled on your systems, right? Have you observed any impact on task CPU utilization measurements due to this configuration? [0]

If cache misses on sched_clock_irqtime are indeed the bottleneck, why not align it to improve performance?

"Align it" meaning what exactly ? Once sched_clock_irqtime is in a read-mostly location everything is fine.

The main bottleneck is the false sharing on these Intel 6980P cpus...

On a dual socket system, this false sharing is using something like 4% of the total memory bandwidth,
and causes apparent high costs on other parts of the kernel.

Quoted 2 lines

[0]. https://lore.kernel.org/all/[email protected]/

What part should I look at, and how is this related to my patch ?

Quoted 75 lines
Quoted 68 lines

This patch only acquires sched_balance_running when sd->last_balance is old enough.

It also moves sched_balance_running into a dedicated cache line on SMP.

Cc Yafang Shao <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Josh Don <[email protected]>
---
 kernel/sched/fair.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)
kernel/sched/fair.c +14 -15
index e43993a4e5807eaffcacaf761c289e8adb354cfd..460008d0dc459b3ca60209565e89c419ea32a4e2 100644
@@ -12144,7 +12144,7 @@ static int active_load_balance_cpu_stop(void *data)
  *   execution, as non-SD_SERIALIZE domains will still be
  *   load-balanced in parallel.
  */
-static atomic_t sched_balance_running = ATOMIC_INIT(0);
+static __cacheline_aligned_in_smp atomic_t sched_balance_running = ATOMIC_INIT(0);
 
 /*
  * Scale the max sched_balance_rq interval with the number of CPUs in the system.
@@ -12220,25 +12220,25 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle)
 
                interval = get_sd_balance_interval(sd, busy);
 
+               if (!time_after_eq(jiffies, sd->last_balance + interval))
+                       goto out;
+
                need_serialize = sd->flags & SD_SERIALIZE;
                if (need_serialize) {
                        if (atomic_cmpxchg_acquire(&sched_balance_running, 0, 1))
                                goto out;
                }
-
-               if (time_after_eq(jiffies, sd->last_balance + interval)) {
-                       if (sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {
-                               /*
-                                * The LBF_DST_PINNED logic could have changed
-                                * env->dst_cpu, so we can't know our idle
-                                * state even if we migrated tasks. Update it.
-                                */
-                               idle = idle_cpu(cpu);
-                               busy = !idle && !sched_idle_cpu(cpu);
-                       }
-                       sd->last_balance = jiffies;
-                       interval = get_sd_balance_interval(sd, busy);
+               if (sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {
+                       /*
+                        * The LBF_DST_PINNED logic could have changed
+                        * env->dst_cpu, so we can't know our idle
+                        * state even if we migrated tasks. Update it.
+                        */
+                       idle = idle_cpu(cpu);
+                       busy = !idle && !sched_idle_cpu(cpu);
                }
+               sd->last_balance = jiffies;
+               interval = get_sd_balance_interval(sd, busy);
                if (need_serialize)
                        atomic_set_release(&sched_balance_running, 0);
 out:
--
Quoted 2 lines

2.49.0.805.g082f7c87e0-goog

--
Regards
Yafang

#3 lore raw
Quoted 12 lines · Eric Dumazet
Quoted 10 lines · Yafang Shao
Quoted 7 lines · Eric Dumazet

rebalance_domains() can attempt to change sched_balance_running more than 350,000 times per second on our servers.

If sched_clock_irqtime and sched_balance_running share the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.

Quoted 8 lines
Quoted 6 lines

CONFIG_IRQ_TIME_ACCOUNTING is enabled on your systems, right? Have you observed any impact on task CPU utilization measurements due to this configuration? [0]

If cache misses on sched_clock_irqtime are indeed the bottleneck, why not align it to improve performance?

"Align it" meaning what exactly ?

Such as :

static __cacheline_aligned_in_smp int sched_clock_irqtime;

Quoted 5 lines

Once sched_clock_irqtime is in a
read-mostly location everything is fine.

The main bottleneck is the false sharing on these Intel 6980P cpus...

Quoted 8 lines

On a dual socket system, this false sharing is using something like 4% of the total memory bandwidth,
and causes apparent high costs on other parts of the kernel.

Quoted 2 lines

[0]. https://lore.kernel.org/all/[email protected]/

What part should I look at, and how is this related to my patch ?

Unrelated to your patch. Please ignore it if you haven't seen this issue.

Signature · 2 lines
Regards
Yafang
#4 lore raw
Quoted 18 lines · Yafang Shao
Quoted 16 lines · Eric Dumazet
Quoted 14 lines · Yafang Shao
Quoted 7 lines · Eric Dumazet

rebalance_domains() can attempt to change sched_balance_running more than 350,000 times per second on our servers.

If sched_clock_irqtime and sched_balance_running share the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.

CONFIG_IRQ_TIME_ACCOUNTING is enabled on your systems, right? Have you observed any impact on task CPU utilization measurements due to this configuration? [0]

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

If cache misses on sched_clock_irqtime are indeed the bottleneck, why not align it to improve performance?

"Align it" meaning what exactly ?

Such as :

static __cacheline_aligned_in_smp int sched_clock_irqtime;

Please look at __read_mostly attribute instead.

Thanks.

Quoted 20 lines
Quoted 6 lines

Once sched_clock_irqtime is in a
read-mostly location everything is fine.

The main bottleneck is the false sharing on these Intel 6980P cpus...

Quoted 8 lines

On a dual socket system, this false sharing is using something like 4% of the total memory bandwidth,
and causes apparent high costs on other parts of the kernel.

Quoted 2 lines

[0]. https://lore.kernel.org/all/[email protected]/

What part should I look at, and how is this related to my patch ?

Unrelated to your patch. Please ignore it if you haven't seen this issue.

--
Regards
Yafang

#5 lore raw
Quoted 23 lines · Eric Dumazet
Quoted 21 lines · Yafang Shao
Quoted 19 lines · Eric Dumazet
Quoted 16 lines · Yafang Shao
Quoted 7 lines · Eric Dumazet

rebalance_domains() can attempt to change sched_balance_running more than 350,000 times per second on our servers.

If sched_clock_irqtime and sched_balance_running share the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.

CONFIG_IRQ_TIME_ACCOUNTING is enabled on your systems, right? Have you observed any impact on task CPU utilization measurements due to this configuration? [0]

If cache misses on sched_clock_irqtime are indeed the bottleneck, why not align it to improve performance?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

"Align it" meaning what exactly ?

Such as :

static __cacheline_aligned_in_smp int sched_clock_irqtime;

Please look at __read_mostly attribute instead.

Right, sched_clock_irqtime is only modified when the TSC becomes unstable, so it makes sense to annotate it as __read_mostly. So why not do it?

Signature · 2 lines
Regards
Yafang
Reviewed-by on v1 1/1
#6 lore raw
In reply to Eric Dumazet
Quoted 4 lines · Eric Dumazet

rebalance_domains() can attempt to change sched_balance_running more than 350,000 times per second on our servers.

Quoted 8 lines

If sched_clock_irqtime and sched_balance_running share the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.

This patch only acquires sched_balance_running when sd->last_balance is old enough.

It also moves sched_balance_running into a dedicated cache line on SMP.

Thanks Eric, looks good to me.

Quoted 64 lines
Cc Yafang Shao <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Josh Don <[email protected]>
---
 kernel/sched/fair.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)
kernel/sched/fair.c +14 -15
index e43993a4e5807eaffcacaf761c289e8adb354cfd..460008d0dc459b3ca60209565e89c419ea32a4e2 100644
@@ -12144,7 +12144,7 @@ static int active_load_balance_cpu_stop(void *data)
  *   execution, as non-SD_SERIALIZE domains will still be
  *   load-balanced in parallel.
  */
-static atomic_t sched_balance_running = ATOMIC_INIT(0);
+static __cacheline_aligned_in_smp atomic_t sched_balance_running = ATOMIC_INIT(0);
 
 /*
  * Scale the max sched_balance_rq interval with the number of CPUs in the system.
@@ -12220,25 +12220,25 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle)
 
                interval = get_sd_balance_interval(sd, busy);
 
+               if (!time_after_eq(jiffies, sd->last_balance + interval))
+                       goto out;
+
                need_serialize = sd->flags & SD_SERIALIZE;
                if (need_serialize) {
                        if (atomic_cmpxchg_acquire(&sched_balance_running, 0, 1))
                                goto out;
                }
-
-               if (time_after_eq(jiffies, sd->last_balance + interval)) {
-                       if (sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {
-                               /*
-                                * The LBF_DST_PINNED logic could have changed
-                                * env->dst_cpu, so we can't know our idle
-                                * state even if we migrated tasks. Update it.
-                                */
-                               idle = idle_cpu(cpu);
-                               busy = !idle && !sched_idle_cpu(cpu);
-                       }
-                       sd->last_balance = jiffies;
-                       interval = get_sd_balance_interval(sd, busy);
+               if (sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {
+                       /*
+                        * The LBF_DST_PINNED logic could have changed
+                        * env->dst_cpu, so we can't know our idle
+                        * state even if we migrated tasks. Update it.
+                        */
+                       idle = idle_cpu(cpu);
+                       busy = !idle && !sched_idle_cpu(cpu);
                }
+               sd->last_balance = jiffies;
+               interval = get_sd_balance_interval(sd, busy);
                if (need_serialize)
                        atomic_set_release(&sched_balance_running, 0);
 out:
--

2.49.0.805.g082f7c87e0-goog

#7 lore raw
In reply to Eric Dumazet
Quoted 10 lines · Eric Dumazet

rebalance_domains() can attempt to change sched_balance_running more than 350,000 times per second on our servers.

If sched_clock_irqtime and sched_balance_running share the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.

This patch only acquires sched_balance_running when sd->last_balance is old enough.

Quoted 7 lines

It also moves sched_balance_running into a dedicated cache line on SMP.

Cc Yafang Shao <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Josh Don <[email protected]>

---

Hi Eric.

Please check this thread which does the same thing AFAIU.
https://lore.kernel.org/all/[email protected]/

You could also try:
https://lore.kernel.org/all/[email protected]/

Thanks

#8 lore raw
Quoted 22 lines · Shrikanth Hegde
Quoted 10 lines · Eric Dumazet

rebalance_domains() can attempt to change sched_balance_running more than 350,000 times per second on our servers.

If sched_clock_irqtime and sched_balance_running share the same cache line, we see a very high cost on hosts with 480 threads dealing with many interrupts.

This patch only acquires sched_balance_running when sd->last_balance is old enough.

Quoted 7 lines

It also moves sched_balance_running into a dedicated cache line on SMP.

Cc Yafang Shao <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Josh Don <[email protected]>

---

Quoted 4 lines

Hi Eric.

Please check this thread which does the same thing AFAIU.
https://lore.kernel.org/all/[email protected]/

Ah perfect, can we get this merged ?

Quoted 6 lines

You could also try:
https://lore.kernel.org/all/[email protected]/

Thanks