Skip to contentSkip to conversation

sched/clock: Avoid false sharing for sched_clock_irqtime

PATCH Superseded Wangyang Guo 2 participants 3 messages Jan 26 – Jan 27

Subsystems (2)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 sched/clock: Avoid false sharing for sched_clock_irqtime

Read-mostly sched_clock_irqtime may share the same cacheline with frequently updated nohz struct. Make it as static_key to avoid false sharing issue.

Details:
In kernel 6.14, we observed ~3% cycles hotspots in irqtime_account_irq when running SPECjbb2015 in a 2-sockets system. Most of cycles spent in reading sched_clock_irqtime, which is a read-mostly var.

perf c2c (cachelien view) shows it has false sharing with nohz struct:
Num RmtHitm LclHitm Offset records Symbol

   6.25%   0.00%   0.00%   0x0       4   [k] _nohz_idle_balance.isra.0
  18.75% 100.00%   0.00%   0x8      14   [k] nohz_balance_exit_idle
   6.25%   0.00%   0.00%   0x8       8   [k] nohz_balance_enter_idle
   6.25%   0.00%   0.00%   0xc       8   [k] sched_balance_newidle
   6.25%   0.00%   0.00%  0x10      31   [k] nohz_balancer_kick
   6.25%   0.00%   0.00%  0x20      16   [k] sched_balance_newidle
  37.50%   0.00%   0.00%  0x38      50   [k] irqtime_account_irq
   6.25%   0.00%   0.00%  0x38      47   [k] account_process_tick
   6.25%   0.00%   0.00%  0x38      12   [k] account_idle_ticks

Offsets:
* 0x0 -- nohz.idle_cpu_mask (r)
* 0x8 -- nohz.nr_cpus (w)
* 0x38 -- sched_clock_irqtime (r), not in nohz, but share cacheline

The layout in /proc/kallsyms can also confirm that:
ffffffff88600d40 b nohz
ffffffff88600d68 B arch_needs_tick_broadcast ffffffff88600d6c b __key.264 ffffffff88600d6c b __key.265 ffffffff88600d70 b dl_generation ffffffff88600d78 b sched_clock_irqtime

With the patch applied, irqtime_account_irq hotspot disappear.

---
V4 -> V3:
- Avoid creating a new workqueue to disable static_key
- Specify kernel version for c2c result in changelog

V2 -> V3:
- Use static_key instead of a __read_mostly var.

V1 -> V2:
- Use __read_mostly instead of __cacheline_aligned to avoid wasting spaces.

History:
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
prev discussions: https://lore.kernel.org/all/[email protected]/T/#u

---
 arch/x86/kernel/tsc.c  | 2 --
 kernel/sched/clock.c   | 3 +++
 kernel/sched/cputime.c | 8 ++++----
 kernel/sched/sched.h   | 4 ++--
 4 files changed, 9 insertions(+), 8 deletions(-)
arch/x86/kernel/tsc.c +0 -2
index 87e749106dda..9a62e18d1bff 100644
@@ -1142,7 +1142,6 @@ static void tsc_cs_mark_unstable(struct clocksource *cs)
 	tsc_unstable = 1;
 	if (using_native_sched_clock())
 		clear_sched_clock_stable();
-	disable_sched_clock_irqtime();
 	pr_info("Marking TSC unstable due to clocksource watchdog\n");
 }
 
@@ -1212,7 +1211,6 @@ void mark_tsc_unstable(char *reason)
 	tsc_unstable = 1;
 	if (using_native_sched_clock())
 		clear_sched_clock_stable();
-	disable_sched_clock_irqtime();
 	pr_info("Marking TSC unstable due to %s\n", reason);
 
 	clocksource_mark_unstable(&clocksource_tsc_early);
kernel/sched/clock.c +3 -0
index f5e6dd6a6b3a..2ae4fbf13431 100644
@@ -173,6 +173,7 @@ notrace static void __sched_clock_work(struct work_struct *work)
 			scd->tick_gtod, __gtod_offset,
 			scd->tick_raw,  __sched_clock_offset);
 
+	disable_sched_clock_irqtime();
 	static_branch_disable(&__sched_clock_stable);
 }
 
@@ -238,6 +239,8 @@ static int __init sched_clock_init_late(void)
 
 	if (__sched_clock_stable_early)
 		__set_sched_clock_stable();
+	else
+		disable_sched_clock_irqtime();  /* disable if clock unstable. */
 
 	return 0;
 }
kernel/sched/cputime.c +4 -4
index 7097de2c8cda..959a86206c64 100644
@@ -12,6 +12,8 @@
 
 #ifdef CONFIG_IRQ_TIME_ACCOUNTING
 
+DEFINE_STATIC_KEY_FALSE(sched_clock_irqtime);
+
 /*
  * There are no locks covering percpu hardirq/softirq time.
  * They are only modified in vtime_account, on corresponding CPU
@@ -25,16 +27,14 @@
  */
 DEFINE_PER_CPU(struct irqtime, cpu_irqtime);
 
-int sched_clock_irqtime;
-
 void enable_sched_clock_irqtime(void)
 {
-	sched_clock_irqtime = 1;
+	static_branch_enable(&sched_clock_irqtime);
 }
 
 void disable_sched_clock_irqtime(void)
 {
-	sched_clock_irqtime = 0;
+	static_branch_disable(&sched_clock_irqtime);
 }
 
 static void irqtime_account_delta(struct irqtime *irqtime, u64 delta,
kernel/sched/sched.h +2 -2
index adfb6e3409d7..ec963314287a 100644
@@ -3172,11 +3172,11 @@ struct irqtime {
 };
 
 DECLARE_PER_CPU(struct irqtime, cpu_irqtime);
-extern int sched_clock_irqtime;
+DECLARE_STATIC_KEY_FALSE(sched_clock_irqtime);
 
 static inline int irqtime_enabled(void)
 {
-	return sched_clock_irqtime;
+	return static_branch_likely(&sched_clock_irqtime);
 }
 
 /*
Signature · 1 line
2.47.3
#1 lore raw

Hi Wangyang.

Quoted 9 lines · Wangyang Guo

Read-mostly sched_clock_irqtime may share the same cacheline with frequently updated nohz struct. Make it as static_key to avoid false sharing issue.

Details:
In kernel 6.14, we observed ~3% cycles hotspots in irqtime_account_irq when running SPECjbb2015 in a 2-sockets system. Most of cycles spent in reading sched_clock_irqtime, which is a read-mostly var.

IMHO, this changelog is meant for latest sched/core. having 6.14 data won't be that relevant.

Also, changelog should be mentioning essence of calling disable in sched_clock_init_late which was in v3 discussions.

Quoted 25 lines

perf c2c (cachelien view) shows it has false sharing with nohz struct:
Num RmtHitm LclHitm Offset records Symbol
6.25% 0.00% 0.00% 0x0 4 [k] _nohz_idle_balance.isra.0

   18.75% 100.00%   0.00%   0x8      14   [k] nohz_balance_exit_idle
    6.25%   0.00%   0.00%   0x8       8   [k] nohz_balance_enter_idle
    6.25%   0.00%   0.00%   0xc       8   [k] sched_balance_newidle
    6.25%   0.00%   0.00%  0x10      31   [k] nohz_balancer_kick
    6.25%   0.00%   0.00%  0x20      16   [k] sched_balance_newidle
   37.50%   0.00%   0.00%  0x38      50   [k] irqtime_account_irq
    6.25%   0.00%   0.00%  0x38      47   [k] account_process_tick
    6.25%   0.00%   0.00%  0x38      12   [k] account_idle_ticks

Offsets:
* 0x0 -- nohz.idle_cpu_mask (r)
* 0x8 -- nohz.nr_cpus (w)
* 0x38 -- sched_clock_irqtime (r), not in nohz, but share cacheline

The layout in /proc/kallsyms can also confirm that:
ffffffff88600d40 b nohz
ffffffff88600d68 B arch_needs_tick_broadcast ffffffff88600d6c b __key.264 ffffffff88600d6c b __key.265 ffffffff88600d70 b dl_generation ffffffff88600d78 b sched_clock_irqtime

Quoted 3 lines

With the patch applied, irqtime_account_irq hotspot disappear.

---

Please dont put a --- here.

git am will strip everything after this and actual diff. So the your signed-off-by and other details won't be captured.

If the intention was keep version history, put it after the signed-off-by for example:
https://lore.kernel.org/lkml/[email protected]/

Quoted 122 lines

V4 -> V3:
- Avoid creating a new workqueue to disable static_key
- Specify kernel version for c2c result in changelog

V2 -> V3:
- Use static_key instead of a __read_mostly var.

V1 -> V2:
- Use __read_mostly instead of __cacheline_aligned to avoid wasting spaces.

History:

   v3: https://lore.kernel.org/all/[email protected]/
   v2: https://lore.kernel.org/all/[email protected]/
   v1: https://lore.kernel.org/all/[email protected]/
   prev discussions: https://lore.kernel.org/all/[email protected]/T/#u
---
  arch/x86/kernel/tsc.c  | 2 --
  kernel/sched/clock.c   | 3 +++
  kernel/sched/cputime.c | 8 ++++----
  kernel/sched/sched.h   | 4 ++--
  4 files changed, 9 insertions(+), 8 deletions(-)
arch/x86/kernel/tsc.c +0 -2
index 87e749106dda..9a62e18d1bff 100644
@@ -1142,7 +1142,6 @@ static void tsc_cs_mark_unstable(struct clocksource *cs)
  	tsc_unstable = 1;
  	if (using_native_sched_clock())
  		clear_sched_clock_stable();
-	disable_sched_clock_irqtime();
  	pr_info("Marking TSC unstable due to clocksource watchdog\n");
  }
 
@@ -1212,7 +1211,6 @@ void mark_tsc_unstable(char *reason)
  	tsc_unstable = 1;
  	if (using_native_sched_clock())
  		clear_sched_clock_stable();
-	disable_sched_clock_irqtime();
  	pr_info("Marking TSC unstable due to %s\n", reason);
 
  	clocksource_mark_unstable(&clocksource_tsc_early);
kernel/sched/clock.c +3 -0
index f5e6dd6a6b3a..2ae4fbf13431 100644
@@ -173,6 +173,7 @@ notrace static void __sched_clock_work(struct work_struct *work)
  			scd->tick_gtod, __gtod_offset,
  			scd->tick_raw,  __sched_clock_offset);
 
+	disable_sched_clock_irqtime();
  	static_branch_disable(&__sched_clock_stable);
  }
 
@@ -238,6 +239,8 @@ static int __init sched_clock_init_late(void)
 
  	if (__sched_clock_stable_early)
  		__set_sched_clock_stable();
+	else
+		disable_sched_clock_irqtime();  /* disable if clock unstable. */
 
  	return 0;
  }
kernel/sched/cputime.c +4 -4
index 7097de2c8cda..959a86206c64 100644
@@ -12,6 +12,8 @@
 
  #ifdef CONFIG_IRQ_TIME_ACCOUNTING
 
+DEFINE_STATIC_KEY_FALSE(sched_clock_irqtime);
+
  /*
   * There are no locks covering percpu hardirq/softirq time.
   * They are only modified in vtime_account, on corresponding CPU
@@ -25,16 +27,14 @@
   */
  DEFINE_PER_CPU(struct irqtime, cpu_irqtime);
 
-int sched_clock_irqtime;
-
  void enable_sched_clock_irqtime(void)
  {
-	sched_clock_irqtime = 1;
+	static_branch_enable(&sched_clock_irqtime);
  }
 
  void disable_sched_clock_irqtime(void)
  {
-	sched_clock_irqtime = 0;
+	static_branch_disable(&sched_clock_irqtime);
  }
 
  static void irqtime_account_delta(struct irqtime *irqtime, u64 delta,
kernel/sched/sched.h +2 -2
index adfb6e3409d7..ec963314287a 100644
@@ -3172,11 +3172,11 @@ struct irqtime {
  };
 
  DECLARE_PER_CPU(struct irqtime, cpu_irqtime);
-extern int sched_clock_irqtime;
+DECLARE_STATIC_KEY_FALSE(sched_clock_irqtime);
 
  static inline int irqtime_enabled(void)
  {
-	return sched_clock_irqtime;
+	return static_branch_likely(&sched_clock_irqtime);
  }
 
  /*
#2 lore raw
Quoted 48 lines · Shrikanth Hegde

Hi Wangyang.

Quoted 9 lines · Wangyang Guo

Read-mostly sched_clock_irqtime may share the same cacheline with frequently updated nohz struct. Make it as static_key to avoid false sharing issue.

Details:
In kernel 6.14, we observed ~3% cycles hotspots in irqtime_account_irq when running SPECjbb2015 in a 2-sockets system. Most of cycles spent in reading sched_clock_irqtime, which is a read-mostly var.

IMHO, this changelog is meant for latest sched/core. having 6.14 data won't be that relevant.

Also, changelog should be mentioning essence of calling disable in sched_clock_init_late which was in v3 discussions.

Quoted 25 lines

perf c2c (cachelien view) shows it has false sharing with nohz struct:
      Num RmtHitm LclHitm  Offset records             Symbol 6.25%   0.00%   0.00%   0x0       4   [k] _nohz_idle_balance.isra.0 18.75% 100.00%   0.00%   0x8      14   [k] nohz_balance_exit_idle 6.25%   0.00%   0.00%   0x8       8   [k] nohz_balance_enter_idle 6.25%   0.00%   0.00%   0xc       8   [k] sched_balance_newidle 6.25%   0.00%   0.00%  0x10      31   [k] nohz_balancer_kick 6.25%   0.00%   0.00%  0x20      16   [k] sched_balance_newidle 37.50%   0.00%   0.00%  0x38      50   [k] irqtime_account_irq 6.25%   0.00%   0.00%  0x38      47   [k] account_process_tick 6.25%   0.00%   0.00%  0x38      12   [k] account_idle_ticks

Offsets:
*  0x0 -- nohz.idle_cpu_mask (r)
*  0x8 -- nohz.nr_cpus (w)
* 0x38 -- sched_clock_irqtime (r), not in nohz, but share cacheline

The layout in /proc/kallsyms can also confirm that:
ffffffff88600d40 b nohz
ffffffff88600d68 B arch_needs_tick_broadcast ffffffff88600d6c b __key.264 ffffffff88600d6c b __key.265 ffffffff88600d70 b dl_generation ffffffff88600d78 b sched_clock_irqtime

Quoted 3 lines

With the patch applied, irqtime_account_irq hotspot disappear.

---

Quoted 8 lines

Please dont put a --- here.

git am will strip everything after this and actual diff. So the your signed-off-by and other details won't be captured.

If the intention was keep version history, put it after the signed-off-by for example:
https://lore.kernel.org/lkml/[email protected]/

Changelog updated in v5:

https://lore.kernel.org/all/[email protected]/

BR
Wangyang