sched/clock: Avoid false sharing for sched_clock_irqtime
PATCH v4 1/1 sched/clock: Avoid false sharing for sched_clock_irqtime
Read-mostly sched_clock_irqtime may share the same cacheline with frequently updated nohz struct. Make it as static_key to avoid false sharing issue.
Details:
In kernel 6.14, we observed ~3% cycles hotspots in irqtime_account_irq when running SPECjbb2015 in a 2-sockets system. Most of cycles spent in reading sched_clock_irqtime, which is a read-mostly var.
perf c2c (cachelien view) shows it has false sharing with nohz struct:
Num RmtHitm LclHitm Offset records Symbol
6.25% 0.00% 0.00% 0x0 4 [k] _nohz_idle_balance.isra.0 18.75% 100.00% 0.00% 0x8 14 [k] nohz_balance_exit_idle 6.25% 0.00% 0.00% 0x8 8 [k] nohz_balance_enter_idle 6.25% 0.00% 0.00% 0xc 8 [k] sched_balance_newidle 6.25% 0.00% 0.00% 0x10 31 [k] nohz_balancer_kick 6.25% 0.00% 0.00% 0x20 16 [k] sched_balance_newidle 37.50% 0.00% 0.00% 0x38 50 [k] irqtime_account_irq 6.25% 0.00% 0.00% 0x38 47 [k] account_process_tick 6.25% 0.00% 0.00% 0x38 12 [k] account_idle_ticks
Offsets:
* 0x0 -- nohz.idle_cpu_mask (r)
* 0x8 -- nohz.nr_cpus (w)
* 0x38 -- sched_clock_irqtime (r), not in nohz, but share cacheline
The layout in /proc/kallsyms can also confirm that:ffffffff88600d40 b nohzffffffff88600d68 B arch_needs_tick_broadcast ffffffff88600d6c b __key.264 ffffffff88600d6c b __key.265 ffffffff88600d70 b dl_generation ffffffff88600d78 b sched_clock_irqtime
With the patch applied, irqtime_account_irq hotspot disappear.
---
V4 -> V3:
- Avoid creating a new workqueue to disable static_key
- Specify kernel version for c2c result in changelog
V2 -> V3:
- Use static_key instead of a __read_mostly var.
V1 -> V2:
- Use __read_mostly instead of __cacheline_aligned to avoid wasting spaces.
History:
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
prev discussions: https://lore.kernel.org/all/[email protected]/T/#u
--- arch/x86/kernel/tsc.c | 2 -- kernel/sched/clock.c | 3 +++ kernel/sched/cputime.c | 8 ++++---- kernel/sched/sched.h | 4 ++-- 4 files changed, 9 insertions(+), 8 deletions(-)
arch/x86/kernel/tsc.c +0 -2
@@ -1142,7 +1142,6 @@ static void tsc_cs_mark_unstable(struct clocksource *cs) tsc_unstable =1 ;if (using_native_sched_clock ())clear_sched_clock_stable (); -disable_sched_clock_irqtime ();pr_info ("Marking TSC unstable due to clocksource watchdog\n" ); }
@@ -1212,7 +1211,6 @@ void mark_tsc_unstable(char *reason) tsc_unstable =1 ;if (using_native_sched_clock ())clear_sched_clock_stable (); -disable_sched_clock_irqtime ();pr_info ("Marking TSC unstable due to %s\n" , reason);clocksource_mark_unstable (&clocksource_tsc_early);
kernel/sched/clock.c +3 -0
@@ -173,6 +173,7 @@ notrace static void __sched_clock_work(struct work_struct *work) scd->tick_gtod, __gtod_offset, scd->tick_raw, __sched_clock_offset); +disable_sched_clock_irqtime ();static_branch_disable (&__sched_clock_stable); }
@@ -238,6 +239,8 @@ static int __init sched_clock_init_late(void)if (__sched_clock_stable_early)__set_sched_clock_stable (); +else +disable_sched_clock_irqtime ();/* disable if clock unstable. */ return 0 ; }
kernel/sched/cputime.c +4 -4
@@ -12,6 +12,8 @@#ifdef CONFIG_IRQ_TIME_ACCOUNTING +DEFINE_STATIC_KEY_FALSE (sched_clock_irqtime); +/* * There are no locks covering percpu hardirq/softirq time. * They are only modified in vtime_account, on corresponding CPU
@@ -25,16 +27,14 @@*/ DEFINE_PER_CPU(struct irqtime, cpu_irqtime); -int sched_clock_irqtime; -void enable_sched_clock_irqtime (void ) { - sched_clock_irqtime =1 ; +static_branch_enable (&sched_clock_irqtime); }void disable_sched_clock_irqtime (void ) { - sched_clock_irqtime =0 ; +static_branch_disable (&sched_clock_irqtime); } static void irqtime_account_delta(struct irqtime *irqtime, u64 delta,
kernel/sched/sched.h +2 -2
@@ -3172,11 +3172,11 @@ struct irqtime { }; DECLARE_PER_CPU(struct irqtime, cpu_irqtime); -extern int sched_clock_irqtime; +DECLARE_STATIC_KEY_FALSE (sched_clock_irqtime);static inline int irqtime_enabled (void ) { -return sched_clock_irqtime; +return static_branch_likely (&sched_clock_irqtime); }/*
Signature · 1 line
2.47.3
Hi Wangyang.
Quoted 9 lines · Wangyang Guo
Read-mostly
sched_clock_irqtimemay share the same cacheline with frequently updated nohz struct. Make it asstatic_keyto avoid false sharing issue.Details:
In kernel 6.14, we observed ~3% cycles hotspots inirqtime_account_irqwhen running SPECjbb2015 in a 2-sockets system. Most of cycles spent in readingsched_clock_irqtime, which is a read-mostly var.
IMHO, this changelog is meant for latest sched/core. having 6.14 data won't be that relevant.
Also, changelog should be mentioning essence of calling disable in sched_clock_init_late which was in v3 discussions.
Quoted 25 lines
perf c2c (cachelien view) shows it has false sharing with nohz struct:
Num RmtHitm LclHitm Offset records Symbol
6.25% 0.00% 0.00%0x04 [k] _nohz_idle_balance.isra.018.75% 100.00% 0.00% 0x8 14 [k] nohz_balance_exit_idle 6.25% 0.00% 0.00% 0x8 8 [k] nohz_balance_enter_idle 6.25% 0.00% 0.00% 0xc 8 [k] sched_balance_newidle 6.25% 0.00% 0.00% 0x10 31 [k] nohz_balancer_kick 6.25% 0.00% 0.00% 0x20 16 [k] sched_balance_newidle 37.50% 0.00% 0.00% 0x38 50 [k] irqtime_account_irq 6.25% 0.00% 0.00% 0x38 47 [k] account_process_tick 6.25% 0.00% 0.00% 0x38 12 [k] account_idle_ticksOffsets:
*0x0-- nohz.idle_cpu_mask (r)
*0x8-- nohz.nr_cpus (w)
*0x38--sched_clock_irqtime(r), not in nohz, but share cachelineThe layout in
/proc/kallsymscan also confirm that:ffffffff88600d40b nohzffffffff88600d68Barch_needs_tick_broadcastffffffff88600d6cb __key.264ffffffff88600d6cb __key.265ffffffff88600d70bdl_generationffffffff88600d78bsched_clock_irqtime
Quoted 3 lines
With the patch applied,
irqtime_account_irqhotspot disappear.---
Please dont put a --- here.
git am will strip everything after this and actual diff. So the your signed-off-by and other details won't be captured.
If the intention was keep version history, put it after the signed-off-by for example:
https://lore.kernel.org/lkml/[email protected]/
Quoted 122 lines
V4 -> V3:
- Avoid creating a new workqueue to disablestatic_key
- Specify kernel version for c2c result in changelogV2 -> V3:
- Usestatic_keyinstead of a__read_mostlyvar.V1 -> V2:
- Use__read_mostlyinstead of__cacheline_alignedto avoid wasting spaces.History:
v3: https://lore.kernel.org/all/[email protected]/ v2: https://lore.kernel.org/all/[email protected]/ v1: https://lore.kernel.org/all/[email protected]/ prev discussions: https://lore.kernel.org/all/[email protected]/T/#uSuggested-by K Prateek Nayak <[email protected]>Suggested-by Peter Zijlstra <[email protected]>Suggested-by Shrikanth Hegde <[email protected]>Reported-by Benjamin Lei <[email protected]>Reviewed-by Tim Chen <[email protected]>Reviewed-by Tianyou Li <[email protected]>Signed-off-by Wangyang Guo <[email protected]>--- arch/x86/kernel/tsc.c | 2 -- kernel/sched/clock.c | 3 +++ kernel/sched/cputime.c | 8 ++++---- kernel/sched/sched.h | 4 ++-- 4 files changed, 9 insertions(+), 8 deletions(-)arch/x86/kernel/tsc.c +0
-2@@ -1142,7 +1142,6 @@ static void tsc_cs_mark_unstable(struct clocksource *cs) tsc_unstable =1 ;if (using_native_sched_clock ())clear_sched_clock_stable (); -disable_sched_clock_irqtime ();pr_info ("Marking TSC unstable due to clocksource watchdog\n" ); }@@ -1212,7 +1211,6 @@ void mark_tsc_unstable(char *reason) tsc_unstable =1 ;if (using_native_sched_clock ())clear_sched_clock_stable (); -disable_sched_clock_irqtime ();pr_info ("Marking TSC unstable due to %s\n" , reason);clocksource_mark_unstable (&clocksource_tsc_early);kernel/sched/clock.c +3
-0@@ -173,6 +173,7 @@ notrace static void __sched_clock_work(struct work_struct *work) scd->tick_gtod, __gtod_offset, scd->tick_raw, __sched_clock_offset); +disable_sched_clock_irqtime ();static_branch_disable (&__sched_clock_stable); }@@ -238,6 +239,8 @@ static int __init sched_clock_init_late(void)if (__sched_clock_stable_early)__set_sched_clock_stable (); +else +disable_sched_clock_irqtime ();/* disable if clock unstable. */ return 0 ; }kernel/sched/cputime.c +4
-4@@ -12,6 +12,8 @@#ifdef CONFIG_IRQ_TIME_ACCOUNTING +DEFINE_STATIC_KEY_FALSE (sched_clock_irqtime); +/* * There are no locks covering percpu hardirq/softirq time. * They are only modified in vtime_account, on corresponding CPU @@ -25,16 +27,14 @@*/ DEFINE_PER_CPU(struct irqtime, cpu_irqtime); -int sched_clock_irqtime; -void enable_sched_clock_irqtime (void ) { - sched_clock_irqtime =1 ; +static_branch_enable (&sched_clock_irqtime); }void disable_sched_clock_irqtime (void ) { - sched_clock_irqtime =0 ; +static_branch_disable (&sched_clock_irqtime); } static void irqtime_account_delta(struct irqtime *irqtime, u64 delta,kernel/sched/sched.h +2
-2@@ -3172,11 +3172,11 @@ struct irqtime { }; DECLARE_PER_CPU(struct irqtime, cpu_irqtime); -extern int sched_clock_irqtime; +DECLARE_STATIC_KEY_FALSE (sched_clock_irqtime);static inline int irqtime_enabled (void ) { -return sched_clock_irqtime; +return static_branch_likely (&sched_clock_irqtime); }/*
Quoted 48 lines · Shrikanth Hegde
Hi Wangyang.
Quoted 9 lines · Wangyang Guo
Read-mostly
sched_clock_irqtimemay share the same cacheline with frequently updated nohz struct. Make it asstatic_keyto avoid false sharing issue.Details:
In kernel 6.14, we observed ~3% cycles hotspots inirqtime_account_irqwhen running SPECjbb2015 in a 2-sockets system. Most of cycles spent in readingsched_clock_irqtime, which is a read-mostly var.IMHO, this changelog is meant for latest sched/core. having 6.14 data won't be that relevant.
Also, changelog should be mentioning essence of calling disable in
sched_clock_init_latewhich was in v3 discussions.Quoted 25 lines
perf c2c (cachelien view) shows it has false sharing with nohz struct:
Num RmtHitm LclHitm Offset records Symbol 6.25% 0.00% 0.00% 0x0 4 [k] _nohz_idle_balance.isra.0 18.75% 100.00% 0.00% 0x8 14 [k]nohz_balance_exit_idle6.25% 0.00% 0.00% 0x8 8 [k]nohz_balance_enter_idle6.25% 0.00% 0.00% 0xc 8 [k]sched_balance_newidle6.25% 0.00% 0.00% 0x10 31 [k]nohz_balancer_kick6.25% 0.00% 0.00% 0x20 16 [k]sched_balance_newidle37.50% 0.00% 0.00% 0x38 50 [k]irqtime_account_irq6.25% 0.00% 0.00% 0x38 47 [k]account_process_tick6.25% 0.00% 0.00% 0x38 12 [k]account_idle_ticksOffsets:
*0x0-- nohz.idle_cpu_mask (r)
*0x8-- nohz.nr_cpus (w)
*0x38--sched_clock_irqtime(r), not in nohz, but share cachelineThe layout in
/proc/kallsymscan also confirm that:ffffffff88600d40b nohzffffffff88600d68Barch_needs_tick_broadcastffffffff88600d6cb __key.264ffffffff88600d6cb __key.265ffffffff88600d70bdl_generationffffffff88600d78bsched_clock_irqtimeQuoted 3 lines
With the patch applied,
irqtime_account_irqhotspot disappear.---
Quoted 8 lines
Please dont put a --- here.
git am will strip everything after this and actual diff. So the your signed-off-by and other details won't be captured.
If the intention was keep version history, put it after the signed-off-by for example:
https://lore.kernel.org/lkml/[email protected]/
Changelog updated in v5:
https://lore.kernel.org/all/[email protected]/
BR
Wangyang