sched/clock: Avoid false sharing for sched_clock_irqtime
PATCH v2 1/1 sched/clock: Avoid false sharing for sched_clock_irqtime
Read-mostly sched_clock_irqtime may share the same cacheline with frequently updated nohz struct. Mark it as __read_mostly to avoid false sharing issue.
Details:
We observed ~3% cycles hotspots in irqtime_account_irq when running SPECjbb2015 in a 2-sockets system. Most of cycles spent in reading sched_clock_irqtime, which is a read-mostly var.
perf c2c (cachelien view) shows it has false sharing with nohz struct:
Num RmtHitm LclHitm Offset records Symbol
6.25% 0.00% 0.00% 0x0 4 [k] _nohz_idle_balance.isra.0 18.75% 100.00% 0.00% 0x8 14 [k] nohz_balance_exit_idle 6.25% 0.00% 0.00% 0x8 8 [k] nohz_balance_enter_idle 6.25% 0.00% 0.00% 0xc 8 [k] sched_balance_newidle 6.25% 0.00% 0.00% 0x10 31 [k] nohz_balancer_kick 6.25% 0.00% 0.00% 0x20 16 [k] sched_balance_newidle 37.50% 0.00% 0.00% 0x38 50 [k] irqtime_account_irq 6.25% 0.00% 0.00% 0x38 47 [k] account_process_tick 6.25% 0.00% 0.00% 0x38 12 [k] account_idle_ticks
Offsets:
* 0x0 -- nohz.idle_cpu_mask (r)
* 0x8 -- nohz.nr_cpus (w)
* 0x38 -- sched_clock_irqtime (r), not in nohz, but share cacheline
The layout in /proc/kallsyms can also confirm that:ffffffff88600d40 b nohzffffffff88600d68 B arch_needs_tick_broadcast ffffffff88600d6c b __key.264 ffffffff88600d6c b __key.265 ffffffff88600d70 b dl_generation ffffffff88600d78 b sched_clock_irqtime
With the patch applied, irqtime_account_irq hotspot disappear.
Changes since V1:
- Use __read_mostly instead of __cacheline_aligned to avoid wasting spaces.
History:
v1: https://lore.kernel.org/all/[email protected]/
prev discussions: https://lore.kernel.org/all/[email protected]/T/#u
--- kernel/sched/cputime.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/cputime.c +1 -1
@@ -25,7 +25,7 @@*/ DEFINE_PER_CPU(struct irqtime, cpu_irqtime); -int sched_clock_irqtime; +int sched_clock_irqtime__read_mostly ;void enable_sched_clock_irqtime (void ) {
Signature · 1 line
2.47.3
Quoted 40 lines · Wangyang Guo
Read-mostly
sched_clock_irqtimemay share the same cacheline with frequently updated nohz struct. Mark it as__read_mostlyto avoid false sharing issue.Details:
We observed ~3% cycles hotspots inirqtime_account_irqwhen running SPECjbb2015 in a 2-sockets system. Most of cycles spent in readingsched_clock_irqtime, which is a read-mostly var.perf c2c (cachelien view) shows it has false sharing with nohz struct:
Num RmtHitm LclHitm Offset records Symbol
6.25% 0.00% 0.00%0x04 [k] _nohz_idle_balance.isra.018.75% 100.00% 0.00% 0x8 14 [k] nohz_balance_exit_idle 6.25% 0.00% 0.00% 0x8 8 [k] nohz_balance_enter_idle 6.25% 0.00% 0.00% 0xc 8 [k] sched_balance_newidle 6.25% 0.00% 0.00% 0x10 31 [k] nohz_balancer_kick 6.25% 0.00% 0.00% 0x20 16 [k] sched_balance_newidle 37.50% 0.00% 0.00% 0x38 50 [k] irqtime_account_irq 6.25% 0.00% 0.00% 0x38 47 [k] account_process_tick 6.25% 0.00% 0.00% 0x38 12 [k] account_idle_ticksOffsets:
*0x0-- nohz.idle_cpu_mask (r)
*0x8-- nohz.nr_cpus (w)
*0x38--sched_clock_irqtime(r), not in nohz, but share cachelineThe layout in
/proc/kallsymscan also confirm that:ffffffff88600d40b nohzffffffff88600d68Barch_needs_tick_broadcastffffffff88600d6cb __key.264ffffffff88600d6cb __key.265ffffffff88600d70bdl_generationffffffff88600d78bsched_clock_irqtimeWith the patch applied,
irqtime_account_irqhotspot disappear.Changes since V1:
- Use__read_mostlyinstead of__cacheline_alignedto avoid wasting spaces.
Quoted 5 lines
History:
v1: https://lore.kernel.org/all/[email protected]/ prev discussions: https://lore.kernel.org/all/[email protected]/T/#uSuggested-by Shrikanth Hegde <[email protected]>
I think it would appropriate to take prateek's tag here.
Quoted 13 lines
Reported-by Benjamin Lei <[email protected]>Reviewed-by Tim Chen <[email protected]>Reviewed-by Tianyou Li <[email protected]>Signed-off-by Wangyang Guo <[email protected]>--- kernel/sched/cputime.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/cputime.c +0
-0
kernel/sched/cputime.c line 31
Quoted 8 lines
kernel/sched/cputime.c +1
-1@@ -25,7 +25,7 @@*/ DEFINE_PER_CPU(struct irqtime, cpu_irqtime); -int sched_clock_irqtime; +int sched_clock_irqtime__read_mostly ;void enable_sched_clock_irqtime (void ) {
Patch should be good on its own.
Peter Zijlstra
maintainerQuoted 4 lines · Wangyang Guo
Read-mostly
sched_clock_irqtimemay share the same cacheline with frequently updated nohz struct. Mark it as__read_mostlyto avoid false sharing issue.
Is there a reason that thing can't be a static_key or so?
K Prateek Nayak
reviewerHello Peter,
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Wangyang Guo
Read-mostly
sched_clock_irqtimemay share the same cacheline with frequently updated nohz struct. Mark it as__read_mostlyto avoid false sharing issue.Is there a reason that thing can't be a
static_keyor so?
Yes, the exact case is stated in commit b9f2b29b9494 ("sched: Don't define sched_clock_irqtime as static key")
https://lore.kernel.org/lkml/173961695743.10177.17683780278419896262.tip-bot2@tip-bot2/
Signature · 2 lines
Thanks and Regards, Prateek
Vincent Guittot
maintainerQuoted 8 lines · Peter Zijlstra
Quoted 4 lines · Wangyang Guo
Read-mostly
sched_clock_irqtimemay share the same cacheline with frequently updated nohz struct. Mark it as__read_mostlyto avoid false sharing issue.Is there a reason that thing can't be a
static_keyor so?
We tried that but disable_sched_clock_irqtime can be called in atomic context with mark_tsc_unstable()
https://lore.kernel.org/all/174161357383.14745.8770394914047302959.tip-bot2@tip-bot2/
Peter Zijlstra
maintainerQuoted 8 lines · Vincent Guittot
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Wangyang Guo
Read-mostly
sched_clock_irqtimemay share the same cacheline with frequently updated nohz struct. Mark it as__read_mostlyto avoid false sharing issue.
Quoted 5 lines
Quoted 1 line
Is there a reason that thing can't be a
static_keyor so?We tried that but
disable_sched_clock_irqtimecan be called in atomic context withmark_tsc_unstable()
https://lore.kernel.org/all/174161357383.14745.8770394914047302959.tip-bot2@tip-bot2/
Right, but if you note, that very same path has clear_sched_clock_stable(), and that is a static_key too.
It's just that that punts by pushing it into a workqueue. So why not lift that workqueue and do both keys or something?
Vincent Guittot
maintainerQuoted 16 lines · Peter Zijlstra
Quoted 8 lines · Vincent Guittot
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Wangyang Guo
Read-mostly
sched_clock_irqtimemay share the same cacheline with frequently updated nohz struct. Mark it as__read_mostlyto avoid false sharing issue.Quoted 5 lines
Quoted 1 line
Is there a reason that thing can't be a
static_keyor so?We tried that but
disable_sched_clock_irqtimecan be called in atomic context withmark_tsc_unstable()
https://lore.kernel.org/all/174161357383.14745.8770394914047302959.tip-bot2@tip-bot2/
Quoted 5 lines
Right, but if you note, that very same path has
clear_sched_clock_stable(), and that is astatic_keytoo.It's just that that punts by pushing it into a workqueue. So why not lift that workqueue and do both keys or something?
fair enough