sched/clock: Avoid false sharing for sched_clock_irqtime
PATCH 1/1 sched/clock: Avoid false sharing for sched_clock_irqtime
Read-mostly sched_clock_irqtime may share the same cacheline with frequently updated nohz struct. Make it as cache aligned to avoid false sharing issue.
Details:
We observed ~3% cycles hotspots in irqtime_account_irq when running SPECjbb2015 in a 2-sockets system. Most of cycles spent in reading sched_clock_irqtime, which is a read-mostly var.
perf c2c (cachelien view) shows it has false sharing with nohz struct:
Num RmtHitm LclHitm Offset records Symbol
6.25% 0.00% 0.00% 0x0 4 [k] _nohz_idle_balance.isra.0 18.75% 100.00% 0.00% 0x8 14 [k] nohz_balance_exit_idle 6.25% 0.00% 0.00% 0x8 8 [k] nohz_balance_enter_idle 6.25% 0.00% 0.00% 0xc 8 [k] sched_balance_newidle 6.25% 0.00% 0.00% 0x10 31 [k] nohz_balancer_kick 6.25% 0.00% 0.00% 0x20 16 [k] sched_balance_newidle 37.50% 0.00% 0.00% 0x38 50 [k] irqtime_account_irq 6.25% 0.00% 0.00% 0x38 47 [k] account_process_tick 6.25% 0.00% 0.00% 0x38 12 [k] account_idle_ticks
Offsets:
* 0x0 -- nohz.idle_cpu_mask (r)
* 0x8 -- nohz.nr_cpus (w)
* 0x38 -- sched_clock_irqtime (r), not in nohz, but share cacheline
The layout in /proc/kallsyms can also confirm that:ffffffff88600d40 b nohzffffffff88600d68 B arch_needs_tick_broadcast ffffffff88600d6c b __key.264 ffffffff88600d6c b __key.265 ffffffff88600d70 b dl_generation ffffffff88600d78 b sched_clock_irqtime
With the patch applied, irqtime_account_irq hotspot disappear.
Previous Discussion:
https://lore.kernel.org/all/[email protected]/T/#u
--- kernel/sched/cputime.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/cputime.c +1 -1
@@ -25,7 +25,7 @@*/ DEFINE_PER_CPU(struct irqtime, cpu_irqtime); -int sched_clock_irqtime; +int sched_clock_irqtime __cacheline_aligned;void enable_sched_clock_irqtime (void ) {
Signature · 1 line
2.47.3
K Prateek Nayak
reviewerHello Wangyang,
Quoted 3 lines · Wangyang Guo
Read-mostly
sched_clock_irqtimemay share the same cacheline with frequently updated nohz struct. Make it as cache aligned to avoid false sharing issue.
nit.
Since you mention this is a "Read-mostly" data ...
[..snip..]
kernel/sched/cputime.c line 28
Quoted 10 lines
kernel/sched/cputime.c +1
-1@@ -25,7 +25,7 @@*/ DEFINE_PER_CPU(struct irqtime, cpu_irqtime); -int sched_clock_irqtime; +int sched_clock_irqtime __cacheline_aligned;
Have you tried annotating it with "__ro_after_init" or "__read_mostly"?
That should also ensure "sched_clock_irqtime" doesn't have false-sharing with the nohz struct without using an entire cacheline worth of space.
Signature · 2 lines
Thanks and Regards, Prateek
Quoted 21 lines · K Prateek Nayak
Hello Wangyang,
Quoted 3 lines · Wangyang Guo
Read-mostly
sched_clock_irqtimemay share the same cacheline with frequently updated nohz struct. Make it as cache aligned to avoid false sharing issue.nit.
Since you mention this is a "Read-mostly" data ...
[..snip..]
Quoted 8 lines
kernel/sched/cputime.c +0
-0@@ -25,7 +25,7 @@*/ DEFINE_PER_CPU(struct irqtime, cpu_irqtime);
Quoted 7 lines
kernel/sched/cputime.c line 28
Quoted 2 lines
kernel/sched/cputime.c +1
-1-int sched_clock_irqtime; +int sched_clock_irqtime __cacheline_aligned;Have you tried annotating it with "
__ro_after_init" or "__read_mostly"?That should also ensure "
sched_clock_irqtime" doesn't have false-sharing with the nohz struct without using an entire cacheline worth of space.
__read_mostly works and avoids wasting spaces. It's better than __cacheline_aligned, patch v2 is sent out as:
https://lore.kernel.org/all/[email protected]/
BR
Wangyang
Hi Wangyang.
Quoted 53 lines · Wangyang Guo
Read-mostly
sched_clock_irqtimemay share the same cacheline with frequently updated nohz struct. Make it as cache aligned to avoid false sharing issue.Details:
We observed ~3% cycles hotspots inirqtime_account_irqwhen running SPECjbb2015 in a 2-sockets system. Most of cycles spent in readingsched_clock_irqtime, which is a read-mostly var.perf c2c (cachelien view) shows it has false sharing with nohz struct:
Num RmtHitm LclHitm Offset records Symbol
6.25% 0.00% 0.00%0x04 [k] _nohz_idle_balance.isra.018.75% 100.00% 0.00% 0x8 14 [k] nohz_balance_exit_idle 6.25% 0.00% 0.00% 0x8 8 [k] nohz_balance_enter_idle 6.25% 0.00% 0.00% 0xc 8 [k] sched_balance_newidle 6.25% 0.00% 0.00% 0x10 31 [k] nohz_balancer_kick 6.25% 0.00% 0.00% 0x20 16 [k] sched_balance_newidle 37.50% 0.00% 0.00% 0x38 50 [k] irqtime_account_irq 6.25% 0.00% 0.00% 0x38 47 [k] account_process_tick 6.25% 0.00% 0.00% 0x38 12 [k] account_idle_ticksOffsets:
*0x0-- nohz.idle_cpu_mask (r)
*0x8-- nohz.nr_cpus (w)
*0x38--sched_clock_irqtime(r), not in nohz, but share cachelineThe layout in
/proc/kallsymscan also confirm that:ffffffff88600d40b nohzffffffff88600d68Barch_needs_tick_broadcastffffffff88600d6cb __key.264ffffffff88600d6cb __key.265ffffffff88600d70bdl_generationffffffff88600d78bsched_clock_irqtimeWith the patch applied,
irqtime_account_irqhotspot disappear.Previous Discussion:
https://lore.kernel.org/all/[email protected]/T/#uSuggested-by Shrikanth Hegde <[email protected]>Reported-by Benjamin Lei <[email protected]>Reviewed-by Tim Chen <[email protected]>Reviewed-by Tianyou Li <[email protected]>Signed-off-by Wangyang Guo <[email protected]>--- kernel/sched/cputime.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/cputime.c +0
-0
kernel/sched/cputime.c line 31
Quoted 8 lines
kernel/sched/cputime.c +1
-1@@ -25,7 +25,7 @@*/ DEFINE_PER_CPU(struct irqtime, cpu_irqtime); -int sched_clock_irqtime; +int sched_clock_irqtime __cacheline_aligned;void enable_sched_clock_irqtime (void ) {
Could you please apply this and see if you observe the issue still.
https://lore.kernel.org/all/[email protected]/
You have CONFIG_CPUMASK_OFFSTACK=y. You still might see occasional cache misses with the patch applied, but it would be quite less.