tick/nohz: Narrow down noise while setting current task's tick dependency
Frederic Weisbecker
maintainerPATCH 1/1 tick/nohz: Narrow down noise while setting current task's tick dependency
So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on nohz_full, a combo that nobody seemed to use in real life.
But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on nohz_full.
The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.
--- kernel/time/tick-sched.c | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-)
kernel/time/tick-sched.c +15 -7
@@ -351,16 +351,24 @@ void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit)EXPORT_SYMBOL_GPL (tick_nohz_dep_clear_cpu);/* -* Set a per-task tick dependency. Posix CPU timers need this in order to elapse -* per task timers. +* Set a per-task tick dependency. RCU need this. Also posix CPU timers +* in order to elapse per task timers. */ void tick_nohz_dep_set_task (struct task_struct *tsk,enum tick_dep_bits bit) { -/* -* We could optimize this with just kicking the target running the task -* if that noise matters for nohz full users. -*/ -tick_nohz_dep_set_all (&tsk->tick_dep_mask, bit); +if (!atomic_fetch_or (BIT (bit), &tsk->tick_dep_mask)) { +if (tsk == current) { +preempt_disable (); +tick_nohz_full_kick (); +preempt_enable (); + }else { +/* +* Some future tick_nohz_full_kick_task() +* should optimize this. +*/ +tick_nohz_full_kick_all (); + } + } }EXPORT_SYMBOL_GPL (tick_nohz_dep_set_task);
Signature · 1 line
2.26.2
Quoted 11 lines · Frederic Weisbecker
So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on
nohz_full, a combo that nobody seemed to use in real life.But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on
nohz_full.The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.
This passes moderate rcutorture testing. If you want me to take it, please let me know, and otherwise:
Quoted 51 lines
Fixes 6a949b7af82d (rcu: Force on tick when invoking lots of callbacks)Reported-by Matt Fleming <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Ingo Molnar <[email protected]>--- kernel/time/tick-sched.c | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-)kernel/time/tick-sched.c +15
-7@@ -351,16 +351,24 @@ void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit)EXPORT_SYMBOL_GPL (tick_nohz_dep_clear_cpu);/* -* Set a per-task tick dependency. Posix CPU timers need this in order to elapse -* per task timers. +* Set a per-task tick dependency. RCU need this. Also posix CPU timers +* in order to elapse per task timers. */ void tick_nohz_dep_set_task (struct task_struct *tsk,enum tick_dep_bits bit) { -/* -* We could optimize this with just kicking the target running the task -* if that noise matters for nohz full users. -*/ -tick_nohz_dep_set_all (&tsk->tick_dep_mask, bit); +if (!atomic_fetch_or (BIT (bit), &tsk->tick_dep_mask)) { +if (tsk == current) { +preempt_disable (); +tick_nohz_full_kick (); +preempt_enable (); + }else { +/* +* Some future tick_nohz_full_kick_task() +* should optimize this. +*/ +tick_nohz_full_kick_all (); + } + } }EXPORT_SYMBOL_GPL (tick_nohz_dep_set_task);Signature · 1 line
2.26.2
Frederic Weisbecker
maintainerQuoted 10 lines · Paul E. McKenney
Quoted 9 lines · Frederic Weisbecker
So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on
nohz_full, a combo that nobody seemed to use in real life.But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on
nohz_full.
Quoted 7 lines
Quoted 2 lines
The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.
This passes moderate rcutorture testing. If you want me to take it, please let me know, and otherwise:
Tested-by Paul E. McKenney <[email protected]>
If you already have a pending urgent queue, I'd love you to take it. If not I can take it.
Thanks.
Quoted 14 lines · Frederic Weisbecker
Quoted 13 lines · Paul E. McKenney
Quoted 11 lines · Frederic Weisbecker
So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on
nohz_full, a combo that nobody seemed to use in real life.But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on
nohz_full.The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.
Quoted 7 lines
Quoted 4 lines
This passes moderate rcutorture testing. If you want me to take it, please let me know, and otherwise:
Tested-by Paul E. McKenney <[email protected]>If you already have a pending urgent queue, I'd love you to take it. If not I can take it.
Nothing urgent yet in -rcu, so if you would like it in the next merge window, please take it through your normal upstream path.
Thanx, Paul
Frederic Weisbecker
maintainerQuoted 18 lines · Paul E. McKenney
Quoted 17 lines · Frederic Weisbecker
Quoted 16 lines · Paul E. McKenney
Quoted 11 lines · Frederic Weisbecker
So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on
nohz_full, a combo that nobody seemed to use in real life.But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on
nohz_full.The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.
This passes moderate rcutorture testing. If you want me to take it, please let me know, and otherwise:
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Tested-by Paul E. McKenney <[email protected]>If you already have a pending urgent queue, I'd love you to take it. If not I can take it.
Nothing urgent yet in -rcu, so if you would like it in the next merge window, please take it through your normal upstream path.
Got it, thanks!
Quoted 35 lines · Frederic Weisbecker
So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on
nohz_full, a combo that nobody seemed to use in real life.But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on
nohz_full.The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.
Fixes 6a949b7af82d (rcu: Force on tick when invoking lots of callbacks)Reported-by Matt Fleming <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Ingo Molnar <[email protected]>--- kernel/time/tick-sched.c | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-)kernel/time/tick-sched.c +1
-2@@ -351,16 +351,24 @@ void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit)EXPORT_SYMBOL_GPL (tick_nohz_dep_clear_cpu);/* -* Set a per-task tick dependency. Posix CPU timers need this in order to elapse -* per task timers. +* Set a per-task tick dependency. RCU need this. Also posix CPU timers
kernel/time/tick-sched.c void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) · line 359
Quoted 10 lines
kernel/time/tick-sched.c +2
-5+* in order to elapse per task timers. */ void tick_nohz_dep_set_task (struct task_struct *tsk,enum tick_dep_bits bit) { -/* -* We could optimize this with just kicking the target running the task -* if that noise matters for nohz full users. -*/ -tick_nohz_dep_set_all (&tsk->tick_dep_mask, bit); +if (!atomic_fetch_or (BIT (bit), &tsk->tick_dep_mask)) {
So why not simply:
tick_nohz_full_kick_cpu(task_cpu(tsk)); ?
If it got preempted, the scheduling involved would already have observed the bit we just set and kept the tick on anyway, same for migration.
Or am I missing something?
Quoted 12 lines
kernel/time/tick-sched.c +12
-0+if (tsk == current) { +preempt_disable (); +tick_nohz_full_kick (); +preempt_enable (); + }else { +/* +* Some future tick_nohz_full_kick_task() +* should optimize this. +*/ +tick_nohz_full_kick_all (); + } + }
Frederic Weisbecker
maintainerQuoted 49 lines · Peter Zijlstra
Quoted 35 lines · Frederic Weisbecker
So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on
nohz_full, a combo that nobody seemed to use in real life.But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on
nohz_full.The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.
Fixes 6a949b7af82d (rcu: Force on tick when invoking lots of callbacks)Reported-by Matt Fleming <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Ingo Molnar <[email protected]>--- kernel/time/tick-sched.c | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-)kernel/time/tick-sched.c +1
-2@@ -351,16 +351,24 @@ void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit)EXPORT_SYMBOL_GPL (tick_nohz_dep_clear_cpu);/* -* Set a per-task tick dependency. Posix CPU timers need this in order to elapse -* per task timers. +* Set a per-task tick dependency. RCU need this. Also posix CPU timers kernel/time/tick-sched.c void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) · line 359
Quoted 10 lines
kernel/time/tick-sched.c +2
-5+* in order to elapse per task timers. */ void tick_nohz_dep_set_task (struct task_struct *tsk,enum tick_dep_bits bit) { -/* -* We could optimize this with just kicking the target running the task -* if that noise matters for nohz full users. -*/ -tick_nohz_dep_set_all (&tsk->tick_dep_mask, bit); +if (!atomic_fetch_or (BIT (bit), &tsk->tick_dep_mask)) {So why not simply:
Quoted 6 lines
tick_nohz_full_kick_cpu(task_cpu(tsk)); ?If it got preempted, the scheduling involved would already have observed the bit we just set and kept the tick on anyway, same for migration.
Or am I missing something?
Hmm, I guess we would then need some sort of ordering like this:
CPU 0 CPU 1
----- -----
p->cpu = smp_processor_id() atomic_fetch_or(bit, p->tick_dep_mask)
smp_mb(); smp_mb(); //actually implied by atomic_fetch_or()
READ p->tick_dep_mask irq_work_on(...., p->cpu)And since p->cpu is already set and visible on context switch, it should work indeed. Now in the case CPU 1 reads a stale task_cpu(), that's fine as CPU 0 sees the new tick_dep_mask, but CPU 1 might be dealing with an offlined CPU, right?
So I guess I should still protect against hotplug with cpus_read_lock() but tick_nohz_dep_set_task() isn't supposed to sleep...
Or preempt_disable() could help us with that somehow? I'm always confused with the guarantees that disabled preemption can offer toward hotplug.
Quoted 14 lines
Quoted 13 lines
kernel/time/tick-sched.c +12
-0+if (tsk == current) { +preempt_disable (); +tick_nohz_full_kick (); +preempt_enable (); + }else { +/* +* Some future tick_nohz_full_kick_task() +* should optimize this. +*/ +tick_nohz_full_kick_all (); + } + }