Skip to contentSkip to conversation

tick/nohz: Narrow down noise while setting current task's tick dependency

PATCH In review Frederic Weisbecker 3 participants 7 messages May 15, 2020 – May 18, 2020 Nohz, dynticks support

lore
Share & tools
lore

Frederic Weisbecker

author maintainer
#0 lore raw

PATCH 1/1 tick/nohz: Narrow down noise while setting current task's tick dependency

So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on nohz_full, a combo that nobody seemed to use in real life.

But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on nohz_full.

The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.

Fixes 6a949b7af82d (rcu: Force on tick when invoking lots of callbacks)
Cc Paul E. McKenney <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/time/tick-sched.c | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)
kernel/time/tick-sched.c +15 -7
index 3e2dc9b8858c..f0199a4ba1ad 100644
@@ -351,16 +351,24 @@ void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit)
 EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_cpu);
 
 /*
- * Set a per-task tick dependency. Posix CPU timers need this in order to elapse
- * per task timers.
+ * Set a per-task tick dependency. RCU need this. Also posix CPU timers
+ * in order to elapse per task timers.
  */
 void tick_nohz_dep_set_task(struct task_struct *tsk, enum tick_dep_bits bit)
 {
-	/*
-	 * We could optimize this with just kicking the target running the task
-	 * if that noise matters for nohz full users.
-	 */
-	tick_nohz_dep_set_all(&tsk->tick_dep_mask, bit);
+	if (!atomic_fetch_or(BIT(bit), &tsk->tick_dep_mask)) {
+		if (tsk == current) {
+			preempt_disable();
+			tick_nohz_full_kick();
+			preempt_enable();
+		} else {
+			/*
+			 * Some future tick_nohz_full_kick_task()
+			 * should optimize this.
+			 */
+			tick_nohz_full_kick_all();
+		}
+	}
 }
 EXPORT_SYMBOL_GPL(tick_nohz_dep_set_task);
 
Signature · 1 line
2.26.2
Tested-by on v1 1/1
#1 lore raw
Quoted 11 lines · Frederic Weisbecker

So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on nohz_full, a combo that nobody seemed to use in real life.

But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on nohz_full.

The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.

This passes moderate rcutorture testing. If you want me to take it, please let me know, and otherwise:

Quoted 51 lines
Fixes 6a949b7af82d (rcu: Force on tick when invoking lots of callbacks)
Cc Paul E. McKenney <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/time/tick-sched.c | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)
kernel/time/tick-sched.c +15 -7
index 3e2dc9b8858c..f0199a4ba1ad 100644
@@ -351,16 +351,24 @@ void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit)
 EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_cpu);
 
 /*
- * Set a per-task tick dependency. Posix CPU timers need this in order to elapse
- * per task timers.
+ * Set a per-task tick dependency. RCU need this. Also posix CPU timers
+ * in order to elapse per task timers.
  */
 void tick_nohz_dep_set_task(struct task_struct *tsk, enum tick_dep_bits bit)
 {
-	/*
-	 * We could optimize this with just kicking the target running the task
-	 * if that noise matters for nohz full users.
-	 */
-	tick_nohz_dep_set_all(&tsk->tick_dep_mask, bit);
+	if (!atomic_fetch_or(BIT(bit), &tsk->tick_dep_mask)) {
+		if (tsk == current) {
+			preempt_disable();
+			tick_nohz_full_kick();
+			preempt_enable();
+		} else {
+			/*
+			 * Some future tick_nohz_full_kick_task()
+			 * should optimize this.
+			 */
+			tick_nohz_full_kick_all();
+		}
+	}
 }
 EXPORT_SYMBOL_GPL(tick_nohz_dep_set_task);
 
Signature · 1 line
2.26.2
author maintainer
#2 lore raw
Quoted 10 lines · Paul E. McKenney
Quoted 9 lines · Frederic Weisbecker

So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on nohz_full, a combo that nobody seemed to use in real life.

But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on nohz_full.

Quoted 7 lines
Quoted 2 lines

The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.

This passes moderate rcutorture testing. If you want me to take it, please let me know, and otherwise:

If you already have a pending urgent queue, I'd love you to take it. If not I can take it.

Thanks.

#3 lore raw
Quoted 14 lines · Frederic Weisbecker
Quoted 13 lines · Paul E. McKenney
Quoted 11 lines · Frederic Weisbecker

So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on nohz_full, a combo that nobody seemed to use in real life.

But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on nohz_full.

The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.

Quoted 7 lines
Quoted 4 lines

This passes moderate rcutorture testing. If you want me to take it, please let me know, and otherwise:

If you already have a pending urgent queue, I'd love you to take it. If not I can take it.

Nothing urgent yet in -rcu, so if you would like it in the next merge window, please take it through your normal upstream path.

Thanx, Paul

author maintainer
#4 lore raw
Quoted 18 lines · Paul E. McKenney
Quoted 17 lines · Frederic Weisbecker
Quoted 16 lines · Paul E. McKenney
Quoted 11 lines · Frederic Weisbecker

So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on nohz_full, a combo that nobody seemed to use in real life.

But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on nohz_full.

The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.

This passes moderate rcutorture testing. If you want me to take it, please let me know, and otherwise:

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

If you already have a pending urgent queue, I'd love you to take it. If not I can take it.

Nothing urgent yet in -rcu, so if you would like it in the next merge window, please take it through your normal upstream path.

Got it, thanks!

#5 lore raw
In reply to Frederic Weisbecker
Quoted 35 lines · Frederic Weisbecker

So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on nohz_full, a combo that nobody seemed to use in real life.

But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on nohz_full.

The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.

Fixes 6a949b7af82d (rcu: Force on tick when invoking lots of callbacks)
Cc Paul E. McKenney <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/time/tick-sched.c | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)
kernel/time/tick-sched.c +1 -2
index 3e2dc9b8858c..f0199a4ba1ad 100644
@@ -351,16 +351,24 @@ void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit)
 EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_cpu);
 
 /*
- * Set a per-task tick dependency. Posix CPU timers need this in order to elapse
- * per task timers.
+ * Set a per-task tick dependency. RCU need this. Also posix CPU timers

kernel/time/tick-sched.c void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) · line 359

Quoted 10 lines
kernel/time/tick-sched.c +2 -5
+ * in order to elapse per task timers.
  */
 void tick_nohz_dep_set_task(struct task_struct *tsk, enum tick_dep_bits bit)
 {
-	/*
-	 * We could optimize this with just kicking the target running the task
-	 * if that noise matters for nohz full users.
-	 */
-	tick_nohz_dep_set_all(&tsk->tick_dep_mask, bit);
+	if (!atomic_fetch_or(BIT(bit), &tsk->tick_dep_mask)) {

So why not simply:

		tick_nohz_full_kick_cpu(task_cpu(tsk)); ?

If it got preempted, the scheduling involved would already have observed the bit we just set and kept the tick on anyway, same for migration.

Or am I missing something?

Quoted 12 lines
kernel/time/tick-sched.c +12 -0
+		if (tsk == current) {
+			preempt_disable();
+			tick_nohz_full_kick();
+			preempt_enable();
+		} else {
+			/*
+			 * Some future tick_nohz_full_kick_task()
+			 * should optimize this.
+			 */
+			tick_nohz_full_kick_all();
+		}
+	}
author maintainer
#6 lore raw
Quoted 49 lines · Peter Zijlstra
Quoted 35 lines · Frederic Weisbecker

So far setting a tick dependency on any task, including current, used to trigger an IPI to all CPUs. That's of course suboptimal but it wasn't an issue as long as it was only used by posix-cpu-timers on nohz_full, a combo that nobody seemed to use in real life.

But RCU started to use task tick dependency on current task to fix stall issues on callbacks processing. These trigger regular and undesired system wide IPIs on nohz_full.

The fix is very easy while setting a tick dependency on the current task, only its CPU needs an IPI.

Fixes 6a949b7af82d (rcu: Force on tick when invoking lots of callbacks)
Cc Paul E. McKenney <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/time/tick-sched.c | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)
kernel/time/tick-sched.c +1 -2
index 3e2dc9b8858c..f0199a4ba1ad 100644
@@ -351,16 +351,24 @@ void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit)
 EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_cpu);
 
 /*
- * Set a per-task tick dependency. Posix CPU timers need this in order to elapse
- * per task timers.
+ * Set a per-task tick dependency. RCU need this. Also posix CPU timers

kernel/time/tick-sched.c void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) · line 359

Quoted 10 lines
kernel/time/tick-sched.c +2 -5
+ * in order to elapse per task timers.
  */
 void tick_nohz_dep_set_task(struct task_struct *tsk, enum tick_dep_bits bit)
 {
-	/*
-	 * We could optimize this with just kicking the target running the task
-	 * if that noise matters for nohz full users.
-	 */
-	tick_nohz_dep_set_all(&tsk->tick_dep_mask, bit);
+	if (!atomic_fetch_or(BIT(bit), &tsk->tick_dep_mask)) {

So why not simply:

Quoted 6 lines
		tick_nohz_full_kick_cpu(task_cpu(tsk)); ?

If it got preempted, the scheduling involved would already have observed the bit we just set and kept the tick on anyway, same for migration.

Or am I missing something?

Hmm, I guess we would then need some sort of ordering like this:

         CPU 0                            CPU 1
         -----                            -----
         p->cpu = smp_processor_id()      atomic_fetch_or(bit, p->tick_dep_mask)
         smp_mb();                        smp_mb(); //actually implied by atomic_fetch_or()
         READ p->tick_dep_mask            irq_work_on(...., p->cpu)

And since p->cpu is already set and visible on context switch, it should work indeed. Now in the case CPU 1 reads a stale task_cpu(), that's fine as CPU 0 sees the new tick_dep_mask, but CPU 1 might be dealing with an offlined CPU, right?

So I guess I should still protect against hotplug with cpus_read_lock() but tick_nohz_dep_set_task() isn't supposed to sleep...

Or preempt_disable() could help us with that somehow? I'm always confused with the guarantees that disabled preemption can offer toward hotplug.

Quoted 14 lines
Quoted 13 lines
kernel/time/tick-sched.c +12 -0
+		if (tsk == current) {
+			preempt_disable();
+			tick_nohz_full_kick();
+			preempt_enable();
+		} else {
+			/*
+			 * Some future tick_nohz_full_kick_task()
+			 * should optimize this.
+			 */
+			tick_nohz_full_kick_all();
+		}
+	}