nohz_full: only wakeup target CPUs when notifying new tick dependency (v2)
Subsystems (2)
PATCH 2/2 nohz: change signal tick dependency to wakeup CPUs of member tasks
In reply to Marcelo TosattiRather than waking up all nohz_full CPUs on the system, only wakeup the target CPUs of member threads of the signal.
Reduces interruptions to nohz_full CPUs.
kernel/time/tick-sched.c +9 -1
@@ -398,7 +398,15 @@ EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_ta*/ void tick_nohz_dep_set_signal (struct signal_struct *sig,enum tick_dep_bits bit) { -tick_nohz_dep_set_all (&sig->tick_dep_mask, bit); +int prev; + + prev =atomic_fetch_or (BIT (bit), &sig->tick_dep_mask); +if (!prev) { +rcu_read_lock (); +for_each_thread (sig, t) +tick_nohz_kick_task (t); +rcu_read_unlock (); + } }void tick_nohz_dep_clear_signal (struct signal_struct *sig,enum tick_dep_bits bit)
cover nohz_full: only wakeup target CPUs when notifying new tick dependency (v2)
When enabling per-CPU posix timers, an IPI to nohz_full CPUs might be performed (to re-read the dependencies and possibly not re-enter nohz_full on a given CPU).
A common case is for applications that run on nohz_full= CPUs to not use POSIX timers (eg DPDK). This patch changes the notification to only IPI the target CPUs where the task(s) whose tick dependencies are being updated are executing.
This reduces interruptions to nohz_full= CPUs.
PATCH 1/2 nohz: only wakeup a single target cpu when kicking a task
When adding a tick dependency to a task, its necessary to wakeup the CPU where the task resides to reevaluate tick dependencies on that CPU.
However the current code wakes up all nohz_full CPUs, which is unnecessary.
Switch to waking up a single CPU, by using ordering of writes to task->cpu and task->tick_dep_mask.
From: Frederic Weisbecker <[email protected]>
kernel/time/tick-sched.c +27 -13
@@ -274,6 +274,31 @@ void tick_nohz_full_kick_cpu(int cpu)irq_work_queue_on (&per_cpu (nohz_full_kick_work, cpu), cpu); } +static void tick_nohz_kick_task (struct task_struct *tsk) +{ +int cpu =task_cpu (tsk); + +/* +* If the task concurrently migrates to another cpu, +* we guarantee it sees the new tick dependency upon +* schedule. +* +* +* set_task_cpu(p, cpu); +* STORE p->cpu = @cpu +* __schedule() (switch to task 'p') +* LOCK rq->lock +* smp_mb__after_spin_lock() STORE p->tick_dep_mask +* tick_nohz_task_switch() smp_mb() (atomic_fetch_or()) +* LOAD p->tick_dep_mask LOAD p->cpu +*/ + +preempt_disable (); +if (cpu_online (cpu)) +tick_nohz_full_kick_cpu (cpu); +preempt_enable (); +} +/* * Kick all full dynticks CPUs in order to force these to re-evaluate * their dependency on the tick and restart it if necessary.
@@ -356,19 +381,8 @@ EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_cp*/ void tick_nohz_dep_set_task (struct task_struct *tsk,enum tick_dep_bits bit) { -if (!atomic_fetch_or (BIT (bit), &tsk->tick_dep_mask)) { -if (tsk == current) { -preempt_disable (); -tick_nohz_full_kick (); -preempt_enable (); - }else { -/* -* Some future tick_nohz_full_kick_task() -* should optimize this. -*/ -tick_nohz_full_kick_all (); - } - } +if (!atomic_fetch_or (BIT (bit), &tsk->tick_dep_mask)) +tick_nohz_kick_task (tsk); }EXPORT_SYMBOL_GPL (tick_nohz_dep_set_task);
Peter Zijlstra
maintainerRe: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
Quoted 37 lines · Marcelo Tosatti
When adding a tick dependency to a task, its necessary to wakeup the CPU where the task resides to reevaluate tick dependencies on that CPU.
However the current code wakes up all
nohz_fullCPUs, which is unnecessary.Switch to waking up a single CPU, by using ordering of writes to
task->cpuandtask->tick_dep_mask.From: Frederic Weisbecker <[email protected]>
Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Signed-off-by Marcelo Tosatti <[email protected]>Index:
linux-2.6/kernel/time/tick-sched.c
===================================================================kernel/time/tick-sched.c +14
-0@@ -274,6 +274,31 @@ void tick_nohz_full_kick_cpu(int cpu)irq_work_queue_on (&per_cpu (nohz_full_kick_work, cpu), cpu); } +static void tick_nohz_kick_task (struct task_struct *tsk) +{ +int cpu =task_cpu (tsk); + +/* +* If the task concurrently migrates to another cpu, +* we guarantee it sees the new tick dependency upon +* schedule. +* +* +* set_task_cpu(p, cpu); +* STORE p->cpu = @cpu +* __schedule() (switch to task 'p') +* LOCK rq->lock
kernel/time/tick-sched.c void tick_nohz_full_kick_cpu(int cpu) · line 300
Quoted 10 lines
kernel/time/tick-sched.c +10
-0+* smp_mb__after_spin_lock() STORE p->tick_dep_mask +* tick_nohz_task_switch() smp_mb() (atomic_fetch_or()) +* LOAD p->tick_dep_mask LOAD p->cpu +*/ + +preempt_disable (); +if (cpu_online (cpu)) +tick_nohz_full_kick_cpu (cpu); +preempt_enable (); +}
So we need to kick the CPU unconditionally, or only when the task is actually running? AFAICT we only care about current->tick_dep_mask.
Peter Zijlstra
maintainerRe: [patch 2/2] nohz: change signal tick dependency to wakeup CPUs of member tasks
In reply to Marcelo TosattiQuoted 16 lines · Marcelo Tosatti
Rather than waking up all
nohz_fullCPUs on the system, only wakeup the target CPUs of member threads of the signal.Reduces interruptions to
nohz_fullCPUs.Signed-off-by Marcelo Tosatti <[email protected]>Index:
linux-2.6/kernel/time/tick-sched.c
===================================================================kernel/time/tick-sched.c +0
-1@@ -398,7 +398,15 @@ EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_ta*/ void tick_nohz_dep_set_signal (struct signal_struct *sig,enum tick_dep_bits bit) { -tick_nohz_dep_set_all (&sig->tick_dep_mask, bit);
kernel/time/tick-sched.c EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_ta · line 410
Quoted 10 lines
kernel/time/tick-sched.c +9
-0+int prev; + + prev =atomic_fetch_or (BIT (bit), &sig->tick_dep_mask); +if (!prev) { +rcu_read_lock (); +for_each_thread (sig, t) +tick_nohz_kick_task (t); +rcu_read_unlock (); + } }
AFAICT, and this makes perfect sense, this function is only ever used while holding sighand->siglock, which makes the RCU read lock superfluous.
Would it make sense to change the signal_struct argument to task_struct, such that we can write:
lockdep_assert_held(&p->sighand->siglock); for_each_thread(p->signal, t) tick_nohz_kick_task(t);
?
Re: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
In reply to Marcelo TosattiQuoted 10 lines · Marcelo Tosatti
kernel/time/tick-sched.c +10
-0+static void tick_nohz_kick_task (struct task_struct *tsk) +{ +int cpu =task_cpu (tsk); + +/* +* If the task concurrently migrates to another cpu, +* we guarantee it sees the new tick dependency upon +* schedule. +* +*
kernel/time/tick-sched.c void tick_nohz_full_kick_cpu(int cpu) · line 296
Quoted 10 lines
kernel/time/tick-sched.c +10
-0+* set_task_cpu(p, cpu); +* STORE p->cpu = @cpu +* __schedule() (switch to task 'p') +* LOCK rq->lock +* smp_mb__after_spin_lock() STORE p->tick_dep_mask +* tick_nohz_task_switch() smp_mb() (atomic_fetch_or()) +* LOAD p->tick_dep_mask LOAD p->cpu +*/ + +preempt_disable ();
Pure question: is preempt_disable() required here? Same question to tick_nohz_full_kick_all().
Quoted 4 lines
kernel/time/tick-sched.c +4
-0+if (cpu_online (cpu)) +tick_nohz_full_kick_cpu (cpu); +preempt_enable (); +}
Signature · 1 line
Peter Xu
Peter Zijlstra
maintainerRe: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
Quoted 16 lines · Peter Xu
Quoted 15 lines · Marcelo Tosatti
kernel/time/tick-sched.c +15
-0+static void tick_nohz_kick_task (struct task_struct *tsk) +{ +int cpu =task_cpu (tsk); + +/* +* If the task concurrently migrates to another cpu, +* we guarantee it sees the new tick dependency upon +* schedule. +* +* +* set_task_cpu(p, cpu); +* STORE p->cpu = @cpu +* __schedule() (switch to task 'p') +* LOCK rq->lock +* smp_mb__after_spin_lock() STORE p->tick_dep_mask
Quoted 8 lines
kernel/time/tick-sched.c void tick_nohz_full_kick_cpu(int cpu) · line 296
Quoted 5 lines
kernel/time/tick-sched.c +5
-0+* tick_nohz_task_switch() smp_mb() (atomic_fetch_or()) +* LOAD p->tick_dep_mask LOAD p->cpu +*/ + +preempt_disable ();Pure question: is
preempt_disable()required here? Same question totick_nohz_full_kick_all().
I think it serializes against hotplug.
Quoted 5 lines
Quoted 5 lines
kernel/time/tick-sched.c +4
-0+if (cpu_online (cpu)) +tick_nohz_full_kick_cpu (cpu); +preempt_enable (); +}
Re: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
In reply to Peter ZijlstraQuoted 43 lines · Peter Zijlstra
Quoted 42 lines · Marcelo Tosatti
When adding a tick dependency to a task, its necessary to wakeup the CPU where the task resides to reevaluate tick dependencies on that CPU.
However the current code wakes up all
nohz_fullCPUs, which is unnecessary.Switch to waking up a single CPU, by using ordering of writes to
task->cpuandtask->tick_dep_mask.From: Frederic Weisbecker <[email protected]>
Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Signed-off-by Marcelo Tosatti <[email protected]>Index:
linux-2.6/kernel/time/tick-sched.c
===================================================================kernel/time/tick-sched.c +19
-0@@ -274,6 +274,31 @@ void tick_nohz_full_kick_cpu(int cpu)irq_work_queue_on (&per_cpu (nohz_full_kick_work, cpu), cpu); } +static void tick_nohz_kick_task (struct task_struct *tsk) +{ +int cpu =task_cpu (tsk); + +/* +* If the task concurrently migrates to another cpu, +* we guarantee it sees the new tick dependency upon +* schedule. +* +* +* set_task_cpu(p, cpu); +* STORE p->cpu = @cpu +* __schedule() (switch to task 'p') +* LOCK rq->lock +* smp_mb__after_spin_lock() STORE p->tick_dep_mask +* tick_nohz_task_switch() smp_mb() (atomic_fetch_or()) +* LOAD p->tick_dep_mask LOAD p->cpu +*/ +
Quoted 8 lines
kernel/time/tick-sched.c void tick_nohz_full_kick_cpu(int cpu) · line 300
Quoted 5 lines
kernel/time/tick-sched.c +5
-0+preempt_disable (); +if (cpu_online (cpu)) +tick_nohz_full_kick_cpu (cpu); +preempt_enable (); +}So we need to kick the CPU unconditionally, or only when the task is actually running? AFAICT we only care about
current->tick_dep_mask.
tick is necessary to execute run_posix_cpu_timers, from tick interrupt, even if task is not running.
Re: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
In reply to Peter XuQuoted 16 lines · Peter Xu
Quoted 15 lines · Marcelo Tosatti
kernel/time/tick-sched.c +15
-0+static void tick_nohz_kick_task (struct task_struct *tsk) +{ +int cpu =task_cpu (tsk); + +/* +* If the task concurrently migrates to another cpu, +* we guarantee it sees the new tick dependency upon +* schedule. +* +* +* set_task_cpu(p, cpu); +* STORE p->cpu = @cpu +* __schedule() (switch to task 'p') +* LOCK rq->lock +* smp_mb__after_spin_lock() STORE p->tick_dep_mask
Quoted 8 lines
kernel/time/tick-sched.c void tick_nohz_full_kick_cpu(int cpu) · line 296
Quoted 5 lines
kernel/time/tick-sched.c +5
-0+* tick_nohz_task_switch() smp_mb() (atomic_fetch_or()) +* LOAD p->tick_dep_mask LOAD p->cpu +*/ + +preempt_disable ();Pure question: is
preempt_disable()required here? Same question totick_nohz_full_kick_all().
Hi Peter,
Don't see why: irq_queue_work_on() disables preemption if necessary.
Quoted 8 lines
Quoted 5 lines
kernel/time/tick-sched.c +4
-0+if (cpu_online (cpu)) +tick_nohz_full_kick_cpu (cpu); +preempt_enable (); +}Signature · 1 line
Peter Xu
Re: [patch 2/2] nohz: change signal tick dependency to wakeup CPUs of member tasks
In reply to Peter ZijlstraQuoted 32 lines · Peter Zijlstra
Quoted 16 lines · Marcelo Tosatti
Rather than waking up all
nohz_fullCPUs on the system, only wakeup the target CPUs of member threads of the signal.Reduces interruptions to
nohz_fullCPUs.Signed-off-by Marcelo Tosatti <[email protected]>Index:
linux-2.6/kernel/time/tick-sched.c
===================================================================kernel/time/tick-sched.c +0
-1@@ -398,7 +398,15 @@ EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_ta*/ void tick_nohz_dep_set_signal (struct signal_struct *sig,enum tick_dep_bits bit) { -tick_nohz_dep_set_all (&sig->tick_dep_mask, bit);kernel/time/tick-sched.c EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_ta · line 410
Quoted 10 lines
kernel/time/tick-sched.c +9
-0+int prev; + + prev =atomic_fetch_or (BIT (bit), &sig->tick_dep_mask); +if (!prev) { +rcu_read_lock (); +for_each_thread (sig, t) +tick_nohz_kick_task (t); +rcu_read_unlock (); + } }AFAICT, and this makes perfect sense, this function is only ever used while holding
sighand->siglock, which makes the RCU read lock superfluous.
Quoted 8 lines
Would it make sense to change the
signal_structargument totask_struct, such that we can write:lockdep_assert_held(&p->sighand->siglock); for_each_thread(p->signal, t) tick_nohz_kick_task(t);?
Makes sense, resending -v3.
Re: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
In reply to Peter ZijlstraQuoted 26 lines · Peter Zijlstra
Quoted 16 lines · Peter Xu
Quoted 15 lines · Marcelo Tosatti
kernel/time/tick-sched.c +15
-0+static void tick_nohz_kick_task (struct task_struct *tsk) +{ +int cpu =task_cpu (tsk); + +/* +* If the task concurrently migrates to another cpu, +* we guarantee it sees the new tick dependency upon +* schedule. +* +* +* set_task_cpu(p, cpu); +* STORE p->cpu = @cpu +* __schedule() (switch to task 'p') +* LOCK rq->lock +* smp_mb__after_spin_lock() STORE p->tick_dep_mask Quoted 8 lines
kernel/time/tick-sched.c void tick_nohz_full_kick_cpu(int cpu) · line 296
Quoted 5 lines
kernel/time/tick-sched.c +5
-0+* tick_nohz_task_switch() smp_mb() (atomic_fetch_or()) +* LOAD p->tick_dep_mask LOAD p->cpu +*/ + +preempt_disable ();Pure question: is
preempt_disable()required here? Same question totick_nohz_full_kick_all().
Quoted 1 line
I think it serializes against hotplug.
Thanks Peter. So is that a lighter but trickier version of get_online_cpus() which is even allowed with spinlock?
I noticed that this method was actually mentioned in the old cpu-hotplug.txt, but it was removed during the convertion to rst:
ff58fa7f556c ("Documentation: Update CPU hotplug and move it to core-api")
Not sure whether it's intended, just to raise this up. If it's still valid, maybe still worth to document it somewhere.
Signature · 1 line
Peter Xu
Frederic Weisbecker
maintainerRe: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
In reply to Peter ZijlstraQuoted 26 lines · Peter Zijlstra
Quoted 16 lines · Peter Xu
Quoted 15 lines · Marcelo Tosatti
kernel/time/tick-sched.c +15
-0+static void tick_nohz_kick_task (struct task_struct *tsk) +{ +int cpu =task_cpu (tsk); + +/* +* If the task concurrently migrates to another cpu, +* we guarantee it sees the new tick dependency upon +* schedule. +* +* +* set_task_cpu(p, cpu); +* STORE p->cpu = @cpu +* __schedule() (switch to task 'p') +* LOCK rq->lock +* smp_mb__after_spin_lock() STORE p->tick_dep_mask Quoted 8 lines
kernel/time/tick-sched.c void tick_nohz_full_kick_cpu(int cpu) · line 296
Quoted 5 lines
kernel/time/tick-sched.c +5
-0+* tick_nohz_task_switch() smp_mb() (atomic_fetch_or()) +* LOAD p->tick_dep_mask LOAD p->cpu +*/ + +preempt_disable ();Pure question: is
preempt_disable()required here? Same question totick_nohz_full_kick_all().
Quoted 1 line
I think it serializes against hotplug.
Exactly!
Thanks.
Frederic Weisbecker
maintainerRe: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
In reply to Marcelo TosattiQuoted 53 lines · Marcelo Tosatti
Quoted 43 lines · Peter Zijlstra
Quoted 42 lines · Marcelo Tosatti
When adding a tick dependency to a task, its necessary to wakeup the CPU where the task resides to reevaluate tick dependencies on that CPU.
However the current code wakes up all
nohz_fullCPUs, which is unnecessary.Switch to waking up a single CPU, by using ordering of writes to
task->cpuandtask->tick_dep_mask.From: Frederic Weisbecker <[email protected]>
Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Signed-off-by Marcelo Tosatti <[email protected]>Index:
linux-2.6/kernel/time/tick-sched.c
===================================================================kernel/time/tick-sched.c +19
-0@@ -274,6 +274,31 @@ void tick_nohz_full_kick_cpu(int cpu)irq_work_queue_on (&per_cpu (nohz_full_kick_work, cpu), cpu); } +static void tick_nohz_kick_task (struct task_struct *tsk) +{ +int cpu =task_cpu (tsk); + +/* +* If the task concurrently migrates to another cpu, +* we guarantee it sees the new tick dependency upon +* schedule. +* +* +* set_task_cpu(p, cpu); +* STORE p->cpu = @cpu +* __schedule() (switch to task 'p') +* LOCK rq->lock +* smp_mb__after_spin_lock() STORE p->tick_dep_mask +* tick_nohz_task_switch() smp_mb() (atomic_fetch_or()) +* LOAD p->tick_dep_mask LOAD p->cpu +*/ +Quoted 8 lines
kernel/time/tick-sched.c void tick_nohz_full_kick_cpu(int cpu) · line 300
Quoted 5 lines
kernel/time/tick-sched.c +5
-0+preempt_disable (); +if (cpu_online (cpu)) +tick_nohz_full_kick_cpu (cpu); +preempt_enable (); +}So we need to kick the CPU unconditionally, or only when the task is actually running? AFAICT we only care about
current->tick_dep_mask.
Quoted 2 lines
tick is necessary to execute
run_posix_cpu_timers, from tick interrupt, even if task is not running.
Yes but if the task isn't running, run_posix_cpu_timers() doesn't have anything to elapse. So indeed we can spare the IPI if the task is not running. Provided ordering makes sure that the task sees the new dependency when it schedules in of course.
Re: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
Quoted 54 lines · Frederic Weisbecker
Quoted 53 lines · Marcelo Tosatti
Quoted 51 lines · Peter Zijlstra
Quoted 37 lines · Marcelo Tosatti
When adding a tick dependency to a task, its necessary to wakeup the CPU where the task resides to reevaluate tick dependencies on that CPU.
However the current code wakes up all
nohz_fullCPUs, which is unnecessary.Switch to waking up a single CPU, by using ordering of writes to
task->cpuandtask->tick_dep_mask.From: Frederic Weisbecker <[email protected]>
Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Signed-off-by Marcelo Tosatti <[email protected]>Index:
linux-2.6/kernel/time/tick-sched.c
===================================================================kernel/time/tick-sched.c +14
-0@@ -274,6 +274,31 @@ void tick_nohz_full_kick_cpu(int cpu)irq_work_queue_on (&per_cpu (nohz_full_kick_work, cpu), cpu); } +static void tick_nohz_kick_task (struct task_struct *tsk) +{ +int cpu =task_cpu (tsk); + +/* +* If the task concurrently migrates to another cpu, +* we guarantee it sees the new tick dependency upon +* schedule. +* +* +* set_task_cpu(p, cpu); +* STORE p->cpu = @cpu +* __schedule() (switch to task 'p') +* LOCK rq->lock kernel/time/tick-sched.c void tick_nohz_full_kick_cpu(int cpu) · line 300
Quoted 10 lines
kernel/time/tick-sched.c +10
-0+* smp_mb__after_spin_lock() STORE p->tick_dep_mask +* tick_nohz_task_switch() smp_mb() (atomic_fetch_or()) +* LOAD p->tick_dep_mask LOAD p->cpu +*/ + +preempt_disable (); +if (cpu_online (cpu)) +tick_nohz_full_kick_cpu (cpu); +preempt_enable (); +}So we need to kick the CPU unconditionally, or only when the task is actually running? AFAICT we only care about
current->tick_dep_mask.
Quoted 7 lines
Quoted 2 lines
tick is necessary to execute
run_posix_cpu_timers, from tick interrupt, even if task is not running.Yes but if the task isn't running,
run_posix_cpu_timers()doesn't have anything to elapse. So indeed we can spare the IPI if the task is not running. Provided ordering makes sure that the task sees the new dependency when it schedules in of course.
True.
* p->on_cpu <- { 0, 1 }:
*
* is set by prepare_task() and cleared by finish_task() such that it will be * set before p is scheduled-in and cleared after p is scheduled-out, both * under rq->lock. Non-zero indicates the task is running on its CPU.
CPU-0 (tick_set_dep) CPU-1 (task switch)
STORE p->tick_dep_masksmp_mb() (atomic_fetch_or())
LOAD p->on_cpu
context_switch(prev, next)
STORE next->on_cpu = 1
... [*]
LOAD current->tick_dep_maskDon't see any explicit memory barrier in the [*] section?
Peter Zijlstra
maintainerRe: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
Quoted 21 lines · Marcelo Tosatti
Quoted 4 lines
Yes but if the task isn't running,
run_posix_cpu_timers()doesn't have anything to elapse. So indeed we can spare the IPI if the task is not running. Provided ordering makes sure that the task sees the new dependency when it schedules in of course.True.
*
p->on_cpu<- { 0, 1 }:
** is set by prepare_task() and cleared by finish_task() such that it will be * set before p is scheduled-in and cleared after p is scheduled-out, both * under rq->lock. Non-zero indicates the task is running on its CPU.CPU-0 (
tick_set_dep) CPU-1 (task switch)STORE
p->tick_dep_masksmp_mb()(atomic_fetch_or())
LOADp->on_cpu
Quoted 6 lines
context_switch(prev, next)
STORE next->on_cpu = 1
... [*]
LOAD current->tick_dep_maskThat load is in tick_nohz_task_switch() right? (which BTW is placed completely wrong) You could easily do something like the below I suppose.
kernel/time/tick-sched.c +8 -0
@@ -410,6 +410,14 @@ void __tick_nohz_task_switch(void) ts =this_cpu_ptr (&tick_cpu_sched);if (ts->tick_stopped) { +/* +* tick_set_dep() (this) +* +* STORE p->tick_dep_mask STORE p->on_cpu +* smp_mb() smp_mb() +* LOAD p->on_cpu LOAD p->tick_dep_mask +*/ +smp_mb ();if (atomic_read (¤t->tick_dep_mask) ||atomic_read (¤t->signal->tick_dep_mask))tick_nohz_full_kick ();
re tick_nohz_task_switch() being placed wrong, it should probably be placed before finish_lock_switch(). Something like so.
kernel/sched/core.c +1 -1
@@ -4084,6 +4084,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)vtime_task_switch (prev);perf_event_task_sched_in (prev, current);finish_task (prev); +tick_nohz_task_switch ();finish_lock_switch (rq);finish_arch_post_lock_switch ();kcov_finish_switch (current);
@@ -4121,7 +4122,6 @@ static struct rq *finish_task_switch(struct task_struct *prev)put_task_struct_rcu_user (prev); } -tick_nohz_task_switch ();return rq; }
kernel/time/tick-sched.c +1 -5
@@ -402,10 +402,8 @@ void __tick_nohz_task_switch(void)unsigned long flags;struct tick_sched *ts; -local_irq_save (flags); -if (!tick_nohz_full_cpu (smp_processor_id ())) - goto out; +return ; ts =this_cpu_ptr (&tick_cpu_sched);
@@ -414,8 +412,6 @@ void __tick_nohz_task_switch(void) atomic_read(¤t->signal->tick_dep_mask))tick_nohz_full_kick (); } -out: -local_irq_restore (flags); }/* Get the boot-time nohz CPU list from the kernel parameters. */
Frederic Weisbecker
maintainerRe: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
Quoted 43 lines · Peter Zijlstra
Quoted 21 lines · Marcelo Tosatti
Quoted 4 lines
Yes but if the task isn't running,
run_posix_cpu_timers()doesn't have anything to elapse. So indeed we can spare the IPI if the task is not running. Provided ordering makes sure that the task sees the new dependency when it schedules in of course.True.
*
p->on_cpu<- { 0, 1 }:
** is set by prepare_task() and cleared by finish_task() such that it will be * set before p is scheduled-in and cleared after p is scheduled-out, both * under rq->lock. Non-zero indicates the task is running on its CPU.CPU-0 (
tick_set_dep) CPU-1 (task switch)STORE
p->tick_dep_masksmp_mb()(atomic_fetch_or())
LOADp->on_cpuQuoted 6 lines
context_switch(prev, next) STORE next->on_cpu = 1 ... [*] LOAD current->tick_dep_maskThat load is in
tick_nohz_task_switch()right? (which BTW is placed completely wrong) You could easily do something like the below I suppose.kernel/time/tick-sched.c +1
-0@@ -410,6 +410,14 @@ void __tick_nohz_task_switch(void) ts =this_cpu_ptr (&tick_cpu_sched);if (ts->tick_stopped) { +/*
kernel/time/tick-sched.c void __tick_nohz_task_switch(void) · line 423
Quoted 10 lines
kernel/time/tick-sched.c +7
-0+* tick_set_dep() (this) +* +* STORE p->tick_dep_mask STORE p->on_cpu +* smp_mb() smp_mb() +* LOAD p->on_cpu LOAD p->tick_dep_mask +*/ +smp_mb ();if (atomic_read (¤t->tick_dep_mask) ||atomic_read (¤t->signal->tick_dep_mask))tick_nohz_full_kick ();
It would then need to be unconditional (whatever value of ts->tick_stopped). Assuming the tick isn't stopped, we may well have an interrupt firing right after schedule() which doesn't see the new value of tick_dep_map.
Alternatively, we could rely on p->on_rq which is set to TASK_ON_RQ_QUEUED at wake up time, prior to the schedule() full barrier. Of course that doesn't mean that the task is actually the one running on the CPU but it's a good sign, considering that we are running in nohz_full mode and it's usually optimized for single task mode.
Also setting a remote task's tick dependency is only used by posix cpu timer in case the user has the bad taste to enqueue on a task running in nohz_full mode. It shouldn't deserve an unconditional full barrier in the schedule path.
If the target is current, as is used by RCU, I guess we can keep a special treatment.
Quoted 17 lines
re
tick_nohz_task_switch()being placed wrong, it should probably be placed beforefinish_lock_switch(). Something like so.kernel/sched/core.c +1
-0@@ -4084,6 +4084,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)vtime_task_switch (prev);perf_event_task_sched_in (prev, current);finish_task (prev); +tick_nohz_task_switch ();finish_lock_switch (rq);finish_arch_post_lock_switch ();kcov_finish_switch (current);
kernel/sched/core.c static struct rq *finish_task_switch(struct task_struct *prev) · line 4124
Quoted 4 lines
kernel/sched/core.c +0
-1@@ -4121,7 +4122,6 @@ static struct rq *finish_task_switch(struct task_struct *prev)put_task_struct_rcu_user (prev); } -tick_nohz_task_switch ();
IIRC, we wanted to keep it outside rq lock because it shouldn't it...
Peter Zijlstra
maintainerRe: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
Quoted 20 lines · Frederic Weisbecker
Quoted 20 lines
re
tick_nohz_task_switch()being placed wrong, it should probably be placed beforefinish_lock_switch(). Something like so.kernel/sched/core.c +1
-0@@ -4084,6 +4084,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)vtime_task_switch (prev);perf_event_task_sched_in (prev, current);finish_task (prev); +tick_nohz_task_switch ();finish_lock_switch (rq);finish_arch_post_lock_switch ();kcov_finish_switch (current);@@ -4121,7 +4122,6 @@ static struct rq *finish_task_switch(struct task_struct *prev)put_task_struct_rcu_user (prev); }
Quoted 3 lines
kernel/sched/core.c static struct rq *finish_task_switch(struct task_struct *prev) · line 4124
Quoted 1 line
kernel/sched/core.c +0
-1-tick_nohz_task_switch ();IIRC, we wanted to keep it outside rq lock because it shouldn't it...
But now you've created a window with IRQs on and cause additional IRQ state changes.
If you're really worried about rq->lock, I suppose we can do:
rq_unlock(rq->lock); tick_nohz_task_switch(); local_irq_enable();
(much like we do at the beginning of __schedule for RCU)
Re: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
In reply to Frederic WeisbeckerQuoted 59 lines · Frederic Weisbecker
Quoted 43 lines · Peter Zijlstra
Quoted 21 lines · Marcelo Tosatti
Quoted 4 lines
Yes but if the task isn't running,
run_posix_cpu_timers()doesn't have anything to elapse. So indeed we can spare the IPI if the task is not running. Provided ordering makes sure that the task sees the new dependency when it schedules in of course.True.
*
p->on_cpu<- { 0, 1 }:
** is set by prepare_task() and cleared by finish_task() such that it will be * set before p is scheduled-in and cleared after p is scheduled-out, both * under rq->lock. Non-zero indicates the task is running on its CPU.CPU-0 (
tick_set_dep) CPU-1 (task switch)STORE
p->tick_dep_masksmp_mb()(atomic_fetch_or())
LOADp->on_cpuQuoted 6 lines
context_switch(prev, next) STORE next->on_cpu = 1 ... [*] LOAD current->tick_dep_maskThat load is in
tick_nohz_task_switch()right? (which BTW is placed completely wrong) You could easily do something like the below I suppose.kernel/time/tick-sched.c +1
-0@@ -410,6 +410,14 @@ void __tick_nohz_task_switch(void) ts =this_cpu_ptr (&tick_cpu_sched);if (ts->tick_stopped) { +/* kernel/time/tick-sched.c void __tick_nohz_task_switch(void) · line 423
Quoted 10 lines
kernel/time/tick-sched.c +7
-0+* tick_set_dep() (this) +* +* STORE p->tick_dep_mask STORE p->on_cpu +* smp_mb() smp_mb() +* LOAD p->on_cpu LOAD p->tick_dep_mask +*/ +smp_mb ();if (atomic_read (¤t->tick_dep_mask) ||atomic_read (¤t->signal->tick_dep_mask))tick_nohz_full_kick ();It would then need to be unconditional (whatever value of
ts->tick_stopped). Assuming the tick isn't stopped, we may well have an interrupt firing right afterschedule()which doesn't see the new value oftick_dep_map.
Quoted 5 lines
Alternatively, we could rely on
p->on_rqwhich is set toTASK_ON_RQ_QUEUEDat wake up time, prior to theschedule()full barrier. Of course that doesn't mean that the task is actually the one running on the CPU but it's a good sign, considering that we are running innohz_fullmode and it's usually optimized for single task mode.
Unfortunately that would require exporting p->on_rq which is internal to the scheduler, locklessly.
(can surely do that if you prefer!)
Quoted 7 lines
Also setting a remote task's tick dependency is only used by posix cpu timer in case the user has the bad taste to enqueue on a task running in
nohz_fullmode. It shouldn't deserve an unconditional full barrier in the schedule path.If the target is current, as is used by RCU, I guess we can keep a special treatment.
To answer PeterZ's original question:
"So we need to kick the CPU unconditionally, or only when the task is actually running? AFAICT we only care about current->tick_dep_mask."
If there is a task sharing signals, executing on a remote CPU, yes that remote CPU should be awakened.
Could skip the IPI if the remote process is not running, however for the purposes of low latency isolated processes, this optimization is not necessary.
So the last posted patchset is good enough for isolated low latency processes.
Do you guys want me to do something or can you take the series as it is?
Quoted 23 lines
Quoted 17 lines
re
tick_nohz_task_switch()being placed wrong, it should probably be placed beforefinish_lock_switch(). Something like so.kernel/sched/core.c +1
-0@@ -4084,6 +4084,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)vtime_task_switch (prev);perf_event_task_sched_in (prev, current);finish_task (prev); +tick_nohz_task_switch ();finish_lock_switch (rq);finish_arch_post_lock_switch ();kcov_finish_switch (current);kernel/sched/core.c static struct rq *finish_task_switch(struct task_struct *prev) · line 4124
Quoted 4 lines
kernel/sched/core.c +0
-1@@ -4121,7 +4122,6 @@ static struct rq *finish_task_switch(struct task_struct *prev)put_task_struct_rcu_user (prev); } -tick_nohz_task_switch ();IIRC, we wanted to keep it outside rq lock because it shouldn't it...
Frederic Weisbecker
maintainerRe: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
Quoted 11 lines · Marcelo Tosatti
Quoted 5 lines · Frederic Weisbecker
Alternatively, we could rely on
p->on_rqwhich is set toTASK_ON_RQ_QUEUEDat wake up time, prior to theschedule()full barrier. Of course that doesn't mean that the task is actually the one running on the CPU but it's a good sign, considering that we are running innohz_fullmode and it's usually optimized for single task mode.Unfortunately that would require exporting
p->on_rqwhich is internal to the scheduler, locklessly.(can surely do that if you prefer!)
May be:
bool task_on_rq(struct task_struct *p) ?
Thanks.
Frederic Weisbecker
maintainerRe: [patch 1/2] nohz: only wakeup a single target cpu when kicking a task
In reply to Peter ZijlstraQuoted 28 lines · Peter Zijlstra
Quoted 20 lines · Frederic Weisbecker
Quoted 20 lines
re
tick_nohz_task_switch()being placed wrong, it should probably be placed beforefinish_lock_switch(). Something like so.kernel/sched/core.c +1
-0@@ -4084,6 +4084,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)vtime_task_switch (prev);perf_event_task_sched_in (prev, current);finish_task (prev); +tick_nohz_task_switch ();finish_lock_switch (rq);finish_arch_post_lock_switch ();kcov_finish_switch (current);@@ -4121,7 +4122,6 @@ static struct rq *finish_task_switch(struct task_struct *prev)put_task_struct_rcu_user (prev); }Quoted 3 lines
kernel/sched/core.c static struct rq *finish_task_switch(struct task_struct *prev) · line 4124
Quoted 1 line
kernel/sched/core.c +0
-1-tick_nohz_task_switch ();IIRC, we wanted to keep it outside rq lock because it shouldn't it...
But now you've created a window with IRQs on and cause additional IRQ state changes.
Quoted 7 lines
If you're really worried about
rq->lock, I suppose we can do:rq_unlock(rq->lock); tick_nohz_task_switch(); local_irq_enable();(much like we do at the beginning of
__schedulefor RCU)
Right. Well I'm not that worried about rq->lock though. If you're ok with it I can just move it before finish_lock_switch().
Thanks.