nohz: Move nohz kick out of scheduler IPI
Subsystems (2)
cover nohz: Move nohz kick out of scheduler IPI
When a full dynticks CPU runs in single task mode then a new task gets enqueued, we notify it through an IPI such that it restarts its tick.
The IPI used here is the scheduler IPI. There are a few reasons for that:
it can be called when interrupts are disabled, it can be called concurrently... These properties altogether aren't offered by the IPI subsystem.
Meanwhile, bloating that way the scheduler IPI with scheduler unrelated code is an abuse of this fast path. We certainly don't want to start a big kernel IPI (BKI, the new shiny piece of my collection of big kernel things aside the big kernel tick, the big reiserfs lock, ..).
So this patchset adds a small helper to the IPI subsystem that allows to queue an IPI from interrupt disabled code and handles concurrent callers as well. Eventually the nohz kick gets converted to this new facility.
Partly inspired by a suggestion from Peter Zijlstra.
Comments?
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/ipi
Thanks,
Frederic
---
Frederic Weisbecker (3):
smp: Non busy-waiting IPI queue
nohz: Move full nohz kick to its own IPI
nohz: Use IPI implicit full barrier against rq->nr_running r/w
include/linux/smp.h | 12 ++++++++++++ include/linux/tick.h | 2 ++ kernel/sched/core.c | 14 ++++++-------- kernel/sched/sched.h | 12 +++++++++--- kernel/smp.c | 24 ++++++++++++++++++++++++ kernel/time/tick-sched.c | 20 ++++++++++++++++++++ 6 files changed, 73 insertions(+), 11 deletions(-)
PATCH 1/3 smp: Non busy-waiting IPI queue
Some IPI users, such as the nohz subsystem, need to be able to send an async IPI (async = non waiting for any other IPI completion) on contexts with disabled interrupts. And we want the IPI subsystem to handle concurrent calls by itself.
Currently the nohz machinery uses the scheduler IPI for this purpose because it can be triggered from any context and doesn't need any serialization from the caller. But this is an abuse of a scheduler fast path. We are bloating it with a job that should use its own IPI.
The current set of IPI functions can't be called when interrupts are disabled otherwise we risk a deadlock when two CPUs wait for each other's IPI completion.
OTOH smp_call_function_single_async() can be called when interrupts are disabled. But then it's up to the caller to serialize the given IPI. This can't be called concurrently without special care.
So we need a version of the async IPI that takes care of concurrent calls.
The proposed solution is to synchronize the IPI with a specific flag that prevents the IPI from being sent if it is already pending but not yet executed. Ordering is maintained such that, if the IPI is not sent because it's already pending, we guarantee it will see the new state of the data we expect it to when it will execute.
This model is close to the irq_work design. It's also partly inspired by suggestions from Peter Zijlstra.
--- include/linux/smp.h | 12 ++++++++++++ kernel/smp.c | 24 ++++++++++++++++++++++++ 2 files changed, 36 insertions(+)
include/linux/smp.h +12 -0
@@ -29,6 +29,18 @@ extern unsigned int total_cpus;int smp_call_function_single (int cpuid,smp_call_func_t func,void *info,int wait); +struct queue_single_data ; +typedef void (*smp_queue_func_t )(struct queue_single_data *qsd); + +struct queue_single_data { +struct call_single_data data; +smp_queue_func_t func; +int pending; +}; + +int smp_queue_function_single (int cpuid,smp_queue_func_t func, +struct queue_single_data *qsd); +/* * Call a function on all processors */
kernel/smp.c +24 -0
@@ -265,6 +265,30 @@ int smp_call_function_single_async(int cpu, struct call_single_data *csd) }EXPORT_SYMBOL_GPL (smp_call_function_single_async); +void generic_smp_queue_function_single_interrupt (void *info) +{ +struct queue_single_data *qsd = info; + +WARN_ON_ONCE (xchg (&qsd->pending,0 ) !=1 ); + qsd->func (qsd); +} + +int smp_queue_function_single (int cpu,smp_queue_func_t func,struct queue_single_data *qsd) +{ +int err; + +if (cmpxchg (&qsd->pending,0 ,1 )) +return 0 ; + + qsd->func = func; + +preempt_disable (); + err =generic_exec_single (cpu, &qsd->data, generic_smp_queue_function_single_interrupt, qsd,0 ); +preempt_enable (); + +return err; +} +/* * smp_call_function_any - Run a function on any of the given cpus * @mask: The mask of cpus it can run on.
Signature · 1 line
1.8.3.1
PATCH 2/3 nohz: Move full nohz kick to its own IPI
In reply to Frederic WeisbeckerNow that we have smp_queue_function_single() which can be used to safely queue IPIs when interrupts are disabled and without worrying about concurrent callers, lets use it for the full dynticks kick to notify a CPU that it's exiting single task mode.
This unbloats a bit the scheduler IPI that the nohz code was abusing for its cool "callable anywhere/anytime" properties.
--- include/linux/tick.h | 2 ++ kernel/sched/core.c | 5 +---- kernel/sched/sched.h | 2 +- kernel/time/tick-sched.c | 20 ++++++++++++++++++++ 4 files changed, 24 insertions(+), 5 deletions(-)
include/linux/tick.h +2 -0
@@ -182,6 +182,7 @@ static inline bool tick_nohz_full_cpu(int cpu)extern void tick_nohz_init (void );extern void __tick_nohz_full_check (void );extern void tick_nohz_full_kick (void ); +extern void tick_nohz_full_kick_cpu (int cpu);extern void tick_nohz_full_kick_all (void );extern void __tick_nohz_task_switch (struct task_struct *tsk);#else
@@ -190,6 +191,7 @@ static inline bool tick_nohz_full_enabled(void) { return false; }static inline bool tick_nohz_full_cpu (int cpu) {return false; }static inline void __tick_nohz_full_check (void ) { }static inline void tick_nohz_full_kick (void ) { } +static inline void tick_nohz_full_kick_cpu (int cpu) { }static inline void tick_nohz_full_kick_all (void ) { }static inline void __tick_nohz_task_switch (struct task_struct *tsk) { }#endif
kernel/sched/core.c +1 -4
@@ -1502,9 +1502,7 @@ void scheduler_ipi(void)*/ preempt_fold_need_resched (); -if (llist_empty (&this_rq ()->wake_list) - && !tick_nohz_full_cpu (smp_processor_id ()) - && !got_nohz_idle_kick ()) +if (llist_empty (&this_rq ()->wake_list) && !got_nohz_idle_kick ())return ;/*
@@ -1521,7 +1519,6 @@ void scheduler_ipi(void)* somewhat pessimize the simple resched case. */ irq_enter (); -tick_nohz_full_check ();sched_ttwu_pending ();/*
kernel/sched/sched.h +1 -1
@@ -1233,7 +1233,7 @@ static inline void inc_nr_running(struct rq *rq)if (tick_nohz_full_cpu (rq->cpu)) {/* Order rq->nr_running write against the IPI */ smp_wmb (); -smp_send_reschedule (rq->cpu); +tick_nohz_full_kick_cpu (rq->cpu); } }#endif
kernel/time/tick-sched.c +20 -0
@@ -230,6 +230,26 @@ void tick_nohz_full_kick(void)irq_work_queue (&__get_cpu_var (nohz_full_kick_work)); } +static DEFINE_PER_CPU(struct queue_single_data, nohz_full_kick_qsd); + +static void nohz_full_kick_queue (struct queue_single_data *qsd) +{ +__tick_nohz_full_check (); +} + +void tick_nohz_full_kick_cpu (int cpu) +{ +if (!tick_nohz_full_cpu (cpu)) +return ; + +if (cpu ==smp_processor_id ()) { +irq_work_queue (&__get_cpu_var (nohz_full_kick_work)); + }else { +smp_queue_function_single (cpu, nohz_full_kick_queue, + &per_cpu (nohz_full_kick_qsd, cpu)); + } +} +static void nohz_full_kick_ipi (void *info) {__tick_nohz_full_check ();
Signature · 1 line
1.8.3.1
PATCH 3/3 nohz: Use IPI implicit full barrier against rq->nr_running r/w
In reply to Frederic WeisbeckerA full dynticks CPU is allowed to stop its tick when a single task runs. Meanwhile when a new task gets enqueued, the CPU must be notified so that it restart its tick to maintain local fairness and other accounting details.
This notification is performed by way of an IPI. Then when the target receives the IPI, we expect it to see the new value of rq->nr_running.
Hence the following ordering scenario:
CPU 0 CPU 1
write rq->running get IPI smp_wmb() smp_rmb() send IPI read rq->nr_running
But Paul Mckenney says that nowadays IPIs imply a full barrier on all architectures. So we can safely remove this pair and rely on the implicit barriers that comes along IPI send/receive. Lets just comment on this new assumption.
--- kernel/sched/core.c | 9 +++++---- kernel/sched/sched.h | 10 ++++++++-- 2 files changed, 13 insertions(+), 6 deletions(-)
kernel/sched/core.c +5 -4
@@ -666,10 +666,11 @@ bool sched_can_stop_tick(void) rq =this_rq (); -/* Make sure rq->nr_running update is visible after the IPI */ -smp_rmb (); - -/* More than one running task need preemption */ +/* +* More than one running task need preemption. +* nr_running update is assumed to be visible +* after IPI is sent from wakers. +*/ if (rq->nr_running >1 )return false;
kernel/sched/sched.h +8 -2
@@ -1231,8 +1231,14 @@ static inline void inc_nr_running(struct rq *rq)#ifdef CONFIG_NO_HZ_FULL if (rq->nr_running ==2 ) {if (tick_nohz_full_cpu (rq->cpu)) { -/* Order rq->nr_running write against the IPI */ -smp_wmb (); +/* +* Tick is needed if more than one task runs on a CPU. +* Send the target an IPI to kick it out of nohz mode. +* +* We assume that IPI implies full memory barrier and the +* new value of rq->nr_running is visible on reception +* from the target. +*/ tick_nohz_full_kick_cpu (rq->cpu); } }
Signature · 1 line
1.8.3.1