Skip to contentSkip to conversation

nohz: Move nohz kick out of scheduler IPI v2

PATCH In review Frederic Weisbecker 2 participants 6 messages Apr 3, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover nohz: Move nohz kick out of scheduler IPI v2

Hi Paul,

Here's an updated version of the patches with your review addressed. I ripped the function parameter and let it be setup on queued IPI object initialization time so that people don't get confused and always use the same function for a given queued object.

Please tell me if you find other issues.

Thanks.

Frederic Weisbecker (2):
smp: Non busy-waiting IPI queue
nohz: Move full nohz kick to its own IPI

 include/linux/smp.h      | 11 +++++++++++
 include/linux/tick.h     |  2 ++
 kernel/sched/core.c      |  5 +----
 kernel/sched/sched.h     |  2 +-
 kernel/smp.c             | 42 ++++++++++++++++++++++++++++++++++++++++++
 kernel/time/tick-sched.c | 21 +++++++++++++++++++++
 6 files changed, 78 insertions(+), 5 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH 1/2 smp: Non busy-waiting IPI queue

Some IPI users, such as the nohz subsystem, need to be able to send an async IPI (async = non waiting for any other IPI completion) on contexts with disabled interrupts. And we want the IPI subsystem to handle concurrent calls by itself.

Currently the nohz machinery uses the scheduler IPI for this purpose because it can be triggered from any context and doesn't need any serialization from the caller. But this is an abuse of a scheduler fast path. We are bloating it with a job that should use its own IPI.

The current set of IPI functions can't be called when interrupts are disabled otherwise we risk a deadlock when two CPUs wait for each other's IPI completion.

OTOH smp_call_function_single_async() can be called when interrupts are disabled. But then it's up to the caller to serialize the given IPI. This can't be called concurrently without special care.

So we need a version of the async IPI that takes care of concurrent calls.

The proposed solution is to synchronize the IPI with a specific flag that prevents the IPI from being sent if it is already pending but not yet executed. Ordering is maintained such that, if the IPI is not sent because it's already pending, we guarantee it will see the new state of the data we expect it to when it will execute.

This model is close to the irq_work design. It's also partly inspired by suggestions from Peter Zijlstra.

Cc Andrew Morton <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Kevin Hilman <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 include/linux/smp.h | 11 +++++++++++
 kernel/smp.c        | 42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+)
include/linux/smp.h +11 -0
index 633f5ed..0de1eff 100644
@@ -29,6 +29,17 @@ extern unsigned int total_cpus;
 int smp_call_function_single(int cpuid, smp_call_func_t func, void *info,
 			     int wait);
 
+struct queue_single_data;
+typedef void (*smp_queue_func_t)(struct queue_single_data *qsd);
+
+struct queue_single_data {
+	struct call_single_data data;
+	smp_queue_func_t func;
+	int pending;
+};
+
+int smp_queue_function_single(int cpu, struct queue_single_data *qsd);
+
 /*
  * Call a function on all processors
  */
kernel/smp.c +42 -0

48 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#2 lore raw

PATCH 2/2 nohz: Move full nohz kick to its own IPI

In reply to Frederic Weisbecker

Now that we have smp_queue_function_single() which can be used to safely queue IPIs when interrupts are disabled and without worrying about concurrent callers, lets use it for the full dynticks kick to notify a CPU that it's exiting single task mode.

This unbloats a bit the scheduler IPI that the nohz code was abusing for its cool "callable anywhere/anytime" properties.

Cc Andrew Morton <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Kevin Hilman <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 include/linux/tick.h     |  2 ++
 kernel/sched/core.c      |  5 +----
 kernel/sched/sched.h     |  2 +-
 kernel/time/tick-sched.c | 21 +++++++++++++++++++++
 4 files changed, 25 insertions(+), 5 deletions(-)
include/linux/tick.h +2 -0
index b84773c..9d3fcc2 100644
@@ -182,6 +182,7 @@ static inline bool tick_nohz_full_cpu(int cpu)
 extern void tick_nohz_init(void);
 extern void __tick_nohz_full_check(void);
 extern void tick_nohz_full_kick(void);
+extern void tick_nohz_full_kick_cpu(int cpu);
 extern void tick_nohz_full_kick_all(void);
 extern void __tick_nohz_task_switch(struct task_struct *tsk);
 #else
@@ -190,6 +191,7 @@ static inline bool tick_nohz_full_enabled(void) { return false; }
 static inline bool tick_nohz_full_cpu(int cpu) { return false; }
 static inline void __tick_nohz_full_check(void) { }
 static inline void tick_nohz_full_kick(void) { }
+static inline void tick_nohz_full_kick_cpu(int cpu) { }
 static inline void tick_nohz_full_kick_all(void) { }
 static inline void __tick_nohz_task_switch(struct task_struct *tsk) { }
 #endif
kernel/sched/core.c +1 -4
index 9cae286..e4b344e 100644
@@ -1499,9 +1499,7 @@ void scheduler_ipi(void)
 	 */
 	preempt_fold_need_resched();
 
-	if (llist_empty(&this_rq()->wake_list)
-			&& !tick_nohz_full_cpu(smp_processor_id())
-			&& !got_nohz_idle_kick())
+	if (llist_empty(&this_rq()->wake_list) && !got_nohz_idle_kick())
 		return;
 
 	/*
@@ -1518,7 +1516,6 @@ void scheduler_ipi(void)
 	 * somewhat pessimize the simple resched case.
 	 */
 	irq_enter();
-	tick_nohz_full_check();
 	sched_ttwu_pending();
 
 	/*
kernel/sched/sched.h +1 -1
index c9007f2..4771063 100644
@@ -1225,7 +1225,7 @@ static inline void inc_nr_running(struct rq *rq)
 		if (tick_nohz_full_cpu(rq->cpu)) {
 			/* Order rq->nr_running write against the IPI */
 			smp_wmb();
-			smp_send_reschedule(rq->cpu);
+			tick_nohz_full_kick_cpu(rq->cpu);
 		}
        }
 #endif
kernel/time/tick-sched.c +21 -0
index 9f8af69..582d3f6 100644
@@ -230,6 +230,27 @@ void tick_nohz_full_kick(void)
 		irq_work_queue(&__get_cpu_var(nohz_full_kick_work));
 }
 
+static void nohz_full_kick_queue(struct queue_single_data *qsd)
+{
+	__tick_nohz_full_check();
+}
+
+static DEFINE_PER_CPU(struct queue_single_data, nohz_full_kick_qsd) = {
+	.func = nohz_full_kick_queue,
+};
+
+void tick_nohz_full_kick_cpu(int cpu)
+{
+	if (!tick_nohz_full_cpu(cpu))
+		return;
+
+	if (cpu == smp_processor_id()) {
+		irq_work_queue(&__get_cpu_var(nohz_full_kick_work));
+	} else {
+		smp_queue_function_single(cpu, &per_cpu(nohz_full_kick_qsd, cpu));
+	}
+}
+
 static void nohz_full_kick_ipi(void *info)
 {
 	__tick_nohz_full_check();
Signature · 1 line
1.8.3.1
#3 lore raw
In reply to Frederic Weisbecker

Sorry I got Jens address wrong once again :-(

Quoted 26 lines · Frederic Weisbecker

Hi Paul,

Here's an updated version of the patches with your review addressed. I ripped the function parameter and let it be setup on queued IPI object initialization time so that people don't get confused and always use the same function for a given queued object.

Please tell me if you find other issues.

Thanks.

Frederic Weisbecker (2):
smp: Non busy-waiting IPI queue
nohz: Move full nohz kick to its own IPI

 include/linux/smp.h      | 11 +++++++++++
 include/linux/tick.h     |  2 ++
 kernel/sched/core.c      |  5 +----
 kernel/sched/sched.h     |  2 +-
 kernel/smp.c             | 42 ++++++++++++++++++++++++++++++++++++++++++
 kernel/time/tick-sched.c | 21 +++++++++++++++++++++
 6 files changed, 78 insertions(+), 5 deletions(-)
Signature · 1 line
1.8.3.1
Reviewed-by on the v1 series
#4 lore raw
In reply to Frederic Weisbecker
Quoted 8 lines · Frederic Weisbecker

Hi Paul,

Here's an updated version of the patches with your review addressed. I ripped the function parameter and let it be setup on queued IPI object initialization time so that people don't get confused and always use the same function for a given queued object.

Please tell me if you find other issues.

For the set:

Quoted 17 lines

Thanks.

Frederic Weisbecker (2):
smp: Non busy-waiting IPI queue
nohz: Move full nohz kick to its own IPI

 include/linux/smp.h      | 11 +++++++++++
 include/linux/tick.h     |  2 ++
 kernel/sched/core.c      |  5 +----
 kernel/sched/sched.h     |  2 +-
 kernel/smp.c             | 42 ++++++++++++++++++++++++++++++++++++++++++
 kernel/time/tick-sched.c | 21 +++++++++++++++++++++
 6 files changed, 78 insertions(+), 5 deletions(-)
Signature · 1 line
1.8.3.1
#5 lore raw
Quoted 8 lines · Paul E. McKenney
Quoted 7 lines · Frederic Weisbecker

Hi Paul,

Here's an updated version of the patches with your review addressed. I ripped the function parameter and let it be setup on queued IPI object initialization time so that people don't get confused and always use the same function for a given queued object.

Quoted 5 lines
Quoted 1 line

Please tell me if you find other issues.

For the set:

Thanks a lot!

Quoted 19 lines
Quoted 18 lines

Thanks.

Frederic Weisbecker (2):
smp: Non busy-waiting IPI queue
nohz: Move full nohz kick to its own IPI

 include/linux/smp.h      | 11 +++++++++++
 include/linux/tick.h     |  2 ++
 kernel/sched/core.c      |  5 +----
 kernel/sched/sched.h     |  2 +-
 kernel/smp.c             | 42 ++++++++++++++++++++++++++++++++++++++++++
 kernel/time/tick-sched.c | 21 +++++++++++++++++++++
 6 files changed, 78 insertions(+), 5 deletions(-)
Signature · 1 line
1.8.3.1