Skip to contentSkip to conversation

sched: migrate_enable: Use per-cpu cpu_stop_work

PATCH In review Scott Wood 3 participants 4 messages Jan 24, 2020 – Feb 3, 2020 Scheduler

lore
Share & tools
lore

Scott Wood

author
#0 lore raw

PATCH 1/2 sched: migrate_enable: Use per-cpu cpu_stop_work

Commit e6c287b1512d ("sched: migrate_enable: Use stop_one_cpu_nowait()") adds a busy wait to deal with an edge case where the migrated thread can resume running on another CPU before the stopper has consumed cpu_stop_work. However, this is done with preemption disabled and can potentially lead to deadlock.

While it is not guaranteed that the cpu_stop_work will be consumed before the migrating thread resumes and exits the stack frame, it is guaranteed that nothing other than the stopper can run on the old cpu between the migrating thread scheduling out and the cpu_stop_work being consumed. Thus, we can store cpu_stop_work in per-cpu data without it being reused too early.

Fixes e6c287b1512d ("sched: migrate_enable: Use stop_one_cpu_nowait()")

---
Ignore the other 1/2 just sent -- forgot the RT in the subject and didn't quite hit Ctrl-C in time.

 kernel/sched/core.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
kernel/sched/core.c +14 -8
index 754f6afb438d..7713e9c34ad1 100644
@@ -8189,6 +8189,9 @@ static void migrate_disabled_sched(struct task_struct *p)
 	p->migrate_disable_scheduled = 1;
 }
 
+static DEFINE_PER_CPU(struct cpu_stop_work, migrate_work);
+static DEFINE_PER_CPU(struct migration_arg, migrate_arg);
+
 void migrate_enable(void)
 {
 	struct task_struct *p = current;
@@ -8227,22 +8230,25 @@ void migrate_enable(void)
 
 	WARN_ON(smp_processor_id() != cpu);
 	if (!is_cpu_allowed(p, cpu)) {
-		struct migration_arg arg = { .task = p };
-		struct cpu_stop_work work;
+		struct migration_arg __percpu *arg;
+		struct cpu_stop_work __percpu *work;
 		struct rq_flags rf;
 
+		work = this_cpu_ptr(&migrate_work);
+		arg = this_cpu_ptr(&migrate_arg);
+		WARN_ON_ONCE(!arg->done && !work->disabled && work->arg);
+
+		arg->task = p;
+		arg->done = false;
+
 		rq = task_rq_lock(p, &rf);
 		update_rq_clock(rq);
-		arg.dest_cpu = select_fallback_rq(cpu, p);
+		arg->dest_cpu = select_fallback_rq(cpu, p);
 		task_rq_unlock(rq, p, &rf);
 
 		stop_one_cpu_nowait(task_cpu(p), migration_cpu_stop,
-				    &arg, &work);
+				    arg, work);
 		__schedule(true);
-		if (!work.disabled) {
-			while (!arg.done)
-				cpu_relax();
-		}
 	}
 
 out:
Signature · 1 line
1.8.3.1
author
#1 lore raw

PATCH 2/2 sched: migrate_enable: Remove __schedule() call

We can rely on preempt_enable() to schedule. Besides simplifying the code, this potentially allows sequences such as the following to be permitted:

migrate_disable();
preempt_disable();
migrate_enable();
preempt_enable();

---
 kernel/sched/core.c | 1 -
 1 file changed, 1 deletion(-)
kernel/sched/core.c +0 -1
index 7713e9c34ad1..ea0536461981 100644
@@ -8248,7 +8248,6 @@ void migrate_enable(void)
 
 		stop_one_cpu_nowait(task_cpu(p), migration_cpu_stop,
 				    arg, work);
-		__schedule(true);
 	}
 
 out:
Signature · 1 line
1.8.3.1
reviewer Reviewed-by on v1 1/2
#2 lore raw
In reply to Scott Wood
Quoted 6 lines · On Fri, 24 Jan 2020 06:11:46 -0500 Scott Wood <[email protected]> wrote:

Commit e6c287b1512d ("sched: migrate_enable: Use stop_one_cpu_nowait()") adds a busy wait to deal with an edge case where the migrated thread can resume running on another CPU before the stopper has consumed cpu_stop_work. However, this is done with preemption disabled and can potentially lead to deadlock.

Quoted 7 lines

While it is not guaranteed that the cpu_stop_work will be consumed before the migrating thread resumes and exits the stack frame, it is guaranteed that nothing other than the stopper can run on the old cpu between the migrating thread scheduling out and the cpu_stop_work being consumed. Thus, we can store cpu_stop_work in per-cpu data without it being reused too early.

Makes sense to me.

-- Steve

#3 lore raw
In reply to Scott Wood
Quoted 13 lines · Scott Wood

Commit e6c287b1512d ("sched: migrate_enable: Use stop_one_cpu_nowait()") adds a busy wait to deal with an edge case where the migrated thread can resume running on another CPU before the stopper has consumed cpu_stop_work. However, this is done with preemption disabled and can potentially lead to deadlock.

While it is not guaranteed that the cpu_stop_work will be consumed before the migrating thread resumes and exits the stack frame, it is guaranteed that nothing other than the stopper can run on the old cpu between the migrating thread scheduling out and the cpu_stop_work being consumed. Thus, we can store cpu_stop_work in per-cpu data without it being reused too early.

Quoted 3 lines
Fixes e6c287b1512d ("sched: migrate_enable: Use stop_one_cpu_nowait()")

Yes, perfect, thank you.

Sebastian