rt: Some fixes for migrate_disable/enable

This RFC suggest three changes from the migrate disable/enable mechanism. Two of them are fixes, and the last one is a suggestion/improvement.

As migrate_disable/enable is RT specific, this patch set is RT specific.

Daniel Bristot de Oliveira (3):
rt: Increase/decrease the nr of migratable tasks when
enabling/disabling migration
rt: Update nr_cpus_allowed if the affinity of a task changes while its
migration is disabled
rt: Checks if task needs migration when re-enabling migration

 kernel/sched/core.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

2.9.4

There is a problem in the migrate_disable()/enable() implementation regarding the number of migratable tasks in the rt/dl RQs. The problem is the following:

When a task is attached to the rt runqueue, it is checked if it either can run in more than one CPU, or if it is with migration disable. If either check is true, the rt_rq->rt_nr_migratory counter is not increased. The counter increases otherwise.

When the task is detached, the same check is done. If either check is true, the rt_rq->rt_nr_migratory counter is not decreased. The counter decreases otherwise. The same check is done in the dl scheduler.

One important thing is that, migrate disable/enable does not touch this counter for tasks attached to the rt rq. So suppose the following chain of events.

Assumptions:

Task A is the only runnable task in A      Task B runs on the CPU B
Task A runs on CFS (non-rt)                Task B has RT priority
Thus, rt_nr_migratory is 0                 B is running

Task A can run on all CPUS.

Timeline:
CPU A/TASK A CPU B/TASK B
A takes the rt mutex X .
A disables migration .
. B tries to take the rt mutex X
. As it is held by A {
. A inherits the rt priority of B
. A is dequeued from CFS RQ of CPU A
. A is enqueued in the RT RQ of CPU A
. As migration is disabled
. rt_nr_migratory in A is not increased
.
A enables migration
A releases the rt mutex X {
A returns to its original priority
A ask to be dequeued from RT RQ {
As migration is now enabled and it can run on all CPUS {
rt_nr_migratory should be decreased
As rt_nr_migratory is 0, rt_nr_migratory under flows
}
}

This variable is important because it notifies if there are more than one runnable & migratable task in the runqueue. If there are more than one tasks, the rt_rq is set as overloaded, and then tries to migrate some tasks. This rule is important to keep the scheduler working conserving, that is, in a system with M CPUs, the M highest priority tasks should be running.

As rt_nr_migratory is unsigned, it will become > 0, notifying that the RQ is overloaded, activating pushing mechanism without need.

This patch fixes this problem by decreasing/increasing the rt/dl_nr_migratory in the migrate disable/enable operations.

---
 kernel/sched/core.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

index 10e832d..aeb3e12 100644
@@ -3376,6 +3376,8 @@ static inline void schedule_debug(struct task_struct *prev)
 void migrate_disable(void)
 {
 	struct task_struct *p = current;
+	struct rq_flags rf;
+	struct rq *rq;
 
 	if (in_atomic() || irqs_disabled()) {
 #ifdef CONFIG_SCHED_DEBUG

@@ -3399,7 +3401,19 @@ void migrate_disable(void)
 	preempt_disable();
 	preempt_lazy_disable();
 	pin_current_cpu();
+	rq = task_rq_lock(p, &rf);
 	p->migrate_disable = 1;
+
+	if (unlikely((p->sched_class == &rt_sched_class ||
+		      p->sched_class == &dl_sched_class) &&
+		      p->nr_cpus_allowed > 1)) {
+		if (p->sched_class == &rt_sched_class)
+			task_rq(p)->rt.rt_nr_migratory--;
+		else
+			task_rq(p)->dl.dl_nr_migratory--;
+	}
+
+	task_rq_unlock(rq, p, &rf);
 	preempt_enable();
 }
 EXPORT_SYMBOL(migrate_disable);

@@ -3407,6 +3421,8 @@ EXPORT_SYMBOL(migrate_disable);
 void migrate_enable(void)
 {
 	struct task_struct *p = current;
+	struct rq_flags rf;
+	struct rq *rq;
 
 	if (in_atomic() || irqs_disabled()) {
 #ifdef CONFIG_SCHED_DEBUG

@@ -3429,12 +3445,24 @@ void migrate_enable(void)
 	}
 
 	preempt_disable();
+	rq = task_rq_lock(p, &rf);
+
 	/*
 	 * Clearing migrate_disable causes tsk_cpus_allowed to
 	 * show the tasks original cpu affinity.
 	 */
 	p->migrate_disable = 0;
 
+	if (unlikely((p->sched_class == &rt_sched_class ||
+		      p->sched_class == &dl_sched_class) &&
+		      p->nr_cpus_allowed > 1)) {
+		if (p->sched_class == &rt_sched_class)
+			task_rq(p)->rt.rt_nr_migratory++;
+		else
+			task_rq(p)->dl.dl_nr_migratory++;
+	}
+
+	task_rq_unlock(rq, p, &rf);
 	unpin_current_cpu();
 	preempt_enable();
 	preempt_lazy_enable();

2.9.4

Currently, if the affinity of a task changes when it is with migration disabled, the nr_cpus_allowed is not being updated, creating an inconsistency between nr_cpus_allowed and cpumask_weight(cpus_allowed)

This patch fixes this problem by calling set_cpus_allowed_common() if the cpumask of a task changes while it is with migration disable.

---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index aeb3e12..0396bf2 100644
@@ -1146,7 +1146,7 @@ void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask)
 	lockdep_assert_held(&p->pi_lock);
 
 	if (__migrate_disabled(p)) {
-		cpumask_copy(&p->cpus_allowed, new_mask);
+		set_cpus_allowed_common(p, new_mask);
 		return;
 	}
 

2.9.4

In the case of an affinity change during a migrate_disable section, __set_cpus_allowed_ptr will not try to move the task from a CPU in which it cannot execute anymore.

So, after enabling migration, if the current task cannot execute in the current CPU anymore, migrate it away.

---
 kernel/sched/core.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

index 0396bf2..207bc85 100644
@@ -3462,6 +3462,34 @@ void migrate_enable(void)
 			task_rq(p)->dl.dl_nr_migratory++;
 	}
 
+	/*
+	 * Check if the task can still run on this CPU. In the case of an
+	 * affinity change during a migrate_disable section,
+	 * __set_cpus_allowed_ptr will not try to move the task from a CPU
+	 * that the task cannot execute anymore.
+	 *
+	 * So, if the current task cannot execute in the current CPU anymore,
+	 * migrate it away.
+	 */
+	if (unlikely(!cpumask_test_cpu(task_cpu(p), &p->cpus_allowed))) {
+		const struct cpumask *cpu_mask = (p->flags & PF_KTHREAD) ?
+			cpu_online_mask : cpu_active_mask;
+
+		int dest_cpu = cpumask_any_and(cpu_mask, &p->cpus_allowed);
+		struct migration_arg arg = {p, dest_cpu};
+
+		/* Need help from migration thread: drop lock and wait. */
+		task_rq_unlock(rq, p, &rf);
+		unpin_current_cpu();
+		preempt_enable();
+		preempt_lazy_enable();
+
+		stop_one_cpu(cpu_of(rq), migration_cpu_stop, &arg);
+		tlb_migrate_finish(p->mm);
+
+		return;
+	}
+
 	task_rq_unlock(rq, p, &rf);
 	unpin_current_cpu();
 	preempt_enable();

2.9.4

Okay, taking this for v4.9 then (mostly the same, except for one superfluous check):

Subject: [PATCH] sched/migrate disable: handle updated task-mask mg-dis section

If task's cpumask changes while in the task is in a migrate_disable() section then we don't react on it after a migrate_enable(). It matters however if current CPU is no longer part of the cpumask. We also miss the ->set_cpus_allowed() callback.
This patch fixes it by setting task->migrate_disable_update once we this "delayed" hook.
This bug was introduced while fixing unrelated issue in migrate_disable() in v4.4-rt3 (update_migrate_disable() got removed during that).

---
 include/linux/sched.h |  1 +
 kernel/sched/core.c   | 59 +++++++++++++++++++++++++++++++++++++++++++++------
 2 files changed, 54 insertions(+), 6 deletions(-)

index 4d779486ad6b..89b453dc7c14 100644
@@ -1535,6 +1535,7 @@ struct task_struct {
 	unsigned int policy;
 #ifdef CONFIG_PREEMPT_RT_FULL
 	int migrate_disable;
+	int migrate_disable_update;
 # ifdef CONFIG_SCHED_DEBUG
 	int migrate_disable_atomic;
 # endif

2.13.1