Skip to contentSkip to conversation

hotplug cpu: migrate a task within its cpuset

PATCH Oleg Nesterov 2 participants 3 messages May 23, 2007

lore
Share & tools
lore
#0 lore raw
Quoted 21 lines · Cliff Wickman

In order to do this, move_task_off_dead_cpu() must make a call to cpuset_cpus_allowed(), which may block.

move_task_off_dead_cpu() has been within a critical region when called from migrate_live_tasks(). So this patch also changes migrate_live_tasks() to enable interrupts before calling move_task_off_dead_cpu(). Since the tasklist_lock is dropped, the list scan must be restarted from the top.

[... snip ...]

- * NOTE: interrupts should be disabled by the caller
+ * NOTE: interrupts are not disabled by the caller
*/
static void move_task_off_dead_cpu(int dead_cpu, struct task_struct *p) {

(quoted diff) +0 -0
@@ -5008,6 +5008,17 @@ restart:
 	if (dest_cpu == NR_CPUS)
 		dest_cpu = any_online_cpu(p->cpus_allowed);
 

Review restart: · line 5020

Quoted 10 lines
(quoted diff) +10 -0
+	/* try to stay on the same cpuset */
+	if (dest_cpu == NR_CPUS) {
+		/*
+		 * Call to cpuset_cpus_allowed may sleep, so we depend
+		 * on move_task_off_dead_cpu() being called in a non-critical
+		 * region.
+		 */
+		p->cpus_allowed = cpuset_cpus_allowed(p);
+		dest_cpu = any_online_cpu(p->cpus_allowed);
+	}

I know nothing about cpuset.c, a _very_ naive question.

Do we really need task_lock() (used by cpuset_cpus_allowed) here ?

If not, probably we can make this simpler. CPU_DEAD takes cpuset_lock(), move_task_off_dead_cpu() uses guarantee_online_cpus() which doesn't sleep, so we don't need other changes.

Possible?

If not, this patch should also change migrate_dead(), it still calls move_task_off_dead_cpu() with irqs disabled, no?

Oleg.

#1 lore raw
Quoted 22 lines · Oleg Nesterov
Quoted 21 lines · Cliff Wickman

In order to do this, move_task_off_dead_cpu() must make a call to cpuset_cpus_allowed(), which may block.

move_task_off_dead_cpu() has been within a critical region when called from migrate_live_tasks(). So this patch also changes migrate_live_tasks() to enable interrupts before calling move_task_off_dead_cpu(). Since the tasklist_lock is dropped, the list scan must be restarted from the top.

[... snip ...]

- * NOTE: interrupts should be disabled by the caller
+ * NOTE: interrupts are not disabled by the caller
*/
static void move_task_off_dead_cpu(int dead_cpu, struct task_struct *p) {

(quoted diff) +0 -0
@@ -5008,6 +5008,17 @@ restart:
 	if (dest_cpu == NR_CPUS)
 		dest_cpu = any_online_cpu(p->cpus_allowed);
 
Quoted 12 lines

Review restart: · line 5020

Quoted 10 lines
(quoted diff) +10 -0
+	/* try to stay on the same cpuset */
+	if (dest_cpu == NR_CPUS) {
+		/*
+		 * Call to cpuset_cpus_allowed may sleep, so we depend
+		 * on move_task_off_dead_cpu() being called in a non-critical
+		 * region.
+		 */
+		p->cpus_allowed = cpuset_cpus_allowed(p);
+		dest_cpu = any_online_cpu(p->cpus_allowed);
+	}

I know nothing about cpuset.c, a _very_ naive question.

Paul Jackson is the cpuset guru.

Quoted 1 line

Do we really need task_lock() (used by cpuset_cpus_allowed) here ?

According to Paul's comment in kernel/cpuset.c
* It is ok to first take manage_sem, then nest callback_sem. We also
* require taking task_lock() when dereferencing a tasks cpuset pointer.
So I'm afraid it is not safe to call guarantee_online_cpus(tsk->cpuset, &mask); without it. Could the task not be exiting?

Quoted 8 lines

If not, probably we can make this simpler. CPU_DEAD takes cpuset_lock(), move_task_off_dead_cpu() uses guarantee_online_cpus() which doesn't sleep, so we don't need other changes.

Possible?

If not, this patch should also change migrate_dead(), it still calls move_task_off_dead_cpu() with irqs disabled, no?

Right, the lock is released but I indeed didn't reenable irqs. How would you suggest doing that? The irq state was saved in local variable "flags" back in migration_call().

Quoted 2 lines

Oleg.

Signature · 4 lines
Cliff Wickman
Silicon Graphics, Inc.
[email protected]
(651) 683-3824
#2 lore raw
Quoted 26 lines · Cliff Wickman
Quoted 11 lines · Oleg Nesterov
Quoted 10 lines · Cliff Wickman

- * NOTE: interrupts should be disabled by the caller
+ * NOTE: interrupts are not disabled by the caller
*/
static void move_task_off_dead_cpu(int dead_cpu, struct task_struct *p) {

(quoted diff) +0 -0
@@ -5008,6 +5008,17 @@ restart:
 	if (dest_cpu == NR_CPUS)
 		dest_cpu = any_online_cpu(p->cpus_allowed);
 
Quoted 12 lines

Review restart: · line 5020

Quoted 10 lines
(quoted diff) +10 -0
+	/* try to stay on the same cpuset */
+	if (dest_cpu == NR_CPUS) {
+		/*
+		 * Call to cpuset_cpus_allowed may sleep, so we depend
+		 * on move_task_off_dead_cpu() being called in a non-critical
+		 * region.
+		 */
+		p->cpus_allowed = cpuset_cpus_allowed(p);
+		dest_cpu = any_online_cpu(p->cpus_allowed);
+	}

I know nothing about cpuset.c, a _very_ naive question.

Quoted 1 line

Paul Jackson is the cpuset guru.

Hopefully Paul can help us...

Quoted 7 lines
Quoted 1 line

Do we really need task_lock() (used by cpuset_cpus_allowed) here ?

According to Paul's comment in kernel/cpuset.c
* It is ok to first take manage_sem, then nest callback_sem. We also
* require taking task_lock() when dereferencing a tasks cpuset pointer.
So I'm afraid it is not safe to call guarantee_online_cpus(tsk->cpuset, &mask); without it. Could the task not be exiting?

But how task_lock() can help? cpuset_exit() doesn't take it, it changes ->cpuset lockless. However, it sets ->cpuset = &top_cpuset, and the comment says:

* Don't even think about derefencing 'cs' after the cpuset use count
* goes to zero, except inside a critical section guarded by manage_mutex
* or callback_mutex.
^^^^^^^^^^^^^^
So, perhaps cpuset_lock() should be enough.

(That said, looking at cpuset_exit() I can't understand why callback_mutex is
enough. If it is not, cpuset_cpus_allowed() is not safe either).

Quoted 4 lines
Quoted 4 lines

If not, probably we can make this simpler. CPU_DEAD takes cpuset_lock(), move_task_off_dead_cpu() uses guarantee_online_cpus() which doesn't sleep, so we don't need other changes.

Quoted 8 lines
Quoted 4 lines

Possible?

If not, this patch should also change migrate_dead(), it still calls move_task_off_dead_cpu() with irqs disabled, no?

Right, the lock is released but I indeed didn't reenable irqs. How would you suggest doing that? The irq state was saved in local variable "flags" back in migration_call().

migration_call(CPU_DEAD) is called with irqs enabled.

Oleg.