hotplug cpu: migrate a task within its cpuset
Quoted 21 lines · Cliff Wickman
In order to do this,
move_task_off_dead_cpu()must make a call tocpuset_cpus_allowed(), which may block.
move_task_off_dead_cpu()has been within a critical region when called frommigrate_live_tasks(). So this patch also changesmigrate_live_tasks()to enable interrupts before callingmove_task_off_dead_cpu(). Since thetasklist_lockis dropped, the list scan must be restarted from the top.[... snip ...]
- * NOTE: interrupts should be disabled by the caller
+ * NOTE: interrupts are not disabled by the caller
*/
static voidmove_task_off_dead_cpu(int dead_cpu, struct task_struct *p){(quoted diff) +0
-0@@ -5008,6 +5008,17 @@ restart: if (dest_cpu == NR_CPUS) dest_cpu = any_online_cpu(p->cpus_allowed);
Review restart: · line 5020
Quoted 10 lines
(quoted diff) +10
-0+ /* try to stay on the same cpuset */ + if (dest_cpu == NR_CPUS) { + /* + * Call to cpuset_cpus_allowed may sleep, so we depend + * on move_task_off_dead_cpu() being called in a non-critical + * region. + */ + p->cpus_allowed = cpuset_cpus_allowed(p); + dest_cpu = any_online_cpu(p->cpus_allowed); + }
I know nothing about cpuset.c, a _very_ naive question.
Do we really need task_lock() (used by cpuset_cpus_allowed) here ?
If not, probably we can make this simpler. CPU_DEAD takes cpuset_lock(), move_task_off_dead_cpu() uses guarantee_online_cpus() which doesn't sleep, so we don't need other changes.
Possible?
If not, this patch should also change migrate_dead(), it still calls move_task_off_dead_cpu() with irqs disabled, no?
Oleg.
Quoted 22 lines · Oleg Nesterov
Quoted 21 lines · Cliff Wickman
In order to do this,
move_task_off_dead_cpu()must make a call tocpuset_cpus_allowed(), which may block.
move_task_off_dead_cpu()has been within a critical region when called frommigrate_live_tasks(). So this patch also changesmigrate_live_tasks()to enable interrupts before callingmove_task_off_dead_cpu(). Since thetasklist_lockis dropped, the list scan must be restarted from the top.[... snip ...]
- * NOTE: interrupts should be disabled by the caller
+ * NOTE: interrupts are not disabled by the caller
*/
static voidmove_task_off_dead_cpu(int dead_cpu, struct task_struct *p){(quoted diff) +0
-0@@ -5008,6 +5008,17 @@ restart: if (dest_cpu == NR_CPUS) dest_cpu = any_online_cpu(p->cpus_allowed);
Quoted 12 lines
Review restart: · line 5020
Quoted 10 lines
(quoted diff) +10
-0+ /* try to stay on the same cpuset */ + if (dest_cpu == NR_CPUS) { + /* + * Call to cpuset_cpus_allowed may sleep, so we depend + * on move_task_off_dead_cpu() being called in a non-critical + * region. + */ + p->cpus_allowed = cpuset_cpus_allowed(p); + dest_cpu = any_online_cpu(p->cpus_allowed); + }I know nothing about
cpuset.c, a _very_ naive question.
Paul Jackson is the cpuset guru.
Quoted 1 line
Do we really need
task_lock()(used bycpuset_cpus_allowed) here ?
According to Paul's comment in kernel/cpuset.c
* It is ok to first take manage_sem, then nest callback_sem. We also
* require taking task_lock() when dereferencing a tasks cpuset pointer.
So I'm afraid it is not safe to call guarantee_online_cpus(tsk->cpuset, &mask); without it. Could the task not be exiting?
Quoted 8 lines
If not, probably we can make this simpler.
CPU_DEADtakescpuset_lock(),move_task_off_dead_cpu()usesguarantee_online_cpus()which doesn't sleep, so we don't need other changes.Possible?
If not, this patch should also change
migrate_dead(), it still callsmove_task_off_dead_cpu()with irqs disabled, no?
Right, the lock is released but I indeed didn't reenable irqs. How would you suggest doing that? The irq state was saved in local variable "flags" back in migration_call().
Quoted 2 lines
Oleg.
Signature · 4 lines
Cliff Wickman Silicon Graphics, Inc. [email protected] (651) 683-3824
Quoted 26 lines · Cliff Wickman
Quoted 11 lines · Oleg Nesterov
Quoted 10 lines · Cliff Wickman
- * NOTE: interrupts should be disabled by the caller
+ * NOTE: interrupts are not disabled by the caller
*/
static voidmove_task_off_dead_cpu(int dead_cpu, struct task_struct *p){(quoted diff) +0
-0@@ -5008,6 +5008,17 @@ restart: if (dest_cpu == NR_CPUS) dest_cpu = any_online_cpu(p->cpus_allowed);Quoted 12 lines
Review restart: · line 5020
Quoted 10 lines
(quoted diff) +10
-0+ /* try to stay on the same cpuset */ + if (dest_cpu == NR_CPUS) { + /* + * Call to cpuset_cpus_allowed may sleep, so we depend + * on move_task_off_dead_cpu() being called in a non-critical + * region. + */ + p->cpus_allowed = cpuset_cpus_allowed(p); + dest_cpu = any_online_cpu(p->cpus_allowed); + }I know nothing about
cpuset.c, a _very_ naive question.
Quoted 1 line
Paul Jackson is the cpuset guru.
Hopefully Paul can help us...
Quoted 7 lines
Quoted 1 line
Do we really need
task_lock()(used bycpuset_cpus_allowed) here ?According to Paul's comment in
kernel/cpuset.c
* It is ok to first takemanage_sem, then nestcallback_sem. We also
* require takingtask_lock()when dereferencing a tasks cpuset pointer.
So I'm afraid it is not safe to callguarantee_online_cpus(tsk->cpuset, &mask); without it. Could the task not be exiting?
But how task_lock() can help? cpuset_exit() doesn't take it, it changes ->cpuset lockless. However, it sets ->cpuset = &top_cpuset, and the comment says:
* Don't even think about derefencing 'cs' after the cpuset use count
* goes to zero, except inside a critical section guarded by manage_mutex
* or callback_mutex.
^^^^^^^^^^^^^^
So, perhaps cpuset_lock() should be enough.
(That said, looking at cpuset_exit() I can't understand why callback_mutex is
enough. If it is not, cpuset_cpus_allowed() is not safe either).
Quoted 4 lines
Quoted 4 lines
If not, probably we can make this simpler.
CPU_DEADtakescpuset_lock(),move_task_off_dead_cpu()usesguarantee_online_cpus()which doesn't sleep, so we don't need other changes.
Quoted 8 lines
Quoted 4 lines
Possible?
If not, this patch should also change
migrate_dead(), it still callsmove_task_off_dead_cpu()with irqs disabled, no?Right, the lock is released but I indeed didn't reenable irqs. How would you suggest doing that? The irq state was saved in local variable "flags" back in
migration_call().
migration_call(CPU_DEAD) is called with irqs enabled.
Oleg.