2.6.21-rc7-mm1: BUG_ON in kthread_bind during _cpu_down
Subsystems (2)
Hi,
The BUG_ON in khthread_bind (line 165 in kthread.c) triggers for me during attempted suspend to disk, when disable_nonboot_cpus() calls _cpu_down() (on x86_64).
This didn't happen with 2.6.21-rc6-mm1.
Greetings,
Rafael
Signature · 3 lines
If you don't have the time to read, you don't have the time or the tools to write. - Stephen King
Quoted 5 lines · On Thu, 26 Apr 2007 01:10:21 +0200 "Rafael J. Wysocki" <[email protected]> wrote:
Hi,
The
BUG_ONinkhthread_bind(line 165 inkthread.c) triggers for me during attempted suspend to disk, whendisable_nonboot_cpus()calls_cpu_down()(onx86_64).
I guess the backtrace would be pretty important here.
Guys, please don't add BUG_ONs unless there is simply no sane way to recover.
Because when someone goofs up, the BUG_ON will kill the whole machine and everyone else who has code being tested in -mm loses a tester.
Plus a BUG_ON *greatly* decreases our chances of getting a trace from the
tester: dead box, nothing in the logs.
kernel/kthread.c renamed from kernel/kthread.c~fix-kthread_create-vs-freezer-theoretical-race-dont-be-obnoxious +4 -1
@@ -162,7 +162,10 @@ EXPORT_SYMBOL(kthread_create);*/ void kthread_bind (struct task_struct *k,unsigned int cpu) { -BUG_ON (k->state !=TASK_UNINTERRUPTIBLE ); +if (k->state !=TASK_UNINTERRUPTIBLE ) { +WARN_ON (1 ); +return ; + }/* Must have done schedule() in kthread() before we set_task_cpu */ wait_task_inactive (k);set_task_cpu (k, cpu);
_
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Thu, 26 Apr 2007 01:10:21 +0200 "Rafael J. Wysocki" <[email protected]> wrote:
Hi,
The
BUG_ONinkhthread_bind(line 165 inkthread.c) triggers for me during attempted suspend to disk, whendisable_nonboot_cpus()calls_cpu_down()(onx86_64).
Caused due to Oleg's patch http://lkml.org/lkml/2007/4/13/93.
Agreed that most of the time a kthread_create(p) is followed by a kthread_bind(p), in which case the assertion WARN_ON(p->state != TASK_UNINTERRUPTIBLE) makes sense.
But, in cpu hotplug case, we need to rebind the stop_machine_run thread from the cpu which has just been offlined to any online cpu. (kernel/cpu.c line 180)
At this point, the thread would be in TASK_INTERRUPTIBLE waiting for us to call a kthread_stop on it.(kernel/kthread.c line 161)
We only need to ensure in kthread_bind that the task which is being bound is not running or exiting. Doesn't matter if it's sleeping in TASK_INTERRUPTIBLE or TASK_UNINTERRUPTIBLE state.
--- kernel/kthread.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletion(-)
kernel/kthread.c +2 -1
@@ -162,7 +162,8 @@ EXPORT_SYMBOL(kthread_create);*/ void kthread_bind (struct task_struct *k,unsigned int cpu) { -if (k->state !=TASK_UNINTERRUPTIBLE ) { +if (k->state !=TASK_UNINTERRUPTIBLE && + k->state !=TASK_INTERRUPTIBLE ) {WARN_ON (1 );return ; }
Signature · 5 lines
Gautham R Shenoy Linux Technology Center IBM India. "Freedom comes with a price tag of responsibility, which is still a bargain, because Freedom is priceless!"
Quoted 21 lines · Gautham R Shenoy
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Thu, 26 Apr 2007 01:10:21 +0200 "Rafael J. Wysocki" <[email protected]> wrote:
Hi,
The
BUG_ONinkhthread_bind(line 165 inkthread.c) triggers for me during attempted suspend to disk, whendisable_nonboot_cpus()calls_cpu_down()(onx86_64).Caused due to Oleg's patch http://lkml.org/lkml/2007/4/13/93.
Agreed that most of the time a
kthread_create(p)is followed by akthread_bind(p), in which case the assertionWARN_ON(p->state != TASK_UNINTERRUPTIBLE)makes sense.But, in cpu hotplug case, we need to rebind the
stop_machine_runthread from the cpu which has just been offlined to any online cpu. (kernel/cpu.cline 180)
At this point, the thread would be inTASK_INTERRUPTIBLEwaiting for us to call akthread_stopon it.(kernel/kthread.c line 161)
Quoted 3 lines
We only need to ensure in
kthread_bindthat the task which is being bound is not running or exiting. Doesn't matter if it's sleeping inTASK_INTERRUPTIBLEorTASK_UNINTERRUPTIBLEstate.
That will probably handle this problem.
However there is a weird interaction with process freezer.
The process freezer can come in and wake up a kernel thread to encourage it to call try_to_freeze_process while it is waiting to be bound.
How do we handle that evil race?
Eric
Quoted 12 lines · Gautham R Shenoy
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Thu, 26 Apr 2007 01:10:21 +0200 "Rafael J. Wysocki" <[email protected]> wrote:
Hi,
The
BUG_ONinkhthread_bind(line 165 inkthread.c) triggers for me during attempted suspend to disk, whendisable_nonboot_cpus()calls_cpu_down()(onx86_64).Caused due to Oleg's patch http://lkml.org/lkml/2007/4/13/93.
Quoted 7 lines
Agreed that most of the time a
kthread_create(p)is followed by akthread_bind(p), in which case the assertionWARN_ON(p->state != TASK_UNINTERRUPTIBLE)makes sense.But, in cpu hotplug case, we need to rebind the
stop_machine_runthread from the cpu which has just been offlined to any online cpu. (kernel/cpu.cline 180)
I can't understand why do we need to re-bind this thread. We are doing kthread_stop()->wake_up() below, at this point move_task_off_dead_cpu() has already cared about this task, no?
Quoted 3 lines
We only need to ensure in
kthread_bindthat the task which is being bound is not running or exiting. Doesn't matter if it's sleeping inTASK_INTERRUPTIBLEorTASK_UNINTERRUPTIBLEstate.
We need to ensure that this task can't be woken after return from wait_task_inactive(k), otherwise set_task_cpu() after that is not safe.
TASK_INTERRUPTIBLE doesn't protect us from freezing.
Couldn't we just remove this kthread_bind() ?
Oleg.
Quoted 9 lines · Oleg Nesterov
I can't understand why do we need to re-bind this thread. We are doing kthread_stop()->wake_up() below, at this point
move_task_off_dead_cpu()has already cared about this task, no?Quoted 3 lines
We only need to ensure in
kthread_bindthat the task which is being bound is not running or exiting. Doesn't matter if it's sleeping inTASK_INTERRUPTIBLEorTASK_UNINTERRUPTIBLEstate.
Quoted 6 lines
We need to ensure that this task can't be woken after return from
wait_task_inactive(k), otherwiseset_task_cpu()after that is not safe.
TASK_INTERRUPTIBLEdoesn't protect us from freezing.Couldn't we just remove this
kthread_bind()?
Fair enough. We are anyway kthread_stop()ping other per-cpu kernel threads after move_task_off_dead_cpu(), so we can do it with the stop_machine_run thread as well.
I just checked with Vatsa if there was any subtle reason why they had put in the kthread_bind() in cpu.c. Vatsa cannot seem to recollect any and I can't see any. So let us just remove the kthread_bind.
--- kernel/cpu.c | 4 ---- 1 files changed, 4 deletions(-)
kernel/cpu.c +0 -4
@@ -176,10 +176,6 @@ static int _cpu_down(unsigned int cpu, i/* This actually kills the CPU. */ __cpu_die (cpu); -/* Move it here so it can run. */ -kthread_bind (p,get_cpu ()); -put_cpu (); -/* CPU is completely dead: tell everyone. Too late to complain. */ if (raw_notifier_call_chain (&cpu_chain,CPU_DEAD | mod, hcpu) ==NOTIFY_BAD )
Signature · 5 lines
Gautham R Shenoy Linux Technology Center IBM India. "Freedom comes with a price tag of responsibility, which is still a bargain, because Freedom is priceless!"
Quoted 14 lines · On Thu, 26 Apr 2007 18:28:38 +0530 Gautham R Shenoy <[email protected]> wrote:
I just checked with Vatsa if there was any subtle reason why they had put in the
kthread_bind()incpu.c. Vatsa cannot seem to recollect any and I can't see any. So let us just remove thekthread_bind.Signed-off-by Gautham R Shenoy <[email protected]>--- kernel/cpu.c | 4 ---- 1 files changed, 4 deletions(-)Index:
linux-2.6.21-rc7/kernel/cpu.c
===================================================================
kernel/cpu.c static int _cpu_down(unsigned int cpu, i · line 181
Quoted 10 lines
kernel/cpu.c +0
-4@@ -176,10 +176,6 @@ static int _cpu_down(unsigned int cpu, i/* This actually kills the CPU. */ __cpu_die (cpu); -/* Move it here so it can run. */ -kthread_bind (p,get_cpu ()); -put_cpu (); -/* CPU is completely dead: tell everyone. Too late to complain. */ if (raw_notifier_call_chain (&cpu_chain,CPU_DEAD | mod, hcpu) ==NOTIFY_BAD )
So I cooked up a changelog and queued up the diff. But I have an uneasy feeling that things are getting a bit close to guesswork here.
We have a huge amount of change pending in the kthread/workqueue/freezer area, partly because I decided not to merge most of the workqueue changes into 2.6.21.
It'd be good if people could take some time to sit down and re-review the code which we presently have. I plan on sending it all off for 2.6.22 and there might be some glitches but it seems to have a good track record so far.
PATCH 1/1 add-suspend-related-notifications-for-cpu-hotplug-cleanup
Quoted 9 lines · Andrew Morton
We have a huge amount of change pending in the kthread/workqueue/freezer area, partly because I decided not to merge most of the workqueue changes into 2.6.21.
It'd be good if people could take some time to sit down and re-review the code which we presently have. I plan on sending it all off for 2.6.22 and there might be some glitches but it seems to have a good track record so far.
Oops. I completely misread the ->cpus_allowed check in try_to_wake_up(), and so workqueue_cpu_callback() needs a simple fix: CPU_UP_CANCELED should rebind cwq->thread. Or, better, we should not bind the task until CPU_ONLINE.
I'll send the patch soon, but it conflicts with Rafael's
add-suspend-related-notifications-for-cpu-hotplug.patch
, so I am sending this cleanup first to make them independent.
Actually, I believe almost all subsystems could be simplified as well, most of them don't care about CPU_TASKS_FROZEN bit.
kernel/workqueue.c renamed from kernel/workqueue.c~1_FROZEN +2 -5
@@ -747,6 +747,8 @@ static int __devinit workqueue_cpu_callbstruct cpu_workqueue_struct *cwq;struct workqueue_struct *wq; + action &= ~CPU_TASKS_FROZEN ; +switch (action) {case CPU_LOCK_ACQUIRE :mutex_lock (&workqueue_mutex);
@@ -757,7 +759,6 @@ static int __devinit workqueue_cpu_callbreturn NOTIFY_OK ;case CPU_UP_PREPARE : -case CPU_UP_PREPARE_FROZEN :cpu_set (cpu, cpu_populated_map); }
@@ -766,23 +767,19 @@ static int __devinit workqueue_cpu_callbswitch (action) {case CPU_UP_PREPARE : -case CPU_UP_PREPARE_FROZEN :if (!create_workqueue_thread (cwq, cpu))break ;printk (KERN_ERR "workqueue for %i failed\n" , cpu);return NOTIFY_BAD ;case CPU_ONLINE : -case CPU_ONLINE_FROZEN :wake_up_process (cwq->thread);break ;case CPU_UP_CANCELED : -case CPU_UP_CANCELED_FROZEN :if (cwq->thread)wake_up_process (cwq->thread);case CPU_DEAD : -case CPU_DEAD_FROZEN :cleanup_workqueue_thread (cwq, cpu);break ; }
libata-core.c: unsafe cancel_delayed_work() usage?
In reply to Andrew MortonQuoted 5 lines · Andrew Morton
It'd be good if people could take some time to sit down and re-review the code which we presently have. I plan on sending it all off for 2.6.22 and there might be some glitches but it seems to have a good track record so far.
Alan, the usage of cancel_delayed_work() in drivers/ata/libata-core.c looks suspicious to me, both ->hotplug_task and ->port_task can re-arm themselves, so cancel_delayed_work + flush_workqueue (or cancel_work_sync) is not enough.
Could you confirm/nack my understanding?
Oleg.
Re: libata-core.c: unsafe cancel_delayed_work() usage?
Quoted 11 lines · Oleg Nesterov
Quoted 4 lines · Andrew Morton
It'd be good if people could take some time to sit down and re-review the code which we presently have. I plan on sending it all off for 2.6.22 and there might be some glitches but it seems to have a good track record so far.
Alan, the usage of
cancel_delayed_work()indrivers/ata/libata-core.clooks suspicious to me, both ->hotplug_task and ->port_task can re-arm themselves, socancel_delayed_work+flush_workqueue(orcancel_work_sync) is not enough.Could you confirm/nack my understanding?
For all rearming workqueues, one must use cancel-rearming-blahblah.
Jeff
Re: libata-core.c: unsafe cancel_delayed_work() usage?
Quoted 9 lines · Jeff Garzik
Quoted 7 lines · Oleg Nesterov
Alan, the usage of
cancel_delayed_work()indrivers/ata/libata-core.clooks suspicious to me, both ->hotplug_task and ->port_task can re-arm themselves,
socancel_delayed_work+flush_workqueue(orcancel_work_sync) is not enough.
Quoted 3 lines
Quoted 1 line
Could you confirm/nack my understanding?
For all rearming workqueues, one must use cancel-rearming-blahblah.
Well, yes and no. cancel_rearming_delayed_work() requires that dwork re-arms itself unconditionally, otherwise it just hangs (actually, the patch to fix this shortcoming is ready). However, ata_pio_task() for example, re-arms only when ATA_BUSY.
So it is very possible that the code is correct, if the caller of ata_port_flush_task() ensures that ATA_BUSY is not possible. Otherwise we should change cancel_rearming_delayed_work() and use it.
Oleg.
kernel/relay.c: a strange usage of delayed_work
In reply to Andrew Mortonrelay_switch_subbuf() does schedule_delayed_work(&buf->wake_readers, 1), wakeup_readers() only does wake_up_interruptible() and nothing more.
Why can't we use a plain timer for this?
In any case, this "wake_up ->read_wait after a minimal possible delay" looks somewhat strange to me, could you explain? just curious.
Oleg.
Re: kernel/relay.c: a strange usage of delayed_work
Quoted 8 lines · Oleg Nesterov
relay_switch_subbuf()doesschedule_delayed_work(&buf->wake_readers, 1),wakeup_readers()only doeswake_up_interruptible()and nothing more.Why can't we use a plain timer for this?
In any case, this "
wake_up->read_wait after a minimal possible delay" looks somewhat strange to me, could you explain? just curious.
The reason it's done that way is that if the event that causes the relay_switch_subbuf() happens to be an event logged from schedule(), and we directly call wake_up_interruptible() at that point, we lock up the machine because it ends up back in schedule(). Deferring it avoids the problem.
I don't see any problem with using a plain timer instead - I'll work up a patch to make that change.
Tom
Quoted 9 lines · Andrew Morton
Quoted 5 lines · On Thu, 26 Apr 2007 01:10:21 +0200 "Rafael J. Wysocki" <[email protected]> wrote:
Hi,
The
BUG_ONinkhthread_bind(line 165 inkthread.c) triggers for me during attempted suspend to disk, whendisable_nonboot_cpus()calls_cpu_down()(onx86_64).I guess the backtrace would be pretty important here.
Yes, I didn't have the time to collect one yestarday.