Skip to contentSkip to conversation

2.6.21-rc7-mm1: BUG_ON in kthread_bind during _cpu_down

PATCH New Rafael J. Wysocki 7 participants 14 messages Apr 25, 2007 – May 4, 2007

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

The BUG_ON in khthread_bind (line 165 in kthread.c) triggers for me during attempted suspend to disk, when disable_nonboot_cpus() calls _cpu_down() (on x86_64).

This didn't happen with 2.6.21-rc6-mm1.

Greetings,
Rafael

Signature · 3 lines
If you don't have the time to read,
you don't have the time or the tools to write.
		- Stephen King
#1 lore raw
Quoted 5 lines · On Thu, 26 Apr 2007 01:10:21 +0200 "Rafael J. Wysocki" <[email protected]> wrote:

Hi,

The BUG_ON in khthread_bind (line 165 in kthread.c) triggers for me during attempted suspend to disk, when disable_nonboot_cpus() calls _cpu_down() (on x86_64).

I guess the backtrace would be pretty important here.

Guys, please don't add BUG_ONs unless there is simply no sane way to recover.

Because when someone goofs up, the BUG_ON will kill the whole machine and everyone else who has code being tested in -mm loses a tester.

Plus a BUG_ON *greatly* decreases our chances of getting a trace from the
tester: dead box, nothing in the logs.

kernel/kthread.c renamed from kernel/kthread.c~fix-kthread_create-vs-freezer-theoretical-race-dont-be-obnoxious +4 -1
@@ -162,7 +162,10 @@ EXPORT_SYMBOL(kthread_create);
  */
 void kthread_bind(struct task_struct *k, unsigned int cpu)
 {
-	BUG_ON(k->state != TASK_UNINTERRUPTIBLE);
+	if (k->state != TASK_UNINTERRUPTIBLE) {
+		WARN_ON(1);
+		return;
+	}
 	/* Must have done schedule() in kthread() before we set_task_cpu */
 	wait_task_inactive(k);
 	set_task_cpu(k, cpu);

_

#2 lore raw
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Thu, 26 Apr 2007 01:10:21 +0200 "Rafael J. Wysocki" <[email protected]> wrote:

Hi,

The BUG_ON in khthread_bind (line 165 in kthread.c) triggers for me during attempted suspend to disk, when disable_nonboot_cpus() calls _cpu_down() (on x86_64).

Caused due to Oleg's patch http://lkml.org/lkml/2007/4/13/93.

Agreed that most of the time a kthread_create(p) is followed by a kthread_bind(p), in which case the assertion WARN_ON(p->state != TASK_UNINTERRUPTIBLE) makes sense.

But, in cpu hotplug case, we need to rebind the stop_machine_run thread from the cpu which has just been offlined to any online cpu. (kernel/cpu.c line 180)
At this point, the thread would be in TASK_INTERRUPTIBLE waiting for us to call a kthread_stop on it.(kernel/kthread.c line 161)

We only need to ensure in kthread_bind that the task which is being bound is not running or exiting. Doesn't matter if it's sleeping in TASK_INTERRUPTIBLE or TASK_UNINTERRUPTIBLE state.

---
 kernel/kthread.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletion(-)
kernel/kthread.c +2 -1
Index: linux-2.6.21-rc7/kernel/kthread.c
===================================================================
@@ -162,7 +162,8 @@ EXPORT_SYMBOL(kthread_create);
  */
 void kthread_bind(struct task_struct *k, unsigned int cpu)
 {
-	if (k->state != TASK_UNINTERRUPTIBLE) {
+	if (k->state != TASK_UNINTERRUPTIBLE &&
+	    k->state != TASK_INTERRUPTIBLE) {
 		WARN_ON(1);
 		return;
 	}
Signature · 5 lines
Gautham R Shenoy
Linux Technology Center
IBM India.
"Freedom comes with a price tag of responsibility, which is still a bargain,
because Freedom is priceless!"
#3 lore raw
Quoted 21 lines · Gautham R Shenoy
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Thu, 26 Apr 2007 01:10:21 +0200 "Rafael J. Wysocki" <[email protected]> wrote:

Hi,

The BUG_ON in khthread_bind (line 165 in kthread.c) triggers for me during attempted suspend to disk, when disable_nonboot_cpus() calls _cpu_down() (on x86_64).

Caused due to Oleg's patch http://lkml.org/lkml/2007/4/13/93.

Agreed that most of the time a kthread_create(p) is followed by a kthread_bind(p), in which case the assertion WARN_ON(p->state != TASK_UNINTERRUPTIBLE) makes sense.

But, in cpu hotplug case, we need to rebind the stop_machine_run thread from the cpu which has just been offlined to any online cpu. (kernel/cpu.c line 180)
At this point, the thread would be in TASK_INTERRUPTIBLE waiting for us to call a kthread_stop on it.(kernel/kthread.c line 161)

Quoted 3 lines

We only need to ensure in kthread_bind that the task which is being bound is not running or exiting. Doesn't matter if it's sleeping in TASK_INTERRUPTIBLE or TASK_UNINTERRUPTIBLE state.

That will probably handle this problem.

However there is a weird interaction with process freezer.

The process freezer can come in and wake up a kernel thread to encourage it to call try_to_freeze_process while it is waiting to be bound.

How do we handle that evil race?

Eric

#4 lore raw
In reply to Gautham R Shenoy
Quoted 12 lines · Gautham R Shenoy
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Thu, 26 Apr 2007 01:10:21 +0200 "Rafael J. Wysocki" <[email protected]> wrote:

Hi,

The BUG_ON in khthread_bind (line 165 in kthread.c) triggers for me during attempted suspend to disk, when disable_nonboot_cpus() calls _cpu_down() (on x86_64).

Caused due to Oleg's patch http://lkml.org/lkml/2007/4/13/93.

Quoted 7 lines

Agreed that most of the time a kthread_create(p) is followed by a kthread_bind(p), in which case the assertion WARN_ON(p->state != TASK_UNINTERRUPTIBLE) makes sense.

But, in cpu hotplug case, we need to rebind the stop_machine_run thread from the cpu which has just been offlined to any online cpu. (kernel/cpu.c line 180)

I can't understand why do we need to re-bind this thread. We are doing kthread_stop()->wake_up() below, at this point move_task_off_dead_cpu() has already cared about this task, no?

Quoted 3 lines

We only need to ensure in kthread_bind that the task which is being bound is not running or exiting. Doesn't matter if it's sleeping in TASK_INTERRUPTIBLE or TASK_UNINTERRUPTIBLE state.

We need to ensure that this task can't be woken after return from wait_task_inactive(k), otherwise set_task_cpu() after that is not safe.

TASK_INTERRUPTIBLE doesn't protect us from freezing.

Couldn't we just remove this kthread_bind() ?

Oleg.

#5 lore raw
Quoted 9 lines · Oleg Nesterov

I can't understand why do we need to re-bind this thread. We are doing kthread_stop()->wake_up() below, at this point move_task_off_dead_cpu() has already cared about this task, no?

Quoted 3 lines

We only need to ensure in kthread_bind that the task which is being bound is not running or exiting. Doesn't matter if it's sleeping in TASK_INTERRUPTIBLE or TASK_UNINTERRUPTIBLE state.

Quoted 6 lines

We need to ensure that this task can't be woken after return from wait_task_inactive(k), otherwise set_task_cpu() after that is not safe.

TASK_INTERRUPTIBLE doesn't protect us from freezing.

Couldn't we just remove this kthread_bind() ?

Fair enough. We are anyway kthread_stop()ping other per-cpu kernel threads after move_task_off_dead_cpu(), so we can do it with the stop_machine_run thread as well.

I just checked with Vatsa if there was any subtle reason why they had put in the kthread_bind() in cpu.c. Vatsa cannot seem to recollect any and I can't see any. So let us just remove the kthread_bind.

---
 kernel/cpu.c |    4 ----
 1 files changed, 4 deletions(-)
kernel/cpu.c +0 -4
Index: linux-2.6.21-rc7/kernel/cpu.c
===================================================================
@@ -176,10 +176,6 @@ static int _cpu_down(unsigned int cpu, i
 	/* This actually kills the CPU. */
 	__cpu_die(cpu);
 
-	/* Move it here so it can run. */
-	kthread_bind(p, get_cpu());
-	put_cpu();
-
 	/* CPU is completely dead: tell everyone.  Too late to complain. */
 	if (raw_notifier_call_chain(&cpu_chain, CPU_DEAD | mod,
 				    hcpu) == NOTIFY_BAD)
Signature · 5 lines
Gautham R Shenoy
Linux Technology Center
IBM India.
"Freedom comes with a price tag of responsibility, which is still a bargain,
because Freedom is priceless!"
#6 lore raw
Quoted 14 lines · On Thu, 26 Apr 2007 18:28:38 +0530 Gautham R Shenoy <[email protected]> wrote:

I just checked with Vatsa if there was any subtle reason why they had put in the kthread_bind() in cpu.c. Vatsa cannot seem to recollect any and I can't see any. So let us just remove the kthread_bind.

---
 kernel/cpu.c |    4 ----
 1 files changed, 4 deletions(-)

Index: linux-2.6.21-rc7/kernel/cpu.c
===================================================================

kernel/cpu.c static int _cpu_down(unsigned int cpu, i · line 181

Quoted 10 lines
kernel/cpu.c +0 -4
@@ -176,10 +176,6 @@ static int _cpu_down(unsigned int cpu, i
 	/* This actually kills the CPU. */
 	__cpu_die(cpu);
 
-	/* Move it here so it can run. */
-	kthread_bind(p, get_cpu());
-	put_cpu();
-
 	/* CPU is completely dead: tell everyone.  Too late to complain. */
 	if (raw_notifier_call_chain(&cpu_chain, CPU_DEAD | mod,
 				    hcpu) == NOTIFY_BAD)

So I cooked up a changelog and queued up the diff. But I have an uneasy feeling that things are getting a bit close to guesswork here.

We have a huge amount of change pending in the kthread/workqueue/freezer area, partly because I decided not to merge most of the workqueue changes into 2.6.21.

It'd be good if people could take some time to sit down and re-review the code which we presently have. I plan on sending it all off for 2.6.22 and there might be some glitches but it seems to have a good track record so far.

#7 lore raw

PATCH 1/1 add-suspend-related-notifications-for-cpu-hotplug-cleanup

Quoted 9 lines · Andrew Morton

We have a huge amount of change pending in the kthread/workqueue/freezer area, partly because I decided not to merge most of the workqueue changes into 2.6.21.

It'd be good if people could take some time to sit down and re-review the code which we presently have. I plan on sending it all off for 2.6.22 and there might be some glitches but it seems to have a good track record so far.

Oops. I completely misread the ->cpus_allowed check in try_to_wake_up(), and so workqueue_cpu_callback() needs a simple fix: CPU_UP_CANCELED should rebind cwq->thread. Or, better, we should not bind the task until CPU_ONLINE.

I'll send the patch soon, but it conflicts with Rafael's

add-suspend-related-notifications-for-cpu-hotplug.patch

, so I am sending this cleanup first to make them independent.

Actually, I believe almost all subsystems could be simplified as well, most of them don't care about CPU_TASKS_FROZEN bit.

kernel/workqueue.c renamed from kernel/workqueue.c~1_FROZEN +2 -5
@@ -747,6 +747,8 @@ static int __devinit workqueue_cpu_callb
 	struct cpu_workqueue_struct *cwq;
 	struct workqueue_struct *wq;
 
+	action &= ~CPU_TASKS_FROZEN;
+
 	switch (action) {
 	case CPU_LOCK_ACQUIRE:
 		mutex_lock(&workqueue_mutex);
@@ -757,7 +759,6 @@ static int __devinit workqueue_cpu_callb
 		return NOTIFY_OK;
 
 	case CPU_UP_PREPARE:
-	case CPU_UP_PREPARE_FROZEN:
 		cpu_set(cpu, cpu_populated_map);
 	}
 
@@ -766,23 +767,19 @@ static int __devinit workqueue_cpu_callb
 
 		switch (action) {
 		case CPU_UP_PREPARE:
-		case CPU_UP_PREPARE_FROZEN:
 			if (!create_workqueue_thread(cwq, cpu))
 				break;
 			printk(KERN_ERR "workqueue for %i failed\n", cpu);
 			return NOTIFY_BAD;
 
 		case CPU_ONLINE:
-		case CPU_ONLINE_FROZEN:
 			wake_up_process(cwq->thread);
 			break;
 
 		case CPU_UP_CANCELED:
-		case CPU_UP_CANCELED_FROZEN:
 			if (cwq->thread)
 				wake_up_process(cwq->thread);
 		case CPU_DEAD:
-		case CPU_DEAD_FROZEN:
 			cleanup_workqueue_thread(cwq, cpu);
 			break;
 		}
#8 lore raw

libata-core.c: unsafe cancel_delayed_work() usage?

In reply to Andrew Morton
Quoted 5 lines · Andrew Morton

It'd be good if people could take some time to sit down and re-review the code which we presently have. I plan on sending it all off for 2.6.22 and there might be some glitches but it seems to have a good track record so far.

Alan, the usage of cancel_delayed_work() in drivers/ata/libata-core.c looks suspicious to me, both ->hotplug_task and ->port_task can re-arm themselves, so cancel_delayed_work + flush_workqueue (or cancel_work_sync) is not enough.

Could you confirm/nack my understanding?

Oleg.

#9 lore raw

Re: libata-core.c: unsafe cancel_delayed_work() usage?

Quoted 11 lines · Oleg Nesterov
Quoted 4 lines · Andrew Morton

It'd be good if people could take some time to sit down and re-review the code which we presently have. I plan on sending it all off for 2.6.22 and there might be some glitches but it seems to have a good track record so far.

Alan, the usage of cancel_delayed_work() in drivers/ata/libata-core.c looks suspicious to me, both ->hotplug_task and ->port_task can re-arm themselves, so cancel_delayed_work + flush_workqueue (or cancel_work_sync) is not enough.

Could you confirm/nack my understanding?

For all rearming workqueues, one must use cancel-rearming-blahblah.

Jeff

#10 lore raw

Re: libata-core.c: unsafe cancel_delayed_work() usage?

Quoted 9 lines · Jeff Garzik
Quoted 7 lines · Oleg Nesterov

Alan, the usage of cancel_delayed_work() in drivers/ata/libata-core.c looks suspicious to me, both ->hotplug_task and ->port_task can re-arm themselves,
so cancel_delayed_work + flush_workqueue (or cancel_work_sync) is not enough.

Quoted 3 lines
Quoted 1 line

Could you confirm/nack my understanding?

For all rearming workqueues, one must use cancel-rearming-blahblah.

Well, yes and no. cancel_rearming_delayed_work() requires that dwork re-arms itself unconditionally, otherwise it just hangs (actually, the patch to fix this shortcoming is ready). However, ata_pio_task() for example, re-arms only when ATA_BUSY.

So it is very possible that the code is correct, if the caller of ata_port_flush_task() ensures that ATA_BUSY is not possible. Otherwise we should change cancel_rearming_delayed_work() and use it.

Oleg.

#11 lore raw

kernel/relay.c: a strange usage of delayed_work

In reply to Andrew Morton

relay_switch_subbuf() does schedule_delayed_work(&buf->wake_readers, 1), wakeup_readers() only does wake_up_interruptible() and nothing more.

Why can't we use a plain timer for this?

In any case, this "wake_up ->read_wait after a minimal possible delay" looks somewhat strange to me, could you explain? just curious.

Oleg.

#12 lore raw

Re: kernel/relay.c: a strange usage of delayed_work

Quoted 8 lines · Oleg Nesterov

relay_switch_subbuf() does schedule_delayed_work(&buf->wake_readers, 1), wakeup_readers() only does wake_up_interruptible() and nothing more.

Why can't we use a plain timer for this?

In any case, this "wake_up ->read_wait after a minimal possible delay" looks somewhat strange to me, could you explain? just curious.

The reason it's done that way is that if the event that causes the relay_switch_subbuf() happens to be an event logged from schedule(), and we directly call wake_up_interruptible() at that point, we lock up the machine because it ends up back in schedule(). Deferring it avoids the problem.

I don't see any problem with using a plain timer instead - I'll work up a patch to make that change.

Tom

#13 lore raw
In reply to Andrew Morton
Quoted 9 lines · Andrew Morton
Quoted 5 lines · On Thu, 26 Apr 2007 01:10:21 +0200 "Rafael J. Wysocki" <[email protected]> wrote:

Hi,

The BUG_ON in khthread_bind (line 165 in kthread.c) triggers for me during attempted suspend to disk, when disable_nonboot_cpus() calls _cpu_down() (on x86_64).

I guess the backtrace would be pretty important here.

Yes, I didn't have the time to collect one yestarday.