Skip to contentSkip to conversation

2.6-bk: cpu hotplug + preempt = smp_processor_id warnings galore

PATCH New Nathan Lynch 5 participants 13 messages Feb 11, 2005 – Feb 16, 2005 Workqueue

lore
Share & tools
lore
#0 lore raw

Hi-

With 2.6.11-rc3-bk7 on ppc64 I am seeing lots of smp_processor_id warnings whenever I hotplug cpus:

# echo 0 > /sys/devices/system/cpu/cpu1/online 
cpu 1 (hwid 1) Ready to die...
BUG: using smp_processor_id() in preemptible [00000001] code:
ksoftirqd/1/5
caller is .ksoftirqd+0xbc/0x1f8
Call Trace:
[c0000000fffbbce0] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000fffbbd60] [c0000000001c9f1c] .smp_processor_id+0x154/0x168
[c0000000fffbbe20] [c00000000005f414] .ksoftirqd+0xbc/0x1f8
[c0000000fffbbed0] [c0000000000764cc] .kthread+0x128/0x134
[c0000000fffbbf90] [c000000000014248] .kernel_thread+0x4c/0x6c

I believe the above warning is caused by the local_softirq_pending call on a "foreign" cpu before ksoftirqd/1 has been stopped. Looking at the code, I think this doesn't indicate a real bug, but it would be better if ksoftirqd didn't check local_softirq_pending after it's been kicked off its cpu, right?

# echo 1 > /sys/devices/system/cpu/cpu1/online 
BUG: using smp_processor_id() in preemptible [00000001] code:
swapper/0
caller is .dedicated_idle+0x68/0x22c
Call Trace:
[c0000000fffafc50] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000fffafcd0] [c0000000001c9f1c] .smp_processor_id+0x154/0x168
[c0000000fffafd90] [c00000000000f998] .dedicated_idle+0x68/0x22c
[c0000000fffafe80] [c00000000000fce8] .cpu_idle+0x34/0x4c
[c0000000fffaff00] [c00000000003a744] .start_secondary+0x10c/0x150
[c0000000fffaff90] [c00000000000bd28] .enable_64b_mode+0x0/0x28

Should ppc64 simply use _smp_processor_id() in its idle loop code (like i386)?

If I online and offline cpus rapidly enough I can eventually get the following:

printk: 49 messages suppressed.

BUG: using smp_processor_id() in preemptible [00000001] code:
events/3/1262
caller is .cache_reap+0x21c/0x2b8
Call Trace:

[c0000000ed67bb90] [ffffffffffffffff] 0xffffffffffffffff (unreliable) [c0000000ed67bc10] [c0000000001c9f1c] .smp_processor_id+0x154/0x168 [c0000000ed67bcd0] [c0000000000938e8] .cache_reap+0x21c/0x2b8 [c0000000ed67bda0] [c00000000006f4bc] .worker_thread+0x230/0x310 [c0000000ed67bed0] [c0000000000764cc] .kthread+0x128/0x134 [c0000000ed67bf90] [c000000000014248] .kernel_thread+0x4c/0x6c

And this will repeat over and over even after I stop hotplugging cpus... from the same events thread so I think it's somehow gotten "stuck"?

Anything I can do to further debug?

Nathan

#1 lore raw
Quoted 59 lines · Nathan Lynch

Hi-

With 2.6.11-rc3-bk7 on ppc64 I am seeing lots of smp_processor_id warnings whenever I hotplug cpus:

# echo 0 > /sys/devices/system/cpu/cpu1/online cpu 1 (hwid 1) Ready to die...
BUG: using smp_processor_id() in preemptible [00000001] code:
ksoftirqd/1/5
caller is .ksoftirqd+0xbc/0x1f8
Call Trace:
[c0000000fffbbce0] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000fffbbd60] [c0000000001c9f1c] .smp_processor_id+0x154/0x168
[c0000000fffbbe20] [c00000000005f414] .ksoftirqd+0xbc/0x1f8
[c0000000fffbbed0] [c0000000000764cc] .kthread+0x128/0x134
[c0000000fffbbf90] [c000000000014248] .kernel_thread+0x4c/0x6c

I believe the above warning is caused by the local_softirq_pending call on a "foreign" cpu before ksoftirqd/1 has been stopped. Looking at the code, I think this doesn't indicate a real bug, but it would be better if ksoftirqd didn't check local_softirq_pending after it's been kicked off its cpu, right?

# echo 1 > /sys/devices/system/cpu/cpu1/online BUG: using smp_processor_id() in preemptible [00000001] code:
swapper/0
caller is .dedicated_idle+0x68/0x22c
Call Trace:
[c0000000fffafc50] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000fffafcd0] [c0000000001c9f1c] .smp_processor_id+0x154/0x168
[c0000000fffafd90] [c00000000000f998] .dedicated_idle+0x68/0x22c
[c0000000fffafe80] [c00000000000fce8] .cpu_idle+0x34/0x4c
[c0000000fffaff00] [c00000000003a744] .start_secondary+0x10c/0x150
[c0000000fffaff90] [c00000000000bd28] .enable_64b_mode+0x0/0x28

Should ppc64 simply use _smp_processor_id() in its idle loop code (like i386)?

If I online and offline cpus rapidly enough I can eventually get the following:

printk: 49 messages suppressed.

BUG: using smp_processor_id() in preemptible [00000001] code:
events/3/1262
caller is .cache_reap+0x21c/0x2b8
Call Trace:

[c0000000ed67bb90] [ffffffffffffffff] 0xffffffffffffffff (unreliable) [c0000000ed67bc10] [c0000000001c9f1c] .smp_processor_id+0x154/0x168 [c0000000ed67bcd0] [c0000000000938e8] .cache_reap+0x21c/0x2b8 [c0000000ed67bda0] [c00000000006f4bc] .worker_thread+0x230/0x310 [c0000000ed67bed0] [c0000000000764cc] .kthread+0x128/0x134 [c0000000ed67bf90] [c000000000014248] .kernel_thread+0x4c/0x6c

And this will repeat over and over even after I stop hotplugging cpus... from the same events thread so I think it's somehow gotten "stuck"?

Anything I can do to further debug?

Quoted 9 lines

Nathan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hi!
Use get_cpu() (It disables preemption) or __smp_processor_id () (on a smp).

Matthias-Christian Ott

#2 lore raw
Quoted 4 lines · Matthias-Christian Ott
Quoted 2 lines · Nathan Lynch

With 2.6.11-rc3-bk7 on ppc64 I am seeing lots of smp_processor_id warnings whenever I hotplug cpus:

...

Quoted 2 lines

Use get_cpu() (It disables preemption) or __smp_processor_id () (on a smp).

It's not necessarily that simple (ok, maybe the idle loop warning is). But at least one of the warnings I listed appears to be caused by a kernel thread that is normally bound to a particular cpu trying to do normal processing on another cpu before it has stopped. Injudicious use of __smp_processor_id or get_cpu in this case would only obscure the problem.

Nathan

#3 lore raw
In reply to Nathan Lynch
Quoted 15 lines · Nathan Lynch

With 2.6.11-rc3-bk7 on ppc64 I am seeing lots of smp_processor_id warnings whenever I hotplug cpus:

# echo 0 > /sys/devices/system/cpu/cpu1/online 
cpu 1 (hwid 1) Ready to die...
BUG: using smp_processor_id() in preemptible [00000001] code:
ksoftirqd/1/5
caller is .ksoftirqd+0xbc/0x1f8
Call Trace:
[c0000000fffbbce0] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000fffbbd60] [c0000000001c9f1c] .smp_processor_id+0x154/0x168
[c0000000fffbbe20] [c00000000005f414] .ksoftirqd+0xbc/0x1f8
[c0000000fffbbed0] [c0000000000764cc] .kthread+0x128/0x134
[c0000000fffbbf90] [c000000000014248] .kernel_thread+0x4c/0x6c
Quoted 5 lines

I believe the above warning is caused by the local_softirq_pending call on a "foreign" cpu before ksoftirqd/1 has been stopped. Looking at the code, I think this doesn't indicate a real bug, but it would be better if ksoftirqd didn't check local_softirq_pending after it's been kicked off its cpu, right?

How about;

Index: linux-2.6.11-rc3-mm2/kernel/softirq.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.11-rc3-mm2/kernel/softirq.c,v retrieving revision 1.1.1.1

kernel/softirq.c +8 -4
diff -u -p -B -r1.1.1.1 softirq.c
@@ -355,8 +355,12 @@ static int ksoftirqd(void * __bind_cpu)
 	set_current_state(TASK_INTERRUPTIBLE);
 
 	while (!kthread_should_stop()) {
-		if (!local_softirq_pending())
+		preempt_disable();
+		if (!local_softirq_pending()) {
+			preempt_enable_no_resched();
 			schedule();
+			preempt_disable();
+		}
 
 		__set_current_state(TASK_RUNNING);
 
@@ -364,14 +368,14 @@ static int ksoftirqd(void * __bind_cpu)
 			/* Preempt disable stops cpu going offline.
 			   If already offline, we'll be on wrong CPU:
 			   don't process */
-			preempt_disable();
 			if (cpu_is_offline((long)__bind_cpu))
 				goto wait_to_die;
 			do_softirq();
-			preempt_enable();
+			preempt_enable_no_resched();
 			cond_resched();
+			preempt_disable();
 		}
-
+		preempt_enable();
 		set_current_state(TASK_INTERRUPTIBLE);
 	}
 	__set_current_state(TASK_RUNNING);
Quoted 12 lines
# echo 1 > /sys/devices/system/cpu/cpu1/online 
BUG: using smp_processor_id() in preemptible [00000001] code:
swapper/0
caller is .dedicated_idle+0x68/0x22c
Call Trace:
[c0000000fffafc50] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000fffafcd0] [c0000000001c9f1c] .smp_processor_id+0x154/0x168
[c0000000fffafd90] [c00000000000f998] .dedicated_idle+0x68/0x22c
[c0000000fffafe80] [c00000000000fce8] .cpu_idle+0x34/0x4c
[c0000000fffaff00] [c00000000003a744] .start_secondary+0x10c/0x150
[c0000000fffaff90] [c00000000000bd28] .enable_64b_mode+0x0/0x28
Quoted 2 lines

Should ppc64 simply use _smp_processor_id() in its idle loop code (like i386)?

I would say so, it's definitely safe.

Quoted 15 lines

If I online and offline cpus rapidly enough I can eventually get the following:

printk: 49 messages suppressed.

BUG: using smp_processor_id() in preemptible [00000001] code:
events/3/1262
caller is .cache_reap+0x21c/0x2b8
Call Trace:

[c0000000ed67bb90] [ffffffffffffffff] 0xffffffffffffffff (unreliable) [c0000000ed67bc10] [c0000000001c9f1c] .smp_processor_id+0x154/0x168 [c0000000ed67bcd0] [c0000000000938e8] .cache_reap+0x21c/0x2b8 [c0000000ed67bda0] [c00000000006f4bc] .worker_thread+0x230/0x310 [c0000000ed67bed0] [c0000000000764cc] .kthread+0x128/0x134 [c0000000ed67bf90] [c000000000014248] .kernel_thread+0x4c/0x6c

Quoted 3 lines

And this will repeat over and over even after I stop hotplugging cpus... from the same events thread so I think it's somehow gotten "stuck"?

Hmm this should be covered by the local_bh_disable() in softirq processing.

#4 lore raw
Quoted 32 lines · Zwane Mwaikambo

How about;

Index: linux-2.6.11-rc3-mm2/kernel/softirq.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.11-rc3-mm2/kernel/softirq.c,v retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 softirq.c

kernel/softirq.c +5 -2
@@ -355,8 +355,12 @@ static int ksoftirqd(void * __bind_cpu)
 	set_current_state(TASK_INTERRUPTIBLE);
 
 	while (!kthread_should_stop()) {
-		if (!local_softirq_pending())
+		preempt_disable();
+		if (!local_softirq_pending()) {
+			preempt_enable_no_resched();
 			schedule();
+			preempt_disable();
+		}
 
 		__set_current_state(TASK_RUNNING);
 
@@ -364,14 +368,14 @@ static int ksoftirqd(void * __bind_cpu)
 			/* Preempt disable stops cpu going offline.
 			   If already offline, we'll be on wrong CPU:
 			   don't process */
-			preempt_disable();
 			if (cpu_is_offline((long)__bind_cpu))
 				goto wait_to_die;
 			do_softirq();

kernel/softirq.c static int ksoftirqd(void * __bind_cpu) · line 381

Quoted 10 lines
kernel/softirq.c +3 -2
-			preempt_enable();
+			preempt_enable_no_resched();
 			cond_resched();
+			preempt_disable();
 		}
-
+		preempt_enable();
 		set_current_state(TASK_INTERRUPTIBLE);
 	}
 	__set_current_state(TASK_RUNNING);

Works ok here.

It looks as if we need to explicitly bind worker threads to a newly onlined cpu. This gets rid of the smp_processor_id warnings from cache_reap. Adding a little more instrumentation to the debug smp_processor_id showed that new worker threads were actually running on the wrong cpu...

Does this look ok?

kernel/workqueue.c +3 -1
Index: linux-2.6.11-rc4-bk2/kernel/workqueue.c
===================================================================
@@ -485,8 +485,10 @@
 
 	case CPU_ONLINE:
 		/* Kick off worker threads. */
-		list_for_each_entry(wq, &workqueues, list)
+		list_for_each_entry(wq, &workqueues, list) {
+			kthread_bind(wq->cpu_wq[hotcpu].thread, hotcpu);
 			wake_up_process(wq->cpu_wq[hotcpu].thread);
+		}
 		break;
 
 	case CPU_UP_CANCELED:
Acked-by
#5 lore raw
Quoted 9 lines · Nathan Lynch

Works ok here.

It looks as if we need to explicitly bind worker threads to a newly onlined cpu. This gets rid of the smp_processor_id warnings from cache_reap. Adding a little more instrumentation to the debug smp_processor_id showed that new worker threads were actually running on the wrong cpu...

Does this look ok?

indeed - looks much better than the 'turn off the warning' solution.

Ingo

#6 lore raw

PATCH 1/1 kthread_bind new worker threads when onlining cpu

Hi Andrew-

Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Nathan Lynch

It looks as if we need to explicitly bind worker threads to a newly onlined cpu. This gets rid of the smp_processor_id warnings from cache_reap. Adding a little more instrumentation to the debug smp_processor_id showed that new worker threads were actually running on the wrong cpu...

Quoted 5 lines
Quoted 1 line

Does this look ok?

indeed - looks much better than the 'turn off the warning' solution.

We weren't binding new worker threads to their cpu when onlining. Using preempt and the debug version of smp_processor_id found this.

kernel/workqueue.c +3 -1
Index: linux-2.6.11-rc4-bk2/kernel/workqueue.c
===================================================================
@@ -485,8 +485,10 @@
 
 	case CPU_ONLINE:
 		/* Kick off worker threads. */
-		list_for_each_entry(wq, &workqueues, list)
+		list_for_each_entry(wq, &workqueues, list) {
+			kthread_bind(wq->cpu_wq[hotcpu].thread, hotcpu);
 			wake_up_process(wq->cpu_wq[hotcpu].thread);
+		}
 		break;
 
 	case CPU_UP_CANCELED:
#7 lore raw

PATCH 1/1 Run softirqs on proper processor on offline

Ensure that we only offline the processor when it's safe and never run softirqs in another processor's ksoftirqd context. This also gets rid of the warnings in ksoftirqd on cpu offline.

Index: linux-2.6.11-rc3-mm2/kernel/softirq.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.11-rc3-mm2/kernel/softirq.c,v retrieving revision 1.1.1.1

kernel/softirq.c +8 -4
diff -u -p -B -r1.1.1.1 softirq.c
@@ -355,8 +355,12 @@ static int ksoftirqd(void * __bind_cpu)
 	set_current_state(TASK_INTERRUPTIBLE);
 
 	while (!kthread_should_stop()) {
-		if (!local_softirq_pending())
+		preempt_disable();
+		if (!local_softirq_pending()) {
+			preempt_enable_no_resched();
 			schedule();
+			preempt_disable();
+		}
 
 		__set_current_state(TASK_RUNNING);
 
@@ -364,14 +368,14 @@ static int ksoftirqd(void * __bind_cpu)
 			/* Preempt disable stops cpu going offline.
 			   If already offline, we'll be on wrong CPU:
 			   don't process */
-			preempt_disable();
 			if (cpu_is_offline((long)__bind_cpu))
 				goto wait_to_die;
 			do_softirq();
-			preempt_enable();
+			preempt_enable_no_resched();
 			cond_resched();
+			preempt_disable();
 		}
-
+		preempt_enable();
 		set_current_state(TASK_INTERRUPTIBLE);
 	}
 	__set_current_state(TASK_RUNNING);
Acked-by on v1 1/1
#8 lore raw

Re: [PATCH] Run softirqs on proper processor on offline

Quoted 5 lines · Zwane Mwaikambo

Ensure that we only offline the processor when it's safe and never run softirqs in another processor's ksoftirqd context. This also gets rid of the warnings in ksoftirqd on cpu offline.

looks good.

Ingo

#9 lore raw

Re: [PATCH] Run softirqs on proper processor on offline

In reply to Zwane Mwaikambo
Quoted 4 lines · Zwane Mwaikambo

Ensure that we only offline the processor when it's safe and never run
softirqs in another processor's ksoftirqd context. This also gets rid of the warnings in ksoftirqd on cpu offline.

I don't get it. ksoftirqd is pinned to its cpu, so why does any of this matter?

#10 lore raw

Re: [PATCH] Run softirqs on proper processor on offline

Quoted 8 lines · Andrew Morton
Quoted 4 lines · Zwane Mwaikambo

Ensure that we only offline the processor when it's safe and never run
softirqs in another processor's ksoftirqd context. This also gets rid of the warnings in ksoftirqd on cpu offline.

I don't get it. ksoftirqd is pinned to its cpu, so why does any of this matter?

We take down ksoftirqds at CPU_DEAD time, so there is a brief period whereupon there is a ksoftirqd thread for an offline processor, it is at this point that ->cpus_allowed won't have it pinned anymore. An online processor would then take down that ksoftirqd and exit it.

#11 lore raw
In reply to Nathan Lynch
Quoted 7 lines · Nathan Lynch

It looks as if we need to explicitly bind worker threads to a newly onlined cpu. This gets rid of the smp_processor_id warnings from cache_reap. Adding a little more instrumentation to the debug smp_processor_id showed that new worker threads were actually running on the wrong cpu...

Does this look ok?

Yeah, does that patch suffice for all the warnings?

#12 lore raw
Quoted 11 lines · Zwane Mwaikambo
Quoted 7 lines · Nathan Lynch

It looks as if we need to explicitly bind worker threads to a newly onlined cpu. This gets rid of the smp_processor_id warnings from cache_reap. Adding a little more instrumentation to the debug smp_processor_id showed that new worker threads were actually running on the wrong cpu...

Does this look ok?

Yeah, does that patch suffice for all the warnings?

Nope, ksoftirqd still requires your patch in order to kill the warnings there.

Nathan