2.6-bk: cpu hotplug + preempt = smp_processor_id warnings galore
Hi-
With 2.6.11-rc3-bk7 on ppc64 I am seeing lots of smp_processor_id warnings whenever I hotplug cpus:
# echo 0 > /sys/devices/system/cpu/cpu1/online cpu 1 (hwid 1) Ready to die... BUG: using smp_processor_id() in preemptible [00000001] code: ksoftirqd/1/5 caller is .ksoftirqd+0xbc/0x1f8 Call Trace: [c0000000fffbbce0] [ffffffffffffffff] 0xffffffffffffffff (unreliable) [c0000000fffbbd60] [c0000000001c9f1c] .smp_processor_id+0x154/0x168 [c0000000fffbbe20] [c00000000005f414] .ksoftirqd+0xbc/0x1f8 [c0000000fffbbed0] [c0000000000764cc] .kthread+0x128/0x134 [c0000000fffbbf90] [c000000000014248] .kernel_thread+0x4c/0x6c
I believe the above warning is caused by the local_softirq_pending call on a "foreign" cpu before ksoftirqd/1 has been stopped. Looking at the code, I think this doesn't indicate a real bug, but it would be better if ksoftirqd didn't check local_softirq_pending after it's been kicked off its cpu, right?
# echo 1 > /sys/devices/system/cpu/cpu1/online BUG: using smp_processor_id() in preemptible [00000001] code: swapper/0 caller is .dedicated_idle+0x68/0x22c Call Trace: [c0000000fffafc50] [ffffffffffffffff] 0xffffffffffffffff (unreliable) [c0000000fffafcd0] [c0000000001c9f1c] .smp_processor_id+0x154/0x168 [c0000000fffafd90] [c00000000000f998] .dedicated_idle+0x68/0x22c [c0000000fffafe80] [c00000000000fce8] .cpu_idle+0x34/0x4c [c0000000fffaff00] [c00000000003a744] .start_secondary+0x10c/0x150 [c0000000fffaff90] [c00000000000bd28] .enable_64b_mode+0x0/0x28
Should ppc64 simply use _smp_processor_id() in its idle loop code (like i386)?
If I online and offline cpus rapidly enough I can eventually get the following:
printk: 49 messages suppressed.
BUG: using smp_processor_id() in preemptible [00000001] code: events/3/1262 caller is .cache_reap+0x21c/0x2b8 Call Trace:
[c0000000ed67bb90] [ffffffffffffffff] 0xffffffffffffffff (unreliable) [c0000000ed67bc10] [c0000000001c9f1c] .smp_processor_id+0x154/0x168 [c0000000ed67bcd0] [c0000000000938e8] .cache_reap+0x21c/0x2b8 [c0000000ed67bda0] [c00000000006f4bc] .worker_thread+0x230/0x310 [c0000000ed67bed0] [c0000000000764cc] .kthread+0x128/0x134 [c0000000ed67bf90] [c000000000014248] .kernel_thread+0x4c/0x6c
And this will repeat over and over even after I stop hotplugging cpus... from the same events thread so I think it's somehow gotten "stuck"?
Anything I can do to further debug?
Nathan
Quoted 59 lines · Nathan Lynch
Hi-
With 2.6.11-rc3-bk7 on ppc64 I am seeing lots of
smp_processor_idwarnings whenever I hotplug cpus:# echo 0 > /sys/devices/system/cpu/cpu1/online cpu 1 (hwid 1) Ready to die... BUG: using smp_processor_id() in preemptible [00000001] code: ksoftirqd/1/5 caller is .ksoftirqd+0xbc/0x1f8 Call Trace: [c0000000fffbbce0] [ffffffffffffffff] 0xffffffffffffffff (unreliable) [c0000000fffbbd60] [c0000000001c9f1c] .smp_processor_id+0x154/0x168 [c0000000fffbbe20] [c00000000005f414] .ksoftirqd+0xbc/0x1f8 [c0000000fffbbed0] [c0000000000764cc] .kthread+0x128/0x134 [c0000000fffbbf90] [c000000000014248] .kernel_thread+0x4c/0x6cI believe the above warning is caused by the
local_softirq_pendingcall on a "foreign" cpu before ksoftirqd/1 has been stopped. Looking at the code, I think this doesn't indicate a real bug, but it would be better if ksoftirqd didn't checklocal_softirq_pendingafter it's been kicked off its cpu, right?# echo 1 > /sys/devices/system/cpu/cpu1/online BUG: using smp_processor_id() in preemptible [00000001] code: swapper/0 caller is .dedicated_idle+0x68/0x22c Call Trace: [c0000000fffafc50] [ffffffffffffffff] 0xffffffffffffffff (unreliable) [c0000000fffafcd0] [c0000000001c9f1c] .smp_processor_id+0x154/0x168 [c0000000fffafd90] [c00000000000f998] .dedicated_idle+0x68/0x22c [c0000000fffafe80] [c00000000000fce8] .cpu_idle+0x34/0x4c [c0000000fffaff00] [c00000000003a744] .start_secondary+0x10c/0x150 [c0000000fffaff90] [c00000000000bd28] .enable_64b_mode+0x0/0x28Should ppc64 simply use
_smp_processor_id()in its idle loop code (like i386)?If I online and offline cpus rapidly enough I can eventually get the following:
printk: 49 messages suppressed.
BUG: using smp_processor_id() in preemptible [00000001] code: events/3/1262 caller is .cache_reap+0x21c/0x2b8 Call Trace:[
c0000000ed67bb90] [ffffffffffffffff]0xffffffffffffffff(unreliable) [c0000000ed67bc10] [c0000000001c9f1c] .smp_processor_id+0x154/0x168 [c0000000ed67bcd0] [c0000000000938e8] .cache_reap+0x21c/0x2b8 [c0000000ed67bda0] [c00000000006f4bc] .worker_thread+0x230/0x310 [c0000000ed67bed0] [c0000000000764cc] .kthread+0x128/0x134 [c0000000ed67bf90] [c000000000014248] .kernel_thread+0x4c/0x6cAnd this will repeat over and over even after I stop hotplugging cpus... from the same events thread so I think it's somehow gotten "stuck"?
Anything I can do to further debug?
Quoted 9 lines
Nathan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Hi!
Use get_cpu() (It disables preemption) or __smp_processor_id () (on a smp).
Matthias-Christian Ott
Quoted 4 lines · Matthias-Christian Ott
Quoted 2 lines · Nathan Lynch
With 2.6.11-rc3-bk7 on ppc64 I am seeing lots of
smp_processor_idwarnings whenever I hotplug cpus:
...
Quoted 2 lines
Use
get_cpu()(It disables preemption) or__smp_processor_id() (on a smp).
It's not necessarily that simple (ok, maybe the idle loop warning is). But at least one of the warnings I listed appears to be caused by a kernel thread that is normally bound to a particular cpu trying to do normal processing on another cpu before it has stopped. Injudicious use of __smp_processor_id or get_cpu in this case would only obscure the problem.
Nathan
Quoted 15 lines · Nathan Lynch
With 2.6.11-rc3-bk7 on ppc64 I am seeing lots of
smp_processor_idwarnings whenever I hotplug cpus:# echo 0 > /sys/devices/system/cpu/cpu1/online cpu 1 (hwid 1) Ready to die... BUG: using smp_processor_id() in preemptible [00000001] code: ksoftirqd/1/5 caller is .ksoftirqd+0xbc/0x1f8 Call Trace: [c0000000fffbbce0] [ffffffffffffffff] 0xffffffffffffffff (unreliable) [c0000000fffbbd60] [c0000000001c9f1c] .smp_processor_id+0x154/0x168 [c0000000fffbbe20] [c00000000005f414] .ksoftirqd+0xbc/0x1f8 [c0000000fffbbed0] [c0000000000764cc] .kthread+0x128/0x134 [c0000000fffbbf90] [c000000000014248] .kernel_thread+0x4c/0x6c
Quoted 5 lines
I believe the above warning is caused by the
local_softirq_pendingcall on a "foreign" cpu before ksoftirqd/1 has been stopped. Looking at the code, I think this doesn't indicate a real bug, but it would be better if ksoftirqd didn't checklocal_softirq_pendingafter it's been kicked off its cpu, right?
How about;
Index: linux-2.6.11-rc3-mm2/kernel/softirq.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.11-rc3-mm2/kernel/softirq.c,v retrieving revision 1.1.1.1
kernel/softirq.c +8 -4
@@ -355,8 +355,12 @@ static int ksoftirqd(void * __bind_cpu)set_current_state (TASK_INTERRUPTIBLE );while (!kthread_should_stop ()) { -if (!local_softirq_pending ()) +preempt_disable (); +if (!local_softirq_pending ()) { +preempt_enable_no_resched ();schedule (); +preempt_disable (); + }__set_current_state (TASK_RUNNING );
@@ -364,14 +368,14 @@ static int ksoftirqd(void * __bind_cpu)/* Preempt disable stops cpu going offline. If already offline, we'll be on wrong CPU: don't process */ -preempt_disable ();if (cpu_is_offline ((long )__bind_cpu)) goto wait_to_die;do_softirq (); -preempt_enable (); +preempt_enable_no_resched ();cond_resched (); +preempt_disable (); } - +preempt_enable ();set_current_state (TASK_INTERRUPTIBLE ); }__set_current_state (TASK_RUNNING );
Quoted 12 lines
# echo 1 > /sys/devices/system/cpu/cpu1/online BUG: using smp_processor_id() in preemptible [00000001] code: swapper/0 caller is .dedicated_idle+0x68/0x22c Call Trace: [c0000000fffafc50] [ffffffffffffffff] 0xffffffffffffffff (unreliable) [c0000000fffafcd0] [c0000000001c9f1c] .smp_processor_id+0x154/0x168 [c0000000fffafd90] [c00000000000f998] .dedicated_idle+0x68/0x22c [c0000000fffafe80] [c00000000000fce8] .cpu_idle+0x34/0x4c [c0000000fffaff00] [c00000000003a744] .start_secondary+0x10c/0x150 [c0000000fffaff90] [c00000000000bd28] .enable_64b_mode+0x0/0x28
Quoted 2 lines
Should ppc64 simply use
_smp_processor_id()in its idle loop code (like i386)?
I would say so, it's definitely safe.
Quoted 15 lines
If I online and offline cpus rapidly enough I can eventually get the following:
printk: 49 messages suppressed.
BUG: using smp_processor_id() in preemptible [00000001] code: events/3/1262 caller is .cache_reap+0x21c/0x2b8 Call Trace:[
c0000000ed67bb90] [ffffffffffffffff]0xffffffffffffffff(unreliable) [c0000000ed67bc10] [c0000000001c9f1c] .smp_processor_id+0x154/0x168 [c0000000ed67bcd0] [c0000000000938e8] .cache_reap+0x21c/0x2b8 [c0000000ed67bda0] [c00000000006f4bc] .worker_thread+0x230/0x310 [c0000000ed67bed0] [c0000000000764cc] .kthread+0x128/0x134 [c0000000ed67bf90] [c000000000014248] .kernel_thread+0x4c/0x6c
Quoted 3 lines
And this will repeat over and over even after I stop hotplugging cpus... from the same events thread so I think it's somehow gotten "stuck"?
Hmm this should be covered by the local_bh_disable() in softirq processing.
Quoted 32 lines · Zwane Mwaikambo
How about;
Index:
linux-2.6.11-rc3-mm2/kernel/softirq.c
===================================================================
RCS file:/home/cvsroot/linux-2.6.11-rc3-mm2/kernel/softirq.c,vretrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1softirq.ckernel/softirq.c +5
-2@@ -355,8 +355,12 @@ static int ksoftirqd(void * __bind_cpu)set_current_state (TASK_INTERRUPTIBLE );while (!kthread_should_stop ()) { -if (!local_softirq_pending ()) +preempt_disable (); +if (!local_softirq_pending ()) { +preempt_enable_no_resched ();schedule (); +preempt_disable (); + }__set_current_state (TASK_RUNNING );@@ -364,14 +368,14 @@ static int ksoftirqd(void * __bind_cpu)/* Preempt disable stops cpu going offline. If already offline, we'll be on wrong CPU: don't process */ -preempt_disable ();if (cpu_is_offline ((long )__bind_cpu)) goto wait_to_die;do_softirq ();
kernel/softirq.c static int ksoftirqd(void * __bind_cpu) · line 381
Quoted 10 lines
kernel/softirq.c +3
-2-preempt_enable (); +preempt_enable_no_resched ();cond_resched (); +preempt_disable (); } - +preempt_enable ();set_current_state (TASK_INTERRUPTIBLE ); }__set_current_state (TASK_RUNNING );
Works ok here.
It looks as if we need to explicitly bind worker threads to a newly onlined cpu. This gets rid of the smp_processor_id warnings from cache_reap. Adding a little more instrumentation to the debug smp_processor_id showed that new worker threads were actually running on the wrong cpu...
Does this look ok?
kernel/workqueue.c +3 -1
@@ -485,8 +485,10 @@case CPU_ONLINE :/* Kick off worker threads. */ -list_for_each_entry (wq, &workqueues, list) +list_for_each_entry (wq, &workqueues, list) { +kthread_bind (wq->cpu_wq[hotcpu].thread, hotcpu);wake_up_process (wq->cpu_wq[hotcpu].thread); + }break ;case CPU_UP_CANCELED :
Ingo Molnar
Acked-byQuoted 9 lines · Nathan Lynch
Works ok here.
It looks as if we need to explicitly bind worker threads to a newly onlined cpu. This gets rid of the
smp_processor_idwarnings fromcache_reap. Adding a little more instrumentation to the debugsmp_processor_idshowed that new worker threads were actually running on the wrong cpu...Does this look ok?
indeed - looks much better than the 'turn off the warning' solution.
Ingo
PATCH 1/1 kthread_bind new worker threads when onlining cpu
Hi Andrew-
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Nathan Lynch
It looks as if we need to explicitly bind worker threads to a newly onlined cpu. This gets rid of the
smp_processor_idwarnings fromcache_reap. Adding a little more instrumentation to the debugsmp_processor_idshowed that new worker threads were actually running on the wrong cpu...
Quoted 5 lines
Quoted 1 line
Does this look ok?
indeed - looks much better than the 'turn off the warning' solution.
Acked-by Ingo Molnar <[email protected]>
We weren't binding new worker threads to their cpu when onlining. Using preempt and the debug version of smp_processor_id found this.
kernel/workqueue.c +3 -1
@@ -485,8 +485,10 @@case CPU_ONLINE :/* Kick off worker threads. */ -list_for_each_entry (wq, &workqueues, list) +list_for_each_entry (wq, &workqueues, list) { +kthread_bind (wq->cpu_wq[hotcpu].thread, hotcpu);wake_up_process (wq->cpu_wq[hotcpu].thread); + }break ;case CPU_UP_CANCELED :
PATCH 1/1 Run softirqs on proper processor on offline
Ensure that we only offline the processor when it's safe and never run softirqs in another processor's ksoftirqd context. This also gets rid of the warnings in ksoftirqd on cpu offline.
Index: linux-2.6.11-rc3-mm2/kernel/softirq.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.11-rc3-mm2/kernel/softirq.c,v retrieving revision 1.1.1.1
kernel/softirq.c +8 -4
@@ -355,8 +355,12 @@ static int ksoftirqd(void * __bind_cpu)set_current_state (TASK_INTERRUPTIBLE );while (!kthread_should_stop ()) { -if (!local_softirq_pending ()) +preempt_disable (); +if (!local_softirq_pending ()) { +preempt_enable_no_resched ();schedule (); +preempt_disable (); + }__set_current_state (TASK_RUNNING );
@@ -364,14 +368,14 @@ static int ksoftirqd(void * __bind_cpu)/* Preempt disable stops cpu going offline. If already offline, we'll be on wrong CPU: don't process */ -preempt_disable ();if (cpu_is_offline ((long )__bind_cpu)) goto wait_to_die;do_softirq (); -preempt_enable (); +preempt_enable_no_resched ();cond_resched (); +preempt_disable (); } - +preempt_enable ();set_current_state (TASK_INTERRUPTIBLE ); }__set_current_state (TASK_RUNNING );
Re: [PATCH] Run softirqs on proper processor on offline
Quoted 5 lines · Zwane Mwaikambo
Ensure that we only offline the processor when it's safe and never run softirqs in another processor's ksoftirqd context. This also gets rid of the warnings in ksoftirqd on cpu offline.
Signed-off-by Zwane Mwaikambo <[email protected]>
looks good.
Ingo
Re: [PATCH] Run softirqs on proper processor on offline
In reply to Zwane MwaikamboQuoted 4 lines · Zwane Mwaikambo
Ensure that we only offline the processor when it's safe and never run
softirqs in another processor's ksoftirqd context. This also gets rid of the warnings in ksoftirqd on cpu offline.
I don't get it. ksoftirqd is pinned to its cpu, so why does any of this matter?
Re: [PATCH] Run softirqs on proper processor on offline
Quoted 8 lines · Andrew Morton
Quoted 4 lines · Zwane Mwaikambo
Ensure that we only offline the processor when it's safe and never run
softirqs in another processor's ksoftirqd context. This also gets rid of the warnings in ksoftirqd on cpu offline.I don't get it. ksoftirqd is pinned to its cpu, so why does any of this matter?
We take down ksoftirqds at CPU_DEAD time, so there is a brief period whereupon there is a ksoftirqd thread for an offline processor, it is at this point that ->cpus_allowed won't have it pinned anymore. An online processor would then take down that ksoftirqd and exit it.
Quoted 7 lines · Nathan Lynch
It looks as if we need to explicitly bind worker threads to a newly onlined cpu. This gets rid of the
smp_processor_idwarnings fromcache_reap. Adding a little more instrumentation to the debugsmp_processor_idshowed that new worker threads were actually running on the wrong cpu...Does this look ok?
Yeah, does that patch suffice for all the warnings?
Quoted 11 lines · Zwane Mwaikambo
Quoted 7 lines · Nathan Lynch
It looks as if we need to explicitly bind worker threads to a newly onlined cpu. This gets rid of the
smp_processor_idwarnings fromcache_reap. Adding a little more instrumentation to the debugsmp_processor_idshowed that new worker threads were actually running on the wrong cpu...Does this look ok?
Yeah, does that patch suffice for all the warnings?
Nope, ksoftirqd still requires your patch in order to kill the warnings there.
Nathan