cpu offline causes backtrace from cmci_rediscover
offlining a CPU in 3.9-rc3 gets me this trace..
numa_remove_cpu cpu 1 node 0: mask now 0,2-3
smpboot: CPU 1 is now offline
BUG: using smp_processor_id() in preemptible [00000000] code: cpu-offline.sh/10591 caller is cmci_rediscover+0x6a/0xe0 Pid: 10591, comm: cpu-offline.sh Not tainted 3.9.0-rc3+ #2 Call Trace: [<ffffffff81333bbd>] debug_smp_processor_id+0xdd/0x100 [<ffffffff8101edba>] cmci_rediscover+0x6a/0xe0 [<ffffffff815f5b9f>] mce_cpu_callback+0x19d/0x1ae [<ffffffff8160ea66>] notifier_call_chain+0x66/0x150 [<ffffffff8107ad7e>] __raw_notifier_call_chain+0xe/0x10 [<ffffffff8104c2e3>] cpu_notify+0x23/0x50 [<ffffffff8104c31e>] cpu_notify_nofail+0xe/0x20 [<ffffffff815ef082>] _cpu_down+0x302/0x350 [<ffffffff815ef106>] cpu_down+0x36/0x50 [<ffffffff815f1c9d>] store_online+0x8d/0xd0 [<ffffffff813edc48>] dev_attr_store+0x18/0x30 [<ffffffff81226eeb>] sysfs_write_file+0xdb/0x150 [<ffffffff811adfb2>] vfs_write+0xa2/0x170 [<ffffffff811ae16c>] sys_write+0x4c/0xa0 [<ffffffff81613019>] system_call_fastpath+0x16/0x1b
Quoted 22 lines · Dave Jones
Date: Tue, 19 Mar 2013 18:44:08 -0400
From: Dave Jones <[email protected]>
To: Linux Kernel <[email protected]>
Cc: [email protected]
Subject: cpu offline causes backtrace fromcmci_rediscover
User-Agent: Mutt/1.5.21 (2010-09-15)offlining a CPU in 3.9-rc3 gets me this trace..
numa_remove_cpucpu 1 node 0: mask now 0,2-3
smpboot: CPU 1 is now offlineBUG: using smp_processor_id() in preemptible [00000000] code: cpu-offline.sh/10591 caller is cmci_rediscover+0x6a/0xe0 Pid: 10591, comm: cpu-offline.sh Not tainted 3.9.0-rc3+ #2 Call Trace: [<ffffffff81333bbd>] debug_smp_processor_id+0xdd/0x100 [<ffffffff8101edba>] cmci_rediscover+0x6a/0xe0 [<ffffffff815f5b9f>] mce_cpu_callback+0x19d/0x1ae [<ffffffff8160ea66>] notifier_call_chain+0x66/0x150 [<ffffffff8107ad7e>] __raw_notifier_call_chain+0xe/0x10 [<ffffffff8104c2e3>] cpu_notify+0x23/0x50 [<ffffffff8104c31e>] cpu_notify_nofail+0xe/0x20
Quoted 9 lines
[<ffffffff815ef082>] _cpu_down+0x302/0x350 [<ffffffff815ef106>] cpu_down+0x36/0x50 [<ffffffff815f1c9d>] store_online+0x8d/0xd0 [<ffffffff813edc48>] dev_attr_store+0x18/0x30 [<ffffffff81226eeb>] sysfs_write_file+0xdb/0x150 [<ffffffff811adfb2>] vfs_write+0xa2/0x170 [<ffffffff811ae16c>] sys_write+0x4c/0xa0 [<ffffffff81613019>] system_call_fastpath+0x16/0x1b
Try this patch:
diff a/arch/x86/kernel/cpu/mcheck/mce_intel.c b/arch/x86/kernel/cpu/mcheck/mce_intel.c index 402c454..692c91e 100644
arch/x86/kernel/cpu/mcheck/mce_intel.c +4 -2
@@ -311,10 +311,12 @@ void cmci_rediscover(int dying)if (cpu == dying)continue ; -if (cpu ==smp_processor_id ()) { +if (cpu ==get_cpu ()) { +put_cpu ();cmci_rediscover_work_func (NULL );continue ; - } + }else +put_cpu ();work_on_cpu (cpu, cmci_rediscover_work_func,NULL ); }
Quoted 44 lines · Chen Gong
Quoted 22 lines · Dave Jones
Date: Tue, 19 Mar 2013 18:44:08 -0400
From: Dave Jones <[email protected]>
To: Linux Kernel <[email protected]>
Cc: [email protected]
Subject: cpu offline causes backtrace fromcmci_rediscover
User-Agent: Mutt/1.5.21 (2010-09-15)offlining a CPU in 3.9-rc3 gets me this trace..
numa_remove_cpucpu 1 node 0: mask now 0,2-3
smpboot: CPU 1 is now offlineBUG: using smp_processor_id() in preemptible [00000000] code: cpu-offline.sh/10591 caller is cmci_rediscover+0x6a/0xe0 Pid: 10591, comm: cpu-offline.sh Not tainted 3.9.0-rc3+ #2 Call Trace: [<ffffffff81333bbd>] debug_smp_processor_id+0xdd/0x100 [<ffffffff8101edba>] cmci_rediscover+0x6a/0xe0 [<ffffffff815f5b9f>] mce_cpu_callback+0x19d/0x1ae [<ffffffff8160ea66>] notifier_call_chain+0x66/0x150 [<ffffffff8107ad7e>] __raw_notifier_call_chain+0xe/0x10 [<ffffffff8104c2e3>] cpu_notify+0x23/0x50 [<ffffffff8104c31e>] cpu_notify_nofail+0xe/0x20Quoted 9 lines
[<ffffffff815ef082>] _cpu_down+0x302/0x350 [<ffffffff815ef106>] cpu_down+0x36/0x50 [<ffffffff815f1c9d>] store_online+0x8d/0xd0 [<ffffffff813edc48>] dev_attr_store+0x18/0x30 [<ffffffff81226eeb>] sysfs_write_file+0xdb/0x150 [<ffffffff811adfb2>] vfs_write+0xa2/0x170 [<ffffffff811ae16c>] sys_write+0x4c/0xa0 [<ffffffff81613019>] system_call_fastpath+0x16/0x1bTry this patch:
diff
a/arch/x86/kernel/cpu/mcheck/mce_intel.cb/arch/x86/kernel/cpu/mcheck/mce_intel.cindex 402c454..692c91e 100644arch/x86/kernel/cpu/mcheck/mce_intel.c +0
-1@@ -311,10 +311,12 @@ void cmci_rediscover(int dying)if (cpu == dying)continue ; -if (cpu ==smp_processor_id ()) {
arch/x86/kernel/cpu/mcheck/mce_intel.c void cmci_rediscover(int dying) · line 322
Quoted 10 lines
arch/x86/kernel/cpu/mcheck/mce_intel.c +4
-1+if (cpu ==get_cpu ()) { +put_cpu ();cmci_rediscover_work_func (NULL );continue ; - } + }else +put_cpu ();work_on_cpu (cpu, cmci_rediscover_work_func,NULL ); }
raw_smp_processor_id()?
-Mike
PATCH 1/1 x86/mce: Rework cmci_rediscover() to play well with CPU hotplug
In reply to Chen GongQuoted 38 lines · Chen Gong
Quoted 16 lines · Dave Jones
offlining a CPU in 3.9-rc3 gets me this trace..
numa_remove_cpucpu 1 node 0: mask now 0,2-3
smpboot: CPU 1 is now offlineBUG: using smp_processor_id() in preemptible [00000000] code: cpu-offline.sh/10591 caller is cmci_rediscover+0x6a/0xe0 Pid: 10591, comm: cpu-offline.sh Not tainted 3.9.0-rc3+ #2 Call Trace: [<ffffffff81333bbd>] debug_smp_processor_id+0xdd/0x100 [<ffffffff8101edba>] cmci_rediscover+0x6a/0xe0 [<ffffffff815f5b9f>] mce_cpu_callback+0x19d/0x1ae [<ffffffff8160ea66>] notifier_call_chain+0x66/0x150 [<ffffffff8107ad7e>] __raw_notifier_call_chain+0xe/0x10 [<ffffffff8104c2e3>] cpu_notify+0x23/0x50 [<ffffffff8104c31e>] cpu_notify_nofail+0xe/0x20Quoted 9 lines
[<ffffffff815ef082>] _cpu_down+0x302/0x350 [<ffffffff815ef106>] cpu_down+0x36/0x50 [<ffffffff815f1c9d>] store_online+0x8d/0xd0 [<ffffffff813edc48>] dev_attr_store+0x18/0x30 [<ffffffff81226eeb>] sysfs_write_file+0xdb/0x150 [<ffffffff811adfb2>] vfs_write+0xa2/0x170 [<ffffffff811ae16c>] sys_write+0x4c/0xa0 [<ffffffff81613019>] system_call_fastpath+0x16/0x1bTry this patch:
diff
a/arch/x86/kernel/cpu/mcheck/mce_intel.cb/arch/x86/kernel/cpu/mcheck/mce_intel.cindex 402c454..692c91e 100644arch/x86/kernel/cpu/mcheck/mce_intel.c +0
-1@@ -311,10 +311,12 @@ void cmci_rediscover(int dying)if (cpu == dying)continue ; -if (cpu ==smp_processor_id ()) {
arch/x86/kernel/cpu/mcheck/mce_intel.c void cmci_rediscover(int dying) · line 322
Quoted 10 lines
arch/x86/kernel/cpu/mcheck/mce_intel.c +4
-1+if (cpu ==get_cpu ()) { +put_cpu ();cmci_rediscover_work_func (NULL );continue ; - } + }else +put_cpu ();work_on_cpu (cpu, cmci_rediscover_work_func,NULL ); }
That doesn't really look right to me. In fact, the function cmci_rediscover() looks like it needs some attention. Let me quote the function here, so that its easier to discuss what's wrong with it..
/*
* After a CPU went down cycle through all the others and rediscover
* Must run in process context.
*/
void cmci_rediscover(int dying)
{
int cpu, banks;
if (!cmci_supported(&banks))
return;
for_each_online_cpu(cpu) {
if (cpu == dying)
continue;
if (cpu == smp_processor_id()) {
cmci_rediscover_work_func(NULL);
continue;
}
work_on_cpu(cpu, cmci_rediscover_work_func, NULL);
}
}First of all, I think the comment that says that it must run in process context, is stale. I think its a remnant of the code which used to do GFP_KERNEL allocations for a temporary cpumask (looking at git logs). The function cmci_discover() takes a spin lock with irqs disabled. So obviously this whole thing can run in atomic context.
And cmci_rediscover() is called from CPU_POST_DEAD handler. So the CPU which was supposed to go offline would have already gone offline and out of the cpu_online_mask. So there is no point checking 'if (cpu == dying)' in that for-loop.
So, how about something like this:
------------------------------------------------------------>
From: Srivatsa S. Bhat <[email protected]>
Subject: [PATCH] x86/mce: Rework cmci_rediscover() to play well with CPU hotplug
Dave Jones reports that offlining a CPU leads to this trace:
numa_remove_cpu cpu 1 node 0: mask now 0,2-3
smpboot: CPU 1 is now offline
BUG: using smp_processor_id() in preemptible [00000000] code: cpu-offline.sh/10591 caller is cmci_rediscover+0x6a/0xe0 Pid: 10591, comm: cpu-offline.sh Not tainted 3.9.0-rc3+ #2 Call Trace: [<ffffffff81333bbd>] debug_smp_processor_id+0xdd/0x100 [<ffffffff8101edba>] cmci_rediscover+0x6a/0xe0 [<ffffffff815f5b9f>] mce_cpu_callback+0x19d/0x1ae [<ffffffff8160ea66>] notifier_call_chain+0x66/0x150 [<ffffffff8107ad7e>] __raw_notifier_call_chain+0xe/0x10 [<ffffffff8104c2e3>] cpu_notify+0x23/0x50 [<ffffffff8104c31e>] cpu_notify_nofail+0xe/0x20 [<ffffffff815ef082>] _cpu_down+0x302/0x350 [<ffffffff815ef106>] cpu_down+0x36/0x50 [<ffffffff815f1c9d>] store_online+0x8d/0xd0 [<ffffffff813edc48>] dev_attr_store+0x18/0x30 [<ffffffff81226eeb>] sysfs_write_file+0xdb/0x150 [<ffffffff811adfb2>] vfs_write+0xa2/0x170 [<ffffffff811ae16c>] sys_write+0x4c/0xa0 [<ffffffff81613019>] system_call_fastpath+0x16/0x1b
However, a look at cmci_rediscover shows that it can be simplified quite a bit, apart from solving the above issue. It invokes functions that take spin locks with interrupts disabled, and hence it can run in atomic context. Also, it is run in the CPU_POST_DEAD phase, so the dying CPU is already dead and out of the cpu_online_mask. So take these points into account and simplify the code, and thereby also fix the above issue.
---
arch/x86/include/asm/mce.h | 4 ++-- arch/x86/kernel/cpu/mcheck/mce.c | 2 +- arch/x86/kernel/cpu/mcheck/mce_intel.c | 25 +++++-------------------- 3 files changed, 8 insertions(+), 23 deletions(-)
arch/x86/include/asm/mce.h +2 -2
@@ -146,13 +146,13 @@ DECLARE_PER_CPU(struct device *, mce_device);void mce_intel_feature_init (struct cpuinfo_x86 *c);void cmci_clear (void );void cmci_reenable (void ); -void cmci_rediscover (int dying); +void cmci_rediscover (void );void cmci_recheck (void );#else static inline void mce_intel_feature_init (struct cpuinfo_x86 *c) { }static inline void cmci_clear (void ) {}static inline void cmci_reenable (void ) {} -static inline void cmci_rediscover (int dying) {} +static inline void cmci_rediscover (void ) {}static inline void cmci_recheck (void ) {}#endif
arch/x86/kernel/cpu/mcheck/mce.c +1 -1
@@ -2358,7 +2358,7 @@ mce_cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)if (action ==CPU_POST_DEAD ) {/* intentionally ignoring frozen here */ -cmci_rediscover (cpu); +cmci_rediscover (); }return NOTIFY_OK ;
arch/x86/kernel/cpu/mcheck/mce_intel.c +5 -20
@@ -285,39 +285,24 @@ void cmci_clear(void)raw_spin_unlock_irqrestore (&cmci_discover_lock, flags); } -static long cmci_rediscover_work_func (void *arg) +static void cmci_rediscover_work_func (void *arg) {int banks;/* Recheck banks in case CPUs don't all have the same */ if (cmci_supported (&banks))cmci_discover (banks); - -return 0 ; } -/* -* After a CPU went down cycle through all the others and rediscover -* Must run in process context. -*/ -void cmci_rediscover (int dying) +/* After a CPU went down cycle through all the others and rediscover */ +void cmci_rediscover (void ) { -int cpu, banks; +int banks;if (!cmci_supported (&banks))return ; -for_each_online_cpu (cpu) { -if (cpu == dying) -continue ; - -if (cpu ==smp_processor_id ()) { -cmci_rediscover_work_func (NULL ); -continue ; - } - -work_on_cpu (cpu, cmci_rediscover_work_func,NULL ); - } +on_each_cpu (cmci_rediscover_work_func,NULL ,1 ); }/*
Borislav Petkov
maintainerRe: [PATCH] x86/mce: Rework cmci_rediscover() to play well with CPU hotplug
+ Tony.
Quoted 228 lines · Srivatsa S. Bhat
Quoted 38 lines · Chen Gong
Quoted 16 lines · Dave Jones
offlining a CPU in 3.9-rc3 gets me this trace..
numa_remove_cpucpu 1 node 0: mask now 0,2-3
smpboot: CPU 1 is now offlineBUG: using smp_processor_id() in preemptible [00000000] code: cpu-offline.sh/10591 caller is cmci_rediscover+0x6a/0xe0 Pid: 10591, comm: cpu-offline.sh Not tainted 3.9.0-rc3+ #2 Call Trace: [<ffffffff81333bbd>] debug_smp_processor_id+0xdd/0x100 [<ffffffff8101edba>] cmci_rediscover+0x6a/0xe0 [<ffffffff815f5b9f>] mce_cpu_callback+0x19d/0x1ae [<ffffffff8160ea66>] notifier_call_chain+0x66/0x150 [<ffffffff8107ad7e>] __raw_notifier_call_chain+0xe/0x10 [<ffffffff8104c2e3>] cpu_notify+0x23/0x50 [<ffffffff8104c31e>] cpu_notify_nofail+0xe/0x20Quoted 9 lines
[<ffffffff815ef082>] _cpu_down+0x302/0x350 [<ffffffff815ef106>] cpu_down+0x36/0x50 [<ffffffff815f1c9d>] store_online+0x8d/0xd0 [<ffffffff813edc48>] dev_attr_store+0x18/0x30 [<ffffffff81226eeb>] sysfs_write_file+0xdb/0x150 [<ffffffff811adfb2>] vfs_write+0xa2/0x170 [<ffffffff811ae16c>] sys_write+0x4c/0xa0 [<ffffffff81613019>] system_call_fastpath+0x16/0x1bTry this patch:
diff
a/arch/x86/kernel/cpu/mcheck/mce_intel.cb/arch/x86/kernel/cpu/mcheck/mce_intel.cindex 402c454..692c91e 100644arch/x86/kernel/cpu/mcheck/mce_intel.c +0
-1@@ -311,10 +311,12 @@ void cmci_rediscover(int dying)if (cpu == dying)continue ; -if (cpu ==smp_processor_id ()) {arch/x86/kernel/cpu/mcheck/mce_intel.c void cmci_rediscover(int dying) · line 322
Quoted 10 lines
arch/x86/kernel/cpu/mcheck/mce_intel.c +4
-1+if (cpu ==get_cpu ()) { +put_cpu ();cmci_rediscover_work_func (NULL );continue ; - } + }else +put_cpu ();work_on_cpu (cpu, cmci_rediscover_work_func,NULL ); }That doesn't really look right to me. In fact, the function
cmci_rediscover()looks like it needs some attention. Let me quote the function here, so that its easier to discuss what's wrong with it../*
* After a CPU went down cycle through all the others and rediscover
* Must run in process context.
*/
voidcmci_rediscover(int dying){ int cpu, banks; if (!cmci_supported(&banks)) return; for_each_online_cpu(cpu) { if (cpu == dying) continue; if (cpu == smp_processor_id()) { cmci_rediscover_work_func(NULL); continue; } work_on_cpu(cpu, cmci_rediscover_work_func, NULL); } }First of all, I think the comment that says that it must run in process context, is stale. I think its a remnant of the code which used to do
GFP_KERNELallocations for a temporary cpumask (looking at git logs). The functioncmci_discover()takes a spin lock with irqs disabled. So obviously this whole thing can run in atomic context.And
cmci_rediscover()is called fromCPU_POST_DEADhandler. So the CPU which was supposed to go offline would have already gone offline and out of thecpu_online_mask. So there is no point checking 'if (cpu == dying)' in that for-loop.So, how about something like this:
------------------------------------------------------------>
From: Srivatsa S. Bhat <[email protected]>
Subject: [PATCH] x86/mce: Reworkcmci_rediscover()to play well with CPU hotplugDave Jones reports that offlining a CPU leads to this trace:
numa_remove_cpucpu 1 node 0: mask now 0,2-3
smpboot: CPU 1 is now offlineBUG: using smp_processor_id() in preemptible [00000000] code: cpu-offline.sh/10591 caller is cmci_rediscover+0x6a/0xe0 Pid: 10591, comm: cpu-offline.sh Not tainted 3.9.0-rc3+ #2 Call Trace: [<ffffffff81333bbd>] debug_smp_processor_id+0xdd/0x100 [<ffffffff8101edba>] cmci_rediscover+0x6a/0xe0 [<ffffffff815f5b9f>] mce_cpu_callback+0x19d/0x1ae [<ffffffff8160ea66>] notifier_call_chain+0x66/0x150 [<ffffffff8107ad7e>] __raw_notifier_call_chain+0xe/0x10 [<ffffffff8104c2e3>] cpu_notify+0x23/0x50 [<ffffffff8104c31e>] cpu_notify_nofail+0xe/0x20 [<ffffffff815ef082>] _cpu_down+0x302/0x350 [<ffffffff815ef106>] cpu_down+0x36/0x50 [<ffffffff815f1c9d>] store_online+0x8d/0xd0 [<ffffffff813edc48>] dev_attr_store+0x18/0x30 [<ffffffff81226eeb>] sysfs_write_file+0xdb/0x150 [<ffffffff811adfb2>] vfs_write+0xa2/0x170 [<ffffffff811ae16c>] sys_write+0x4c/0xa0 [<ffffffff81613019>] system_call_fastpath+0x16/0x1bHowever, a look at
cmci_rediscovershows that it can be simplified quite a bit, apart from solving the above issue. It invokes functions that take spin locks with interrupts disabled, and hence it can run in atomic context. Also, it is run in theCPU_POST_DEADphase, so the dying CPU is already dead and out of thecpu_online_mask. So take these points into account and simplify the code, and thereby also fix the above issue.Reported-by Dave Jones <[email protected]>Signed-off-by Srivatsa S. Bhat <[email protected]>---
arch/x86/include/asm/mce.h | 4 ++-- arch/x86/kernel/cpu/mcheck/mce.c | 2 +- arch/x86/kernel/cpu/mcheck/mce_intel.c | 25 +++++-------------------- 3 files changed, 8 insertions(+), 23 deletions(-)arch/x86/include/asm/mce.h +2
-2@@ -146,13 +146,13 @@ DECLARE_PER_CPU(struct device *, mce_device);void mce_intel_feature_init (struct cpuinfo_x86 *c);void cmci_clear (void );void cmci_reenable (void ); -void cmci_rediscover (int dying); +void cmci_rediscover (void );void cmci_recheck (void );#else static inline void mce_intel_feature_init (struct cpuinfo_x86 *c) { }static inline void cmci_clear (void ) {}static inline void cmci_reenable (void ) {} -static inline void cmci_rediscover (int dying) {} +static inline void cmci_rediscover (void ) {}static inline void cmci_recheck (void ) {}#endif arch/x86/kernel/cpu/mcheck/mce.c +1
-1@@ -2358,7 +2358,7 @@ mce_cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)if (action ==CPU_POST_DEAD ) {/* intentionally ignoring frozen here */ -cmci_rediscover (cpu); +cmci_rediscover (); }return NOTIFY_OK ;arch/x86/kernel/cpu/mcheck/mce_intel.c +5
-21@@ -285,39 +285,24 @@ void cmci_clear(void)raw_spin_unlock_irqrestore (&cmci_discover_lock, flags); } -static long cmci_rediscover_work_func (void *arg) +static void cmci_rediscover_work_func (void *arg) {int banks;/* Recheck banks in case CPUs don't all have the same */ if (cmci_supported (&banks))cmci_discover (banks); - -return 0 ; } -/* -* After a CPU went down cycle through all the others and rediscover -* Must run in process context. -*/ -void cmci_rediscover (int dying) +/* After a CPU went down cycle through all the others and rediscover */ +void cmci_rediscover (void ) { -int cpu, banks; +int banks;if (!cmci_supported (&banks))return ; -for_each_online_cpu (cpu) { -if (cpu == dying) -continue ; - -if (cpu ==smp_processor_id ()) { -cmci_rediscover_work_func (NULL ); -continue ; - } - -work_on_cpu (cpu, cmci_rediscover_work_func,NULL ); - } +on_each_cpu (cmci_rediscover_work_func,NULL ,1 ); }/* -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Re: [PATCH] x86/mce: Rework cmci_rediscover() to play well with CPU hotplug
In reply to Srivatsa S. BhatLooks lots cleaner. Applied on top of 3.9.rc3 and took it for a spin offlining and onlining cpus at random intervals. First time round I saw a few splats like the one below. But after a reboot I can no longer reproduce.
-Tony
INFO: task devkit-power-da:19861 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
devkit-power-da D ffff8810114cd048 0 19861 1 0x00000080
ffff88101063b828 0000000000000082 ffff8810114cca80 ffff88101063bfd8 ffff88101063bfd8 ffff88101063bfd8 ffff88081fa18040 ffff8810114cca80 ffff88101063b808 ffff88080c0cf800 0000000000000000 ffff88101063b8b0
Call Trace: [<ffffffff815dcd39>] schedule+0x29/0x70 [<ffffffff81478865>] usb_kill_urb+0x85/0xc0 [<ffffffff810771f0>] ? wake_up_bit+0x40/0x40 [<ffffffff81479b08>] usb_start_wait_urb+0xd8/0x160 [<ffffffff81479dec>] usb_control_msg+0xcc/0x110 [<ffffffff81479ef3>] ? usb_get_status+0x43/0xc0 [<ffffffff81479f33>] usb_get_status+0x83/0xc0 [<ffffffff8146e707>] ? usb_set_device_state+0x127/0x160 [<ffffffff81471fa6>] usb_port_resume+0x2c6/0x630 [<ffffffff810135d1>] ? __switch_to+0x181/0x4a0 [<ffffffff81485935>] generic_resume+0x15/0x30 [<ffffffff8147c405>] usb_resume_both+0x105/0x150 [<ffffffff8147d8aa>] usb_runtime_resume+0x1a/0x20 [<ffffffff813b5bd1>] __rpm_callback+0x31/0x90 [<ffffffff813b5c5f>] rpm_callback+0x2f/0x90 [<ffffffff813b6d5c>] rpm_resume+0x40c/0x670 [<ffffffff810771f0>] ? wake_up_bit+0x40/0x40 [<ffffffff813b725c>] __pm_runtime_resume+0x5c/0x90 [<ffffffff8147cee9>] usb_autoresume_device+0x29/0x60 [<ffffffff81482b50>] usbdev_open+0x110/0x210 [<ffffffff8116dc4c>] chrdev_open+0x9c/0x180 [<ffffffff811677df>] do_dentry_open+0x20f/0x2c0 [<ffffffff8116dbb0>] ? cdev_put+0x30/0x30 [<ffffffff811678c5>] finish_open+0x35/0x50 [<ffffffff81177e5e>] do_last+0x6de/0xde0 [<ffffffff811749a8>] ? inode_permission+0x18/0x50 [<ffffffff81174a58>] ? link_path_walk+0x78/0x880 [<ffffffff81178617>] path_openat+0xb7/0x4a0 [<ffffffff81178cc1>] do_filp_open+0x41/0xa0 [<ffffffff81185382>] ? __alloc_fd+0x42/0x110 [<ffffffff81168c24>] do_sys_open+0xf4/0x1e0 [<ffffffff81014959>] ? do_notify_resume+0x59/0x80 [<ffffffff81168d31>] sys_open+0x21/0x30 [<ffffffff815e6159>] system_call_fastpath+0x16/0x1b
Re: [PATCH] x86/mce: Rework cmci_rediscover() to play well with CPU hotplug
In reply to Srivatsa S. BhatOn 03/20/2013 03:31 PM, Srivatsa S. Bhat wrote:
[...]
Quoted 120 lines
------------------------------------------------------------>
From: Srivatsa S. Bhat <[email protected]>
Subject: [PATCH] x86/mce: Reworkcmci_rediscover()to play well with CPU hotplugDave Jones reports that offlining a CPU leads to this trace:
numa_remove_cpucpu 1 node 0: mask now 0,2-3
smpboot: CPU 1 is now offlineBUG: using smp_processor_id() in preemptible [00000000] code: cpu-offline.sh/10591 caller is cmci_rediscover+0x6a/0xe0 Pid: 10591, comm: cpu-offline.sh Not tainted 3.9.0-rc3+ #2 Call Trace: [<ffffffff81333bbd>] debug_smp_processor_id+0xdd/0x100 [<ffffffff8101edba>] cmci_rediscover+0x6a/0xe0 [<ffffffff815f5b9f>] mce_cpu_callback+0x19d/0x1ae [<ffffffff8160ea66>] notifier_call_chain+0x66/0x150 [<ffffffff8107ad7e>] __raw_notifier_call_chain+0xe/0x10 [<ffffffff8104c2e3>] cpu_notify+0x23/0x50 [<ffffffff8104c31e>] cpu_notify_nofail+0xe/0x20 [<ffffffff815ef082>] _cpu_down+0x302/0x350 [<ffffffff815ef106>] cpu_down+0x36/0x50 [<ffffffff815f1c9d>] store_online+0x8d/0xd0 [<ffffffff813edc48>] dev_attr_store+0x18/0x30 [<ffffffff81226eeb>] sysfs_write_file+0xdb/0x150 [<ffffffff811adfb2>] vfs_write+0xa2/0x170 [<ffffffff811ae16c>] sys_write+0x4c/0xa0 [<ffffffff81613019>] system_call_fastpath+0x16/0x1bHowever, a look at
cmci_rediscovershows that it can be simplified quite a bit, apart from solving the above issue. It invokes functions that take spin locks with interrupts disabled, and hence it can run in atomic context. Also, it is run in theCPU_POST_DEADphase, so the dying CPU is already dead and out of thecpu_online_mask. So take these points into account and simplify the code, and thereby also fix the above issue.Reported-by Dave Jones <[email protected]>Signed-off-by Srivatsa S. Bhat <[email protected]>---
arch/x86/include/asm/mce.h | 4 ++-- arch/x86/kernel/cpu/mcheck/mce.c | 2 +- arch/x86/kernel/cpu/mcheck/mce_intel.c | 25 +++++-------------------- 3 files changed, 8 insertions(+), 23 deletions(-)arch/x86/include/asm/mce.h +2
-2@@ -146,13 +146,13 @@ DECLARE_PER_CPU(struct device *, mce_device);void mce_intel_feature_init (struct cpuinfo_x86 *c);void cmci_clear (void );void cmci_reenable (void ); -void cmci_rediscover (int dying); +void cmci_rediscover (void );void cmci_recheck (void );#else static inline void mce_intel_feature_init (struct cpuinfo_x86 *c) { }static inline void cmci_clear (void ) {}static inline void cmci_reenable (void ) {} -static inline void cmci_rediscover (int dying) {} +static inline void cmci_rediscover (void ) {}static inline void cmci_recheck (void ) {}#endif arch/x86/kernel/cpu/mcheck/mce.c +1
-1@@ -2358,7 +2358,7 @@ mce_cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)if (action ==CPU_POST_DEAD ) {/* intentionally ignoring frozen here */ -cmci_rediscover (cpu); +cmci_rediscover (); }return NOTIFY_OK ;arch/x86/kernel/cpu/mcheck/mce_intel.c +4
-14@@ -285,39 +285,24 @@ void cmci_clear(void)raw_spin_unlock_irqrestore (&cmci_discover_lock, flags); } -static long cmci_rediscover_work_func (void *arg) +static void cmci_rediscover_work_func (void *arg) {int banks;/* Recheck banks in case CPUs don't all have the same */ if (cmci_supported (&banks))cmci_discover (banks); - -return 0 ; } -/* -* After a CPU went down cycle through all the others and rediscover -* Must run in process context. -*/ -void cmci_rediscover (int dying) +/* After a CPU went down cycle through all the others and rediscover */ +void cmci_rediscover (void ) { -int cpu, banks; +int banks;if (!cmci_supported (&banks))return ; -for_each_online_cpu (cpu) { -if (cpu == dying) -continue ; - -if (cpu ==smp_processor_id ()) {
arch/x86/kernel/cpu/mcheck/mce_intel.c void cmci_clear(void) · line 308
Quoted 10 lines
arch/x86/kernel/cpu/mcheck/mce_intel.c +1
-6-cmci_rediscover_work_func (NULL ); -continue ; - } - -work_on_cpu (cpu, cmci_rediscover_work_func,NULL ); - } +on_each_cpu (cmci_rediscover_work_func,NULL ,1 ); }/*
Hi Boris,
Tony mentioned that this patch worked fine for him. So could you kindly pick up this patch?
Regards,
Srivatsa S. Bhat
Borislav Petkov
maintainerRe: [PATCH] x86/mce: Rework cmci_rediscover() to play well with CPU hotplug
Quoted 2 lines · Srivatsa S. Bhat
Tony mentioned that this patch worked fine for him. So could you kindly pick up this patch?
Normally, Tony picks up the Intel side of MCE. Tony, want me to do it?
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--RE: [PATCH] x86/mce: Rework cmci_rediscover() to play well with CPU hotplug
Quoted 4 lines
Quoted 2 lines
Tony mentioned that this patch worked fine for him. So could you kindly pick up this patch?
Normally, Tony picks up the Intel side of MCE. Tony, want me to do it?
I'll pick it up. Thanks.
-Tony ���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m����������?�I�