MCE Bug?
Subsystems (6)
A message that is not in the archive (sent off-list or not yet received): FC9702EC51E4CA40A875703BEBD6CEF801AE738D@SHSMSX101.ccr.corp.intel.com
Hi Rui,
Quoted 1 line · Rui Y
Is it a known problem? I'm based on Linux 4.1.0-rc3-7.
Yeah, I triggered a similar thing in conjunction with testing Gong's stuff:
https://lkml.kernel.org/r/[email protected]
And I was waiting in line to get a chance to do some injection on our EINJ box here too. But it seems you have the required setup already so if you want to give those changes a run, I've uploaded them here:
git://git.kernel.org/pub/scm/linux/kernel/git/ras/ras.git#tip-ras
It'll be much appreciated.
Thanks!
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--
Luck, Tony
maintainerQuoted 7 lines · Borislav Petkov
And I was waiting in line to get a chance to do some injection on our EINJ box here too. But it seems you have the required setup already so if you want to give those changes a run, I've uploaded them here:
git://git.kernel.org/pub/scm/linux/kernel/git/ras/ras.git#tip-rasIt'll be much appreciated.
and the answer is <drum roll> ....
no. :-(
Here's the console log:
[ 0.190921] Initializing cgroup subsys blkio [ 0.195701] Initializing cgroup subsys memory [ 0.200621] Initializing cgroup subsys devices [ 0.205591] Initializing cgroup subsys freezer [ 0.210563] Initializing cgroup subsys net_cls [ 0.215535] Initializing cgroup subsys perf_event [ 0.220837] Initializing cgroup subsys hugetlb [ 0.225884] CPU: Physical Processor ID: 0 [ 0.230368] CPU: Processor Core ID: 0 [ 0.235373] mce: CPU supports 22 MCE banks [ 0.239995] CPU0: Thermal monitoring enabled (TM1) [ 0.245388] process: using mwait in idle threads [ 0.250554] Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 1024 [ 0.257367] Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 1024, 1GB 4 [ 0.265023] Freeing SMP alternatives memory: 28K (ffffffff81cd9000 - ffffffff81ce0000) [ 0.284681] ftrace: allocating 25590 entries in 100 pages [ 0.301981] x2apic: IRQ remapping doesn't support X2APIC mode [ 0.308540] Switched APIC routing to physical flat. [ 0.314866] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.331616] smpboot: CPU0: Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz (fam: 06, model: 3f, stepping: 04) [ 0.342176] Performance Events: PEBS fmt2+, 16-deep LBR, Haswell events, full-width counters, Intel PMU driver. [ 0.353531] ... version: 3 [ 0.358009] ... bit width: 48 [ 0.362576] ... generic registers: 4 [ 0.367056] ... value mask: 0000ffffffffffff [ 0.372992] ... max period: 0000ffffffffffff [ 0.378926] ... fixed-purpose events: 3 [ 0.383404] ... event mask: 000000070000000f [ 0.392352] x86: Booting SMP configuration: [ 0.397027] .... node #0, CPUs: #1 [ 0.423364] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.432694] #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 [ 0.706722] .... node #1, CPUs: #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 [ 1.094522] .... node #2, CPUs: #36 [ 1.192483] mce: [Hardware Error]: Machine check events logged [ 1.209188] #37 [ 1.209279] BUG: unable to handle kernel [ 1.215830] #38 [ 1.217982] NULL pointer dereference at 0000000000000008 [ 1.223925] IP: [<ffffffff810980a1>] process_one_work+0x31/0x420 [ 1.225696] #39PGD 0 [ 1.233428] Oops: 0000 [#1] SMP [ 1.237059] Modules linked in: [ 1.240486] CPU: 36 PID: 263 Comm: kworker/36:0 Not tainted 4.1.0-rc8 #1 [ 1.247969] #40 [ 1.247969] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0065.R01.1505011640 05/01/2015 [ 1.261708] #41 [ 1.261711] task: ffff88181c284470 ti: ffff88181bd94000 task.ti: ffff88181bd94000 [ 1.272235] RIP: 0010:[<ffffffff810980a1>] [ 1.275305] #42 [<ffffffff810980a1>] process_one_work+0x31/0x420 [ 1.283854] RSP: 0000:ffff88181bd97e08 EFLAGS: 00010046 [ 1.289799] RAX: 0000000fffffffe0 RBX: ffffffff81d0fa20 RCX: 0000000000000000 [ 1.297780] #43 [ 1.297780] RDX: 0000000fffffff00 RSI: ffffffff81d0fa20 RDI: ffff88181c2660c0 [ 1.307914] RBP: ffff88181bd97e48 R08: ffff88181f416ec0 R09: ffff88181c284470 [ 1.315892] #44 [ 1.315892] R10: 0000000000000002 R11: ffffffff8109e5ac R12: ffff88181c2660c0 [ 1.326023] #45 [ 1.326024] R13: ffff88181f416ec0 R14: 0000000000000000 R15: ffff88181c2660f0 [ 1.336151] FS: 0000000000000000(0000) GS:ffff88181f400000(0000) knlGS:0000000000000000 [ 1.345207] #46 [ 1.345209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.353782] CR2: 00000000000000b8 CR3: 00000000019ca000 CR4: 00000000001406e0 [ 1.361764] #47 [ 1.361765] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1.371900] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1.379879] #48 [ 1.379879] Stack: [ 1.384277] 0000000000000282 00000000bce8578f ffff88181f416f50 ffff88181c2660c0[ 1.390883] #49
[ 1.394749] ffff88181f416ed8 ffff88181f416ec0 ffff88181c284470 ffff88181c2660f0 [ 1.403058] ffff88181bd97eb8 ffffffff81098982[ 1.407588] #50 ffff88181f4171a0 ffff88181f416ed8 [ 1.413538] Call Trace: [ 1.416274] [<ffffffff81098982>] worker_thread+0x112/0x520 [ 1.422518] [<ffffffff81098870>] ? rescuer_thread+0x3e0/0x3e0 [ 1.429048] #51 [ 1.429048] [<ffffffff8109e6d8>] kthread+0xd8/0xf0 [ 1.436650] [<ffffffff8109e600>] ? kthread_create_on_node+0x1b0/0x1b0 [ 1.443964] #52 [ 1.443966] [<ffffffff816a0322>] ret_from_fork+0x42/0x70 [ 1.452164] [<ffffffff8109e600>] ? kthread_create_on_node+0x1b0/0x1b0 [ 1.459469] #53 [ 1.459471] Code: 48 89 e5 41 57 41 56 45 31 f6 41 55 41 54 49 89 fc 53 48 89 f3 48 83 ec 18 48 8b 06 4c 8b 6f 48 48 89 [ 1.473959] [ 1.473959] .... node #3, CPUs: #54 c2 30 d2 a8 04 4c 0f 45 f2 <49> 8b 46 08 44 8b b8 00 01 00 00 41 c1 ef 05 44 89 f8 83 e0 01 [ 1.489433] RIP [<ffffffff810980a1>] process_one_work+0x31/0x420 [ 1.496261] RSP <ffff88181bd97e08> [ 1.500160] CR2: 0000000000000008 [ 1.503872] ---[ end trace 8229a011b97532a0 ]--- [ 1.509027] Kernel panic - not syncing: Fatal exception [ 1.514890] ---[ end Kernel panic - not syncing: Fatal exception
Luck, Tony
maintainerQuoted 3 lines
if you want to give those changes a run, I've uploaded them here:
git://git.kernel.org/pub/scm/linux/kernel/git/ras/ras.git#tip-ras
Latest experiments show that sometimes checking kventd_up() before calling schedule_work() helps ... but mostly only when I fake some early logs from low numbered cpus. I added some traces to the real case of a left-over fatal error and got this splat:
[ 0.331551] smpboot: CPU0: Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz (fam: 06, model: 3f, stepping: 04) [ 0.342117] Performance Events: PEBS fmt2+, 16-deep LBR, Haswell events, full-width counters, Intel PMU driver. [ 0.353471] ... version: 3 [ 0.357948] ... bit width: 48 [ 0.362523] ... generic registers: 4 [ 0.367000] ... value mask: 0000ffffffffffff [ 0.372935] ... max period: 0000ffffffffffff [ 0.378870] ... fixed-purpose events: 3 [ 0.383347] ... event mask: 000000070000000f [ 0.392357] x86: Booting SMP configuration: [ 0.397031] .... node #0, CPUs: #1 [ 0.423373] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.432705] #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 [ 0.706878] .... node #1, CPUs: #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 [ 1.094625] .... node #2, CPUs: #36 [ 1.112958] mcelog: cpu 36 bank 8 status be00000000010090 [ 1.119201] mcelog() stashed at entry=0 [ 1.203602] mce: [Hardware Error]: Machine check events logged [ 1.220313] #37 [ 1.220412] BUG: unable to handle kernel [ 1.226954] #38 [ 1.229107] NULL pointer dereference at 0000000000000008 [ 1.235052] IP: [<ffffffff810980a1>] process_one_work+0x31/0x420 [ 1.236829] #39PGD 0 [ 1.244558] Oops: 0000 [#1] SMP [ 1.248189] Modules linked in: [ 1.251617] CPU: 36 PID: 263 Comm: kworker/36:0 Not tainted 4.1.0-rc8 #9 [ 1.259100] #40 [ 1.259100] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0065.R01.1505011640 05/01/2015 [ 1.272832] #41 [ 1.272833] task: ffff88181c1f4470 ti: ffff88181bd24000 task.ti: ffff88181bd24000 [ 1.283350] RIP: 0010:[<ffffffff810980a1>] [ 1.286433] #42 [<ffffffff810980a1>] process_one_work+0x31/0x420 [ 1.294976] RSP: 0000:ffff88181bd27e08 EFLAGS: 00010046
I.e. we die on the first attempt to log ... but that attempt is a long way into bringing up all the cpus. CPU#36 is the first one from socket2 (counting 0, 1, 2, 3).
-Tony ���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m����������?�I�
Quoted 7 lines · Tony
Quoted 3 lines
if you want to give those changes a run, I've uploaded them here:
git://git.kernel.org/pub/scm/linux/kernel/git/ras/ras.git#tip-rasLatest experiments show that sometimes checking
kventd_up()before callingschedule_work()helps ... but mostly only when I fake some early logs from low numbered cpus. I added some traces to the real case of a left-over fatal error and got this splat:
Hmm, and calling mce_log from __mcheck_cpu_init_generic() as you suggested yesterday seems to work on this box here:
[ 1.588713] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-4620 0 @ 2.20GHz (fam: 06, model: 2d, stepping : 07) [ 1.592727] Performance Events: PEBS fmt1+, 16-deep LBR, SandyBridge events, full-w Broken BIOS d etected, complain to your hardware vendor. [ 1.997344] [Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR 38d is 330) [ 2.000146] Intel PMU driver. [ 2.001376] ... version: 3 [ 2.002919] ... bit width: 48 [ 2.004626] ... generic registers: 4 [ 2.006137] ... value mask: 0000ffffffffffff [ 2.008064] ... max period: 0000ffffffffffff [ 2.010010] ... fixed-purpose events: 3 [ 2.011528] ... event mask: 000000070000000f [ 2.017257] x86: Booting SMP configuration: [ 2.019232] .... node #0, CPUs: #1 [ 2.033848] microcode: CPU1 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.038730] mce: [Hardware Error]: Machine check events logged [ 2.050735] #2 [ 2.050735] microcode: CPU2 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.056163] mce: [Hardware Error]: Machine check events logged [ 2.068133] #3 [ 2.068140] microcode: CPU3 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.07412.324641] microcode: CPU4 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.479404] #5
Stuff gets logged just fine, no splats later.
Hmmm, more staring...
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--PATCH 1/1 x86/mce: Kill drain_mcelog_buffer()
In reply to Luck, TonyFrom: Borislav Petkov <[email protected]>
Subject: [PATCH] x86/mce: Kill drain_mcelog_buffer()
This used to flush out MCEs logged during early boot and which were in the MCA registers from a previous system run. No need for that now, since we're moving to a genpool.
--- arch/x86/include/asm/mce.h | 2 +- arch/x86/kernel/cpu/mcheck/mce.c | 44 ++-------------------------------------- drivers/acpi/acpi_extlog.c | 2 +- drivers/edac/i7core_edac.c | 2 +- drivers/edac/mce_amd.c | 2 +- drivers/edac/sb_edac.c | 2 +- 6 files changed, 7 insertions(+), 47 deletions(-)
arch/x86/include/asm/mce.h +1 -1
@@ -140,7 +140,7 @@ struct mce_vendor_flags {extern struct mce_vendor_flags mce_flags;extern struct mca_config mca_cfg; -extern void mce_register_decode_chain (struct notifier_block *nb,bool drain); +extern void mce_register_decode_chain (struct notifier_block *nb);extern void mce_unregister_decode_chain (struct notifier_block *nb);#include <linux/percpu.h>
arch/x86/kernel/cpu/mcheck/mce.c +2 -42
64 lines not on this page · raw message
drivers/acpi/acpi_extlog.c +1 -1
@@ -286,7 +286,7 @@ static int __init extlog_init(void)*/ old_edac_report_status =get_edac_report_status ();set_edac_report_status (EDAC_REPORTING_DISABLED ); -mce_register_decode_chain (&extlog_mce_dec, true); +mce_register_decode_chain (&extlog_mce_dec);/* enable OS to be involved to take over management from BIOS */ ((struct extlog_l1_head *)extlog_l1_addr)->flags |=FLAG_OS_OPTIN ;
drivers/edac/i7core_edac.c +1 -1
@@ -2424,7 +2424,7 @@ static int __init i7core_init(void) pci_rc =pci_register_driver (&i7core_driver);if (pci_rc >=0 ) { -mce_register_decode_chain (&i7_mce_dec, true); +mce_register_decode_chain (&i7_mce_dec);return 0 ; }
drivers/edac/mce_amd.c +1 -1
@@ -895,7 +895,7 @@ static int __init mce_amd_init(void)pr_info ("MCE: In-kernel MCE decoding enabled.\n" ); -mce_register_decode_chain (&amd_mce_dec_nb, true); +mce_register_decode_chain (&amd_mce_dec_nb);return 0 ; }
drivers/edac/sb_edac.c +1 -1
@@ -2556,7 +2556,7 @@ static int __init sbridge_init(void) pci_rc =pci_register_driver (&sbridge_driver);if (pci_rc >=0 ) { -mce_register_decode_chain (&sbridge_mce_dec, true); +mce_register_decode_chain (&sbridge_mce_dec);if (get_edac_report_status () ==EDAC_REPORTING_DISABLED )sbridge_printk (KERN_WARNING ,"Loading driver, error reporting disabled.\n" );return 0 ;
Signature · 8 lines
2.3.5
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--PATCH 1/1 x86/mce: Initialize workqueues only once
In reply to Luck, TonyQuoted 7 lines · Tony
Quoted 3 lines
if you want to give those changes a run, I've uploaded them here:
git://git.kernel.org/pub/scm/linux/kernel/git/ras/ras.git#tip-rasLatest experiments show that sometimes checking
kventd_up()before callingschedule_work()helps ... but mostly only when I fake some early logs from low numbered cpus. I added some traces to the real case of a left-over fatal error and got this splat:
Here's the fix:
--
From: Borislav Petkov <[email protected]>
Subject: [PATCH] x86/mce: Initialize workqueues only once
96d98bfd0366 ("x86/mce: Don't use percpu workqueues") dropped the per-CPU workqueues in the MCE code but left the initialization per-CPU. This lead to early boot time splats (below) in the workqueues code because we were overwriting the workqueue during INIT_WORK() on each new CPU which would appear.
And since mcheck_cpu_init() happens very early, using an initcall to do this one-time initialization doesn't fly. Using work->func as a check whether the workqueue has been initialized already might break in the future if someone changes ->func, so doesn't work either.
So let's have a simple static boolean flag to do that one-time work.
mce: [Hardware Error]: Machine check events logged
BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 IP: [<ffffffff810980a1>] process_one_work+0x31/0x420 PGD 0 Oops: 0000 [#1] SMP Modules linked in: CPU: 36 PID: 263 Comm: kworker/36:0 Not tainted 4.1.0-rc8 #1 Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0065.R01.1505011640 05/01/2015 task: ffff88181c284470 ti: ffff88181bd94000 task.ti: ffff88181bd94000 RIP: 0010:[<ffffffff810980a1>] process_one_work+0x31/0x420 RSP: 0000:ffff88181bd97e08 EFLAGS: 00010046 RAX: 0000000fffffffe0 RBX: ffffffff81d0fa20 RCX: 0000000000000000 RDX: 0000000fffffff00 RSI: ffffffff81d0fa20 RDI: ffff88181c2660c0 RBP: ffff88181bd97e48 R08: ffff88181f416ec0 R09: ffff88181c284470 R10: 0000000000000002 R11: ffffffff8109e5ac R12: ffff88181c2660c0 R13: ffff88181f416ec0 R14: 0000000000000000 R15: ffff88181c2660f0 ^^^^^^^^^^^^^^^^^
27: 4c 0f 45 f2 cmovne %rdx,%r14 2b:* 49 8b 46 08 mov 0x8(%r14),%rax <-- trapping instruction 2f: 44 8b b8 00 01 00 00 mov 0x100(%rax),%r15d
...
Call Trace: worker_thread ? rescuer_thread kthread ? kthread_create_on_node ret_from_fork ? kthread_create_on_node Code: 48 89 e5 41 57 41 56 45 31 f6 41 55 41 54 49 89 fc 53 48 89 f3 48 83 ec 18 48 8b 06 4c 8b 6f 48 48 89 c2 30 d2 a8 04 4c 0f 45 f2 <49> 8b 46 08 44 8b b8 00 01 00 00 41 c1 ef 05 44 89 f8 83 e0 01 RIP [<ffffffff810980a1>] process_one_work RSP <ffff88181bd97e08> CR2: 0000000000000008 ---[ end trace 8229a011b97532a0 ]--- Kernel panic - not syncing: Fatal exception ---[ end Kernel panic - not syncing: Fatal exception
--- arch/x86/kernel/cpu/mcheck/mce.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +7 -2
@@ -1645,6 +1645,8 @@ void (*machine_check_vector)(struct pt_regs *, long error_code) =*/ void mcheck_cpu_init (struct cpuinfo_x86 *c) { +static bool __mce_init_once; +if (mca_cfg.disabled)return ;
@@ -1665,8 +1667,11 @@ void mcheck_cpu_init(struct cpuinfo_x86 *c)return ; } -INIT_WORK (&mce_work, mce_process_work); -init_irq_work (&mce_irq_work, mce_irq_work_cb); +if (!__mce_init_once) { +INIT_WORK (&mce_work, mce_process_work); +init_irq_work (&mce_irq_work, mce_irq_work_cb); + __mce_init_once =1 ; + } machine_check_vector = do_machine_check;
Signature · 8 lines
2.3.5
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Re: [PATCH] x86/mce: Initialize workqueues only once
Quoted 1 line · Borislav Petkov
Here's the fix:
Injected error:
[ 201.237241] EINJ: Error INJection is initialized.
[ 246.146290] [Firmware Bug]: APEI: Invalid bit width + offset in GAR [0x1234/64/0/3/0]
[ 257.868432] Disabling lock debugging due to kernel taint
[ 259.808068] mce: [Hardware Error]: CPU 61: Machine Check Exception: 5 Bank 7: be00000000010090
[ 257.868432] mce: [Hardware Error]: Machine check events logged
[ 259.808068] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff81344bbf> {intel_idle+0xbf/0x130}
[ 259.808068] mce: [Hardware Error]: TSC 98aa47539f28 ADDR 1200 MISC 142303c86
[ 259.808068] mce: [Hardware Error]: PROCESSOR 0:306e7 TIME 1434716502 SOCKET 0 APIC 3 microcode 700
[ 259.808068] EDAC sbridge MC3: HANDLING MCE MEMORY ERROR
[ 259.808068] EDAC sbridge MC3: CPU 61: Machine Check Exception: 5 Bank 7: be00000000010090
[ 259.808068] EDAC sbridge MC3: TSC 98aa47539f28
[ 259.808068] EDAC sbridge MC3: ADDR 1200 EDAC sbridge MC3: MISC 142303c86
[ 259.808068] EDAC sbridge MC3: PROCESSOR 0:306e7 TIME 1434716502 SOCKET 0 APIC 3
[ 259.808068] EDAC MC3: 0 UE memory read error on CPU_SrcID#0_Channel#0_DIMM#0 (channel:0 slot:0 page:0x1 offset:0x200 grain:32 - recoverable area:DRAM err_code:0001:0090 socket:0 channel_mask:4 rank:0)Let's see whether something's in the MCA MSRs when this monster reboots (it takes forever...).
Signature · 8 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at http://www.tux.org/lkml/Luck, Tony
maintainerRe: [PATCH] x86/mce: Initialize workqueues only once (alternate proposal)
In reply to Borislav Petkov96d98bfd0366 ("x86/mce: Don't use percpu workqueues") dropped the per-CPU workqueues in the MCE code but left the initialization per-CPU. This lead to early boot time splats (below) in the workqueues code because we were overwriting the workqueue during INIT_WORK() on each new CPU which would appear.
Move initialization to mcheck_init() so it happens only once.
mce: [Hardware Error]: Machine check events logged
BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff810980a1>] process_one_work+0x31/0x420
PGD 0
Oops: 0000 [#1] SMP
Modules linked in:
CPU: 36 PID: 263 Comm: kworker/36:0 Not tainted 4.1.0-rc8 #1
Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0065.R01.1505011640
+05/01/2015
task: ffff88181c284470 ti: ffff88181bd94000 task.ti: ffff88181bd94000
RIP: 0010:[<ffffffff810980a1>] process_one_work+0x31/0x420
RSP: 0000:ffff88181bd97e08 EFLAGS: 00010046
RAX: 0000000fffffffe0 RBX: ffffffff81d0fa20 RCX: 0000000000000000
RDX: 0000000fffffff00 RSI: ffffffff81d0fa20 RDI: ffff88181c2660c0
RBP: ffff88181bd97e48 R08: ffff88181f416ec0 R09: ffff88181c284470
R10: 0000000000000002 R11: ffffffff8109e5ac R12: ffff88181c2660c0
R13: ffff88181f416ec0 R14: 0000000000000000 R15: ffff88181c2660f0
^^^^^^^^^^^^^^^^^27: 4c 0f 45 f2 cmovne %rdx,%r14 2b:* 49 8b 46 08 mov 0x8(%r14),%rax <-- trapping instruction 2f: 44 8b b8 00 01 00 00 mov 0x100(%rax),%r15d
...
Call Trace: worker_thread ? rescuer_thread kthread ? kthread_create_on_node ret_from_fork ? kthread_create_on_node Code: 48 89 e5 41 57 41 56 45 31 f6 41 55 41 54 49 89 fc 53 48 89 f3 48 83 ec 18 48 8b 06 4c
+8b 6f 48 48 89 c2 30 d2 a8 04 4c 0f 45 f2 <49> 8b 46 08 44 8b b8 00 01 00 00 41 c1 ef 05 44 +89 f8 83 e0 01
RIP [<ffffffff810980a1>] process_one_work
RSP <ffff88181bd97e08>
CR2: 0000000000000008
---[ end trace 8229a011b97532a0 ]--- Kernel panic - not syncing: Fatal exception ---[ end Kernel panic - not syncing: Fatal exception
--- arch/x86/kernel/cpu/mcheck/mce.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +3 -3
@@ -1665,9 +1665,6 @@ void mcheck_cpu_init(struct cpuinfo_x86 *c)return ; } -INIT_WORK (&mce_work, mce_process_work); -init_irq_work (&mce_irq_work, mce_irq_work_cb); - machine_check_vector = do_machine_check;__mcheck_cpu_init_generic ();
@@ -1994,6 +1991,9 @@ int __init mcheck_init(void)mce_register_decode_chain (&mce_srao_nb);mcheck_vendor_init_severity (); +INIT_WORK (&mce_work, mce_process_work); +init_irq_work (&mce_irq_work, mce_irq_work_cb); +return 0 ; }
Signature · 5 lines
2.1.4 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] x86/mce: Initialize workqueues only once (alternate proposal)
Quoted 72 lines · Tony
96d98bfd0366("x86/mce: Don't use percpu workqueues") dropped the per-CPU workqueues in the MCE code but left the initialization per-CPU. This lead to early boot time splats (below) in the workqueues code because we were overwriting the workqueue duringINIT_WORK()on each new CPU which would appear.Move initialization to
mcheck_init()so it happens only once.mce: [Hardware Error]: Machine check events logged
BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 IP: [<ffffffff810980a1>] process_one_work+0x31/0x420 PGD 0 Oops: 0000 [#1] SMP Modules linked in: CPU: 36 PID: 263 Comm: kworker/36:0 Not tainted 4.1.0-rc8 #1 Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0065.R01.1505011640 +05/01/2015 task: ffff88181c284470 ti: ffff88181bd94000 task.ti: ffff88181bd94000 RIP: 0010:[<ffffffff810980a1>] process_one_work+0x31/0x420 RSP: 0000:ffff88181bd97e08 EFLAGS: 00010046 RAX: 0000000fffffffe0 RBX: ffffffff81d0fa20 RCX: 0000000000000000 RDX: 0000000fffffff00 RSI: ffffffff81d0fa20 RDI: ffff88181c2660c0 RBP: ffff88181bd97e48 R08: ffff88181f416ec0 R09: ffff88181c284470 R10: 0000000000000002 R11: ffffffff8109e5ac R12: ffff88181c2660c0 R13: ffff88181f416ec0 R14: 0000000000000000 R15: ffff88181c2660f0 ^^^^^^^^^^^^^^^^^27: 4c 0f 45 f2 cmovne %rdx,%r14 2b:* 49 8b 46 08 mov 0x8(%r14),%rax <-- trapping instruction 2f: 44 8b b8 00 01 00 00 mov 0x100(%rax),%r15d...
Call Trace: worker_thread ? rescuer_thread kthread ? kthread_create_on_node ret_from_fork ? kthread_create_on_node Code: 48 89 e5 41 57 41 56 45 31 f6 41 55 41 54 49 89 fc 53 48 89 f3 48 83 ec 18 48 8b 06 4c+8b 6f 48 48 89 c2 30 d2 a8 04 4c 0f 45 f2 <49> 8b 46 08 44 8b b8 00 01 00 00 41 c1 ef 05 44 +89 f8 83 e0 01
RIP [<ffffffff810980a1>]process_one_work
RSP <ffff88181bd97e08>
CR2: 0000000000000008---[ end trace 8229a011b97532a0 ]--- Kernel panic - not syncing: Fatal exception ---[ end Kernel panic - not syncing: Fatal exceptionReported-by Rui Wang <[email protected]>Debugged-by Borislav Petkov <[email protected]>Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/mcheck/mce.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)arch/x86/kernel/cpu/mcheck/mce.c +0
-3@@ -1665,9 +1665,6 @@ void mcheck_cpu_init(struct cpuinfo_x86 *c)return ; } -INIT_WORK (&mce_work, mce_process_work); -init_irq_work (&mce_irq_work, mce_irq_work_cb); - machine_check_vector = do_machine_check;__mcheck_cpu_init_generic ();
arch/x86/kernel/cpu/mcheck/mce.c int __init mcheck_init(void) · line 1997
Quoted 7 lines
arch/x86/kernel/cpu/mcheck/mce.c +3
-0@@ -1994,6 +1991,9 @@ int __init mcheck_init(void)mce_register_decode_chain (&mce_srao_nb);mcheck_vendor_init_severity (); +INIT_WORK (&mce_work, mce_process_work); +init_irq_work (&mce_irq_work, mce_irq_work_cb); +return 0 ;
Hmm, and I was under the impression that mcheck_init() runs much later... Not really.
Anyway, your version is better, I've replaced mine with it.
Thanks.
Signature · 8 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at http://www.tux.org/lkml/