Injecting SLB miltihit crashes kernel 5.9.0-rc5
Subsystems (2)
Hello,
Using the SLB mutihit injection test module (which I did not write so I do not want to post it here) to verify updates on my 5.3 frankernekernel I found that the kernel crashes with Oops: kernel bad access.
I tested on latest upstream kernel build that I have at hand and the result is te same (minus the message - nothing was logged and the kernel simply rebooted).
Since the whole effort to write a real mode MCE handler was supposed to prevent this maybe the SLB injection module should be added to the kernel selftests?
Thanks
Michal
Michael Ellerman
reviewerQuoted 9 lines · Michal Suchánek
Hello,
Using the SLB mutihit injection test module (which I did not write so I do not want to post it here) to verify updates on my 5.3 frankernekernel I found that the kernel crashes with Oops: kernel bad access.
I tested on latest upstream kernel build that I have at hand and the result is te same (minus the message - nothing was logged and the kernel simply rebooted).
That's disappointing.
Quoted 3 lines
Since the whole effort to write a real mode MCE handler was supposed to prevent this maybe the SLB injection module should be added to the kernel selftests?
Yes I'd like to see it upstream. I think it should be integrated into LKDTM, which contains other dangerous things like that and is designed for testing how the kernel handles/recovers from bad conditions.
cheers
Nicholas Piggin
reviewerExcerpts from Michael Ellerman's message of September 15, 2020 10:54 pm:
Quoted 7 lines
Quoted 6 lines · Michal Suchánek
Hello,
Using the SLB mutihit injection test module (which I did not write so I do not want to post it here) to verify updates on my 5.3 frankernekernel I found that the kernel crashes with Oops: kernel bad access.
Quoted 5 lines
Quoted 3 lines
I tested on latest upstream kernel build that I have at hand and the result is te same (minus the message - nothing was logged and the kernel simply rebooted).
That's disappointing.
It seems to work okay with qemu and mambo injection on upstream (powernv_defconfig). I wonder why that nmi_enter is crashing. Can you post the output of a successful test with the patch reverted?
qemu injection test output -
[ 195.279885][ C0] Disabling lock debugging due to kernel taint [ 195.280891][ C0] MCE: CPU0: machine check (Warning) Host SLB Multihit DAR: 00000000deadbeef [Recovered] [ 195.282117][ C0] MCE: CPU0: NIP: [c00000000003c2b4] isa300_idle_stop_mayloss+0x68/0x6c [ 195.283631][ C0] MCE: CPU0: Initiator CPU [ 195.284432][ C0] MCE: CPU0: Probable Software error (some chance of hardware cause) [ 220.711577][ T90] MCE: CPU0: machine check (Warning) Host SLB Multihit DAR: 00000000deadbeef [Recovered] [ 220.712805][ T90] MCE: CPU0: PID: 90 Comm: yes NIP: [00007fff7fdac2e0] [ 220.713553][ T90] MCE: CPU0: Initiator CPU [ 220.714021][ T90] MCE: CPU0: Probable Software error (some chance of hardware cause)
Thanks,
Nick
PATCH 1/1 Revert "powerpc/64s: machine check interrupt update NMI accounting"
In reply to Michal SuchánekThis reverts commit 116ac378bb3ff844df333e7609e7604651a0db9d.
This commit causes the kernel to oops and reboot when injecting a SLB multihit which causes a MCE.
Before this commit a SLB multihit was corrected by the kernel and the system continued to operate normally.
--- arch/powerpc/kernel/mce.c | 7 ------- arch/powerpc/kernel/traps.c | 18 +++--------------- 2 files changed, 3 insertions(+), 22 deletions(-)
arch/powerpc/kernel/mce.c +0 -7
@@ -591,14 +591,10 @@ EXPORT_SYMBOL_GPL(machine_check_print_event_info);long notrace machine_check_early (struct pt_regs *regs) {long handled =0 ; -bool nested =in_nmi ();u8 ftrace_enabled =this_cpu_get_ftrace_enabled ();this_cpu_set_ftrace_enabled (0 ); -if (!nested) -nmi_enter (); -hv_nmi_check_nonrecoverable (regs);/*
@@ -607,9 +603,6 @@ long notrace machine_check_early(struct pt_regs *regs)if (ppc_md.machine_check_early) handled = ppc_md.machine_check_early (regs); -if (!nested) -nmi_exit (); -this_cpu_set_ftrace_enabled (ftrace_enabled);return handled;
arch/powerpc/kernel/traps.c +3 -15
@@ -827,19 +827,7 @@ void machine_check_exception(struct pt_regs *regs) {int recover =0 ; -/* -* BOOK3S_64 does not call this handler as a non-maskable interrupt -* (it uses its own early real-mode handler to handle the MCE proper -* and then raises irq_work to call this handler when interrupts are -* enabled). -* -* This is silly. The BOOK3S_64 should just call a different function -* rather than expecting semantics to magically change. Something -* like 'non_nmi_machine_check_exception()', perhaps? -*/ -const bool nmi = !IS_ENABLED (CONFIG_PPC_BOOK3S_64 ); - -if (nmi)nmi_enter (); +nmi_enter ();__this_cpu_inc (irq_stat.mce_exceptions);
@@ -865,7 +853,7 @@ void machine_check_exception(struct pt_regs *regs)if (check_io_access (regs)) goto bail; -if (nmi)nmi_exit (); +nmi_exit ();die ("Machine check" , regs,SIGBUS );
@@ -876,7 +864,7 @@ void machine_check_exception(struct pt_regs *regs)return ; bail: -if (nmi)nmi_exit (); +nmi_exit (); }void SMIException (struct pt_regs *regs)
Signature · 1 line
2.28.0
Re: [PATCH] Revert "powerpc/64s: machine check interrupt update NMI accounting"
Quoted 11 lines · Michal Suchanek
This reverts commit
116ac378bb3ff844df333e7609e7604651a0db9d.This commit causes the kernel to oops and reboot when injecting a SLB multihit which causes a MCE.
Before this commit a SLB multihit was corrected by the kernel and the system continued to operate normally.
Fixes 116ac378bb3f ("powerpc/64s: machine check interrupt update NMI accounting")Signed-off-by Michal Suchanek <[email protected]>
Ever since 69ea03b56ed2 ("hardirq/nmi: Allow nested nmi_enter()") nmi_enter() supports nesting natively.
Re: [PATCH] Revert "powerpc/64s: machine check interrupt update NMI accounting"
Quoted 9 lines · On Tue, Sep 15, 2020 at 08:16:42PM +0200, [email protected] wrote:
Quoted 8 lines · Michal Suchanek
This reverts commit
116ac378bb3ff844df333e7609e7604651a0db9d.This commit causes the kernel to oops and reboot when injecting a SLB multihit which causes a MCE.
Before this commit a SLB multihit was corrected by the kernel and the system continued to operate normally.
Quoted 6 lines
Quoted 3 lines
Fixes 116ac378bb3f ("powerpc/64s: machine check interrupt update NMI accounting")Signed-off-by Michal Suchanek <[email protected]>Ever since
69ea03b56ed2("hardirq/nmi: Allow nestednmi_enter()")nmi_enter()supports nesting natively.
And this patch was merged in parallel with this native nesting support and conflicted with it - hence the explicit nesting in the hunk that did not conflict.
Either way the bug is present on kernels both with and without 69ea03b56ed2. So besides the conflict 69ea03b56ed2 does not affect this problem.
Thanks
Michal
Quoted 9 lines · Michal Suchánek
Hello,
Using the SLB mutihit injection test module (which I did not write so I do not want to post it here) to verify updates on my 5.3 frankernekernel I found that the kernel crashes with Oops: kernel bad access.
I tested on latest upstream kernel build that I have at hand and the result is te same (minus the message - nothing was logged and the kernel simply rebooted).
Yes, SLB multihit recovery is broken upstream. Fix is on the way.
Quoted 4 lines
Since the whole effort to write a real mode MCE handler was supposed to prevent this maybe the SLB injection module should be added to the kernel selftests?
Yes. We are working on adding SLB injection selftest patches will be posted soon.
Thanks,
-Mahesh.
Quoted 5 lines
Thanks
Michal