Fixes for 3 separate NMI reentrancy bugs
Nicholas Piggin
reviewerv4 cover Fixes for 3 separate NMI reentrancy bugs
This series fixes several similar but unrelated bugs with NMIs clobbering live registers without noticing it, because MSR[RI] is set. Pretty rare bugs, but serious silent corruption consequences.
For the most part these can be observed and tested quite easily with the mambo simulator, except that it does not seem to follow the architecture wrt leaving MSR[RI] unchanged for HV interrupts. Mambo clears MSR[RI], so you have to account for that manually.
Since v1:
- Fixed several build bugs.
Since v2:
- Improved changelog and comments.
- Fixed the NIA test for virt mode interrupts.
Since v3:
- Fixed HPT crash due to use of PACA_EXGEN rather than EXSLB in the SLB interrupt handlers.
Nicholas Piggin (4):
powerpc/64s: Fix HV NMI vs HV interrupt recoverability test powerpc/64s: system reset interrupt preserve HSRRs powerpc/64s: Prepare to handle data interrupts vs d-side MCE
reentrancy
powerpc/64s: Fix data interrupts vs d-side MCE reentrancy
arch/powerpc/include/asm/asm-prototypes.h | 8 ++ arch/powerpc/include/asm/nmi.h | 2 + arch/powerpc/kernel/exceptions-64s.S | 92 +++++++++++++++++++---- arch/powerpc/kernel/mce.c | 3 + arch/powerpc/kernel/traps.c | 91 +++++++++++++++++++++- 5 files changed, 179 insertions(+), 17 deletions(-)
Signature · 1 line
2.18.0
Nicholas Piggin
reviewerPATCH v4 1/4 powerpc/64s: Fix HV NMI vs HV interrupt recoverability test
HV interrupts that use HSRR registers do not enter with MSR[RI] clear, but their entry code is not recoverable vs NMI, due to shared use of HSPRG1 as a scratch register to save r13.
This means that a system reset or machine check that hits in HSRR interrupt entry can cause r13 to be silently corrupted.
Fix this by marking NMIs non-recoverable if they land in HV interrupt ranges.
--- arch/powerpc/include/asm/asm-prototypes.h | 8 +++ arch/powerpc/include/asm/nmi.h | 2 + arch/powerpc/kernel/exceptions-64s.S | 8 +++ arch/powerpc/kernel/mce.c | 3 ++ arch/powerpc/kernel/traps.c | 66 +++++++++++++++++++++++ 5 files changed, 87 insertions(+)
arch/powerpc/include/asm/asm-prototypes.h +8 -0
@@ -51,6 +51,14 @@ int exit_vmx_usercopy(void);int enter_vmx_ops (void );void *exit_vmx_ops (void *dest); +/* Exceptions */ +#ifdef CONFIG_PPC_POWERNV +extern unsigned long real_trampolines_start; +extern unsigned long real_trampolines_end; +extern unsigned long virt_trampolines_start; +extern unsigned long virt_trampolines_end; +#endif +/* Traps */ long machine_check_early (struct pt_regs *regs);long hmi_exception_realmode (struct pt_regs *regs);
arch/powerpc/include/asm/nmi.h +2 -0
@@ -14,4 +14,6 @@ extern void arch_trigger_cpumask_backtrace(const cpumask_t *mask,#define arch_trigger_cpumask_backtrace arch_trigger_cpumask_backtrace#endif +extern void hv_nmi_check_nonrecoverable (struct pt_regs *regs); +#endif /* _ASM_NMI_H */
arch/powerpc/kernel/exceptions-64s.S +8 -0
@@ -68,6 +68,14 @@ OPEN_FIXED_SECTION(real_vectors, 0x0100, 0x1900) OPEN_FIXED_SECTION(real_trampolines, 0x1900, 0x4000) OPEN_FIXED_SECTION(virt_vectors, 0x4000, 0x5900) OPEN_FIXED_SECTION(virt_trampolines, 0x5900, 0x7000) + +#ifdef CONFIG_PPC_POWERNV + .globl real_trampolines_start + .globl real_trampolines_end + .globl virt_trampolines_start + .globl virt_trampolines_end +#endif + #if defined(CONFIG_PPC_PSERIES) || defined(CONFIG_PPC_POWERNV) /* * Data area reserved for FWNMI option.
arch/powerpc/kernel/mce.c +3 -0
@@ -31,6 +31,7 @@#include <asm/machdep.h> #include <asm/mce.h> +#include <asm/nmi.h> static DEFINE_PER_CPU(int, mce_nest_count); static DEFINE_PER_CPU(struct machine_check_event[MAX_MC_EVT], mce_event);
@@ -488,6 +489,8 @@ long machine_check_early(struct pt_regs *regs) {long handled =0 ; +hv_nmi_check_nonrecoverable (regs); +/* * See if platform is capable of handling machine check. */
arch/powerpc/kernel/traps.c +66 -0
78 lines not on this page · raw message
Signature · 1 line
2.18.0
Re: [v4, 1/4] powerpc/64s: Fix HV NMI vs HV interrupt recoverability test
Quoted 11 lines · Nicholas Piggin
HV interrupts that use HSRR registers do not enter with MSR[RI] clear, but their entry code is not recoverable vs NMI, due to shared use of HSPRG1 as a scratch register to save r13.
This means that a system reset or machine check that hits in HSRR interrupt entry can cause r13 to be silently corrupted.
Fix this by marking NMIs non-recoverable if they land in HV interrupt ranges.
Signed-off-by Nicholas Piggin <[email protected]>
Series applied to powerpc next, thanks.
https://git.kernel.org/powerpc/c/ccd477028a202993b9ddca5d2404fdac
cheers
Nicholas Piggin
reviewerPATCH v4 2/4 powerpc/64s: system reset interrupt preserve HSRRs
In reply to Nicholas PigginCode that uses HSRR registers is not required to clear MSR[RI] by convention, however the system reset NMI itself may use HSRR registers (e.g., to call OPAL) and clobber them.
Rather than introduce the requirement to clear RI in order to use HSRRs, have system reset interrupt save and restore HSRRs.
--- arch/powerpc/kernel/traps.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-)
arch/powerpc/kernel/traps.c +24 -1
@@ -442,14 +442,32 @@ void hv_nmi_check_nonrecoverable(struct pt_regs *regs)void system_reset_exception (struct pt_regs *regs) { +unsigned long hsrr0, hsrr1; +bool nested =in_nmi (); +bool saved_hsrrs = false; +/* * Avoid crashes in case of nested NMI exceptions. Recoverability * is determined by RI and in_nmi */ -bool nested =in_nmi ();if (!nested)nmi_enter (); +/* +* System reset can interrupt code where HSRRs are live and MSR[RI]=1. +* The system reset interrupt itself may clobber HSRRs (e.g., to call +* OPAL), so save them here and restore them before returning. +* +* Machine checks don't need to save HSRRs, as the real mode handler +* is careful to avoid them, and the regular handler is not delivered +* as an NMI. +*/ +if (cpu_has_feature (CPU_FTR_HVMODE )) { + hsrr0 =mfspr (SPRN_HSRR0 ); + hsrr1 =mfspr (SPRN_HSRR1 ); + saved_hsrrs = true; + } +hv_nmi_check_nonrecoverable (regs);__this_cpu_inc (irq_stat.sreset_irqs);
@@ -499,6 +517,11 @@ void system_reset_exception(struct pt_regs *regs)if (!(regs->msr &MSR_RI ))nmi_panic (regs,"Unrecoverable System Reset" ); +if (saved_hsrrs) { +mtspr (SPRN_HSRR0 , hsrr0); +mtspr (SPRN_HSRR1 , hsrr1); + } +if (!nested)nmi_exit ();
Signature · 1 line
2.18.0
Nicholas Piggin
reviewerPATCH v4 3/4 powerpc/64s: Prepare to handle data interrupts vs d-side MCE reentrancy
In reply to Nicholas PigginA subsequent fix for data interrupts (those that set DAR / DSISR) requires some interrupt macros to be open-coded, and also requires the 0x300 interrupt handler to be moved out-of-line.
This patch does that without changing behaviour, which makes the later fix a smaller change.
--- arch/powerpc/kernel/exceptions-64s.S | 48 ++++++++++++++++++++++++---- 1 file changed, 42 insertions(+), 6 deletions(-)
arch/powerpc/kernel/exceptions-64s.S +42 -6
@@ -574,8 +574,23 @@ EXC_COMMON_BEGIN(mce_return) RFI_TO_KERNEL b . -EXC_REAL(data_access, 0x300, 0x80) -EXC_VIRT(data_access, 0x4300, 0x80, 0x300) +EXC_REAL_BEGIN(data_access, 0x300, 0x80) +SET_SCRATCH0(r13) /* save r13 */ +EXCEPTION_PROLOG_0(PACA_EXGEN) + b tramp_real_data_access +EXC_REAL_END(data_access, 0x300, 0x80) + +TRAMP_REAL_BEGIN(tramp_real_data_access) +EXCEPTION_PROLOG_1(PACA_EXGEN, KVMTEST_PR, 0x300) +EXCEPTION_PROLOG_2(data_access_common, EXC_STD) + +EXC_VIRT_BEGIN(data_access, 0x4300, 0x80) +SET_SCRATCH0(r13) /* save r13 */ +EXCEPTION_PROLOG_0(PACA_EXGEN) +EXCEPTION_PROLOG_1(PACA_EXGEN, NOTEST, 0x300) +EXCEPTION_PROLOG_2_RELON(data_access_common, EXC_STD) +EXC_VIRT_END(data_access, 0x4300, 0x80) + TRAMP_KVM_SKIP(PACA_EXGEN, 0x300) EXC_COMMON_BEGIN(data_access_common)
@@ -604,11 +619,20 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) EXC_REAL_BEGIN(data_access_slb, 0x380, 0x80) -EXCEPTION_PROLOG(PACA_EXSLB, data_access_slb_common, EXC_STD, KVMTEST_PR, 0x380); +SET_SCRATCH0(r13) /* save r13 */ +EXCEPTION_PROLOG_0(PACA_EXSLB) + b tramp_real_data_access_slb EXC_REAL_END(data_access_slb, 0x380, 0x80) +TRAMP_REAL_BEGIN(tramp_real_data_access_slb) +EXCEPTION_PROLOG_1(PACA_EXSLB, KVMTEST_PR, 0x380) +EXCEPTION_PROLOG_2(data_access_slb_common, EXC_STD) + EXC_VIRT_BEGIN(data_access_slb, 0x4380, 0x80) -EXCEPTION_RELON_PROLOG(PACA_EXSLB, data_access_slb_common, EXC_STD, NOTEST, 0x380); +SET_SCRATCH0(r13) /* save r13 */ +EXCEPTION_PROLOG_0(PACA_EXSLB) +EXCEPTION_PROLOG_1(PACA_EXSLB, NOTEST, 0x380) +EXCEPTION_PROLOG_2_RELON(data_access_slb_common, EXC_STD) EXC_VIRT_END(data_access_slb, 0x4380, 0x80) TRAMP_KVM_SKIP(PACA_EXSLB, 0x380)
@@ -711,8 +735,20 @@ TRAMP_KVM_HV(PACA_EXGEN, 0x500) EXC_COMMON_ASYNC(hardware_interrupt_common, 0x500, do_IRQ) -EXC_REAL(alignment, 0x600, 0x100) -EXC_VIRT(alignment, 0x4600, 0x100, 0x600) +EXC_REAL_BEGIN(alignment, 0x600, 0x100) +SET_SCRATCH0(r13) /* save r13 */ +EXCEPTION_PROLOG_0(PACA_EXGEN) +EXCEPTION_PROLOG_1(PACA_EXGEN, KVMTEST_PR, 0x600) +EXCEPTION_PROLOG_2(alignment_common, EXC_STD) +EXC_REAL_END(alignment, 0x600, 0x100) + +EXC_VIRT_BEGIN(alignment, 0x4600, 0x100) +SET_SCRATCH0(r13) /* save r13 */ +EXCEPTION_PROLOG_0(PACA_EXGEN) +EXCEPTION_PROLOG_1(PACA_EXGEN, NOTEST, 0x600) +EXCEPTION_PROLOG_2_RELON(alignment_common, EXC_STD) +EXC_VIRT_END(alignment, 0x4600, 0x100) + TRAMP_KVM(PACA_EXGEN, 0x600) EXC_COMMON_BEGIN(alignment_common) mfspr r10,SPRN_DAR
Signature · 1 line
2.18.0
Nicholas Piggin
reviewerPATCH v4 4/4 powerpc/64s: Fix data interrupts vs d-side MCE reentrancy
In reply to Nicholas PigginHandlers for interrupts that set DAR / DSISR, set MSR[RI] before those SPRs are read. If a d-side machine check hits in this window, DAR / DSISR will be clobbered silently, leading to random corruption.
Fix this by having handlers save those registers before setting MSR[RI].
--- arch/powerpc/kernel/exceptions-64s.S | 36 ++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 10 deletions(-)
arch/powerpc/kernel/exceptions-64s.S +26 -10
@@ -582,12 +582,25 @@ EXC_REAL_END(data_access, 0x300, 0x80) TRAMP_REAL_BEGIN(tramp_real_data_access) EXCEPTION_PROLOG_1(PACA_EXGEN, KVMTEST_PR, 0x300) + /* + * DAR/DSISR must be read before setting MSR[RI], because + * a d-side MCE will clobber those registers so is not + * recoverable if they are live. + */ + mfspr r10,SPRN_DAR + mfspr r11,SPRN_DSISR + std r10,PACA_EXGEN+EX_DAR(r13) + stw r11,PACA_EXGEN+EX_DSISR(r13) EXCEPTION_PROLOG_2(data_access_common, EXC_STD) EXC_VIRT_BEGIN(data_access, 0x4300, 0x80) SET_SCRATCH0(r13) /* save r13 */ EXCEPTION_PROLOG_0(PACA_EXGEN) EXCEPTION_PROLOG_1(PACA_EXGEN, NOTEST, 0x300) + mfspr r10,SPRN_DAR + mfspr r11,SPRN_DSISR + std r10,PACA_EXGEN+EX_DAR(r13) + stw r11,PACA_EXGEN+EX_DSISR(r13) EXCEPTION_PROLOG_2_RELON(data_access_common, EXC_STD) EXC_VIRT_END(data_access, 0x4300, 0x80)
@@ -598,11 +611,8 @@ EXC_COMMON_BEGIN(data_access_common) * Here r13 points to the paca, r9 contains the saved CR, * SRR0 and SRR1 are saved in r11 and r12, * r9 - r13 are saved in paca->exgen. + * EX_DAR and EX_DSISR have saved DAR/DSISR */ - mfspr r10,SPRN_DAR - std r10,PACA_EXGEN+EX_DAR(r13) - mfspr r10,SPRN_DSISR - stw r10,PACA_EXGEN+EX_DSISR(r13) EXCEPTION_PROLOG_COMMON(0x300, PACA_EXGEN) RECONCILE_IRQ_STATE(r10, r11) ld r12,_MSR(r1)
@@ -626,20 +636,22 @@ EXC_REAL_END(data_access_slb, 0x380, 0x80) TRAMP_REAL_BEGIN(tramp_real_data_access_slb) EXCEPTION_PROLOG_1(PACA_EXSLB, KVMTEST_PR, 0x380) + mfspr r10,SPRN_DAR + std r10,PACA_EXSLB+EX_DAR(r13) EXCEPTION_PROLOG_2(data_access_slb_common, EXC_STD) EXC_VIRT_BEGIN(data_access_slb, 0x4380, 0x80) SET_SCRATCH0(r13) /* save r13 */ EXCEPTION_PROLOG_0(PACA_EXSLB) EXCEPTION_PROLOG_1(PACA_EXSLB, NOTEST, 0x380) + mfspr r10,SPRN_DAR + std r10,PACA_EXSLB+EX_DAR(r13) EXCEPTION_PROLOG_2_RELON(data_access_slb_common, EXC_STD) EXC_VIRT_END(data_access_slb, 0x4380, 0x80) TRAMP_KVM_SKIP(PACA_EXSLB, 0x380) EXC_COMMON_BEGIN(data_access_slb_common) - mfspr r10,SPRN_DAR - std r10,PACA_EXSLB+EX_DAR(r13) EXCEPTION_PROLOG_COMMON(0x380, PACA_EXSLB) ld r4,PACA_EXSLB+EX_DAR(r13) std r4,_DAR(r1)
@@ -739,6 +751,10 @@ EXC_REAL_BEGIN(alignment, 0x600, 0x100) SET_SCRATCH0(r13) /* save r13 */ EXCEPTION_PROLOG_0(PACA_EXGEN) EXCEPTION_PROLOG_1(PACA_EXGEN, KVMTEST_PR, 0x600) + mfspr r10,SPRN_DAR + mfspr r11,SPRN_DSISR + std r10,PACA_EXGEN+EX_DAR(r13) + stw r11,PACA_EXGEN+EX_DSISR(r13) EXCEPTION_PROLOG_2(alignment_common, EXC_STD) EXC_REAL_END(alignment, 0x600, 0x100)
@@ -746,15 +762,15 @@ EXC_VIRT_BEGIN(alignment, 0x4600, 0x100) SET_SCRATCH0(r13) /* save r13 */ EXCEPTION_PROLOG_0(PACA_EXGEN) EXCEPTION_PROLOG_1(PACA_EXGEN, NOTEST, 0x600) + mfspr r10,SPRN_DAR + mfspr r11,SPRN_DSISR + std r10,PACA_EXGEN+EX_DAR(r13) + stw r11,PACA_EXGEN+EX_DSISR(r13) EXCEPTION_PROLOG_2_RELON(alignment_common, EXC_STD) EXC_VIRT_END(alignment, 0x4600, 0x100) TRAMP_KVM(PACA_EXGEN, 0x600) EXC_COMMON_BEGIN(alignment_common) - mfspr r10,SPRN_DAR - std r10,PACA_EXGEN+EX_DAR(r13) - mfspr r10,SPRN_DSISR - stw r10,PACA_EXGEN+EX_DSISR(r13) EXCEPTION_PROLOG_COMMON(0x600, PACA_EXGEN) ld r3,PACA_EXGEN+EX_DAR(r13) lwz r4,PACA_EXGEN+EX_DSISR(r13)
Signature · 1 line
2.18.0