powerpc/64: fast interrupt exits
Subsystems (4)
Nicholas Piggin
reviewerv4 cover powerpc/64: fast interrupt exits
This series attempts to improve the speed of interrupts and system calls in three major ways.
Firstly, the SRR/HSRR registers do not need to be reloaded if they were clobbered for the duration of the interrupt and the return NIP and MSR did not changed. 64e does not implement this part, but it could quite easily.
Secondly, mtmsrd and mtspr are reduced by various means, this is mostly specific to 64s.
Lastly, an alternate interrupt return location facility is added for soft-masked asynchronous interrupts, and then that's used to set everything up for return without having to disable MSR RI or EE.
After this series, the entire system call / interrupt handler fast path can execute with no mtsprs and one mtmsrd to enable interrupts initially, and the system call vectored path doesn't even need to do that. This gives a decent performance benefit. On POWER9 with a powernv_defconfig without VIRT_CPU_ACCOUNTING_NATIVE, no meltdown workarounds, gettid sc system call goes from 481 -> 344 cycles, gettid scv 345->299 cycles, and page fault 1225->1064 cycles.
Since v3:
- Added Christophe's series to the end (with one broken out patch at the start and minor tweaks).
- Fix lbz of SOFTE that broke on BE, should be loaded with ld.
- Fix case where fast_interrupt_return was not marking SRRs as clobbered correctly, causing random userspace segfaults etc.
- Fix cpu lock inversion problem in the workaround static branch switching code.
- Removed the .L local name from some labels for the restart handlers, which makes the disassembly easier to read.
- Fixed several nitpicks found by kbuild robot (duplicate include, missing prototypes, etc).
- Fix 64e compile bug [from mpe].
- Fix KUAP re-locking in some restart cases.
Christophe Leroy (6):
powerpc/interrupt: Fix CONFIG ifdef typo powerpc/interrupt: Rename and lightly change
syscall_exit_prepare_main()
powerpc/interrupt: Refactor interrupt_exit_user_prepare() powerpc/interrupt: Interchange
prep_irq_for_{kernel_enabled/user}_exit()
powerpc/interrupt: Refactor prep_irq_for_{user/kernel_enabled}_exit() powerpc/interrupt: Remove prep_irq_for_user_exit()
Nicholas Piggin (11):
powerpc: remove interrupt exit helpers unused argument powerpc/64s: introduce different functions to return from SRR vs HSRR
interrupts
powerpc/64s: avoid reloading (H)SRR registers if they are still valid powerpc/64: handle MSR EE and RI in interrupt entry wrapper powerpc/64: move interrupt return asm to interrupt_64.S powerpc/64s: system call avoid setting MSR[RI] until we set MSR[EE] powerpc/64s: save one more register in the masked interrupt handler powerpc/64: allow alternate return locations for soft-masked
interrupts
powerpc/64: interrupt soft-enable race fix
powerpc/64: treat low kernel text as irqs soft-masked powerpc/64: use interrupt restart table to speed up return from
interrupt
arch/powerpc/Kconfig.debug | 5 + arch/powerpc/include/asm/asm-prototypes.h | 9 +- arch/powerpc/include/asm/head-64.h | 2 +- arch/powerpc/include/asm/hw_irq.h | 23 +- arch/powerpc/include/asm/interrupt.h | 60 +- arch/powerpc/include/asm/paca.h | 7 + arch/powerpc/include/asm/ppc_asm.h | 8 + arch/powerpc/include/asm/ptrace.h | 75 ++- arch/powerpc/kernel/asm-offsets.c | 7 + arch/powerpc/kernel/entry_64.S | 516 -------------- arch/powerpc/kernel/exceptions-64e.S | 53 +- arch/powerpc/kernel/exceptions-64s.S | 219 +++--- arch/powerpc/kernel/fpu.S | 4 + arch/powerpc/kernel/head_64.S | 5 +- arch/powerpc/kernel/interrupt.c | 413 +++++++----- arch/powerpc/kernel/interrupt_64.S | 742 +++++++++++++++++++++ arch/powerpc/kernel/irq.c | 95 +++ arch/powerpc/kernel/kgdb.c | 2 +- arch/powerpc/kernel/kprobes-ftrace.c | 2 +- arch/powerpc/kernel/kprobes.c | 10 +- arch/powerpc/kernel/process.c | 26 +- arch/powerpc/kernel/prom_init.c | 3 + arch/powerpc/kernel/rtas.c | 14 +- arch/powerpc/kernel/signal.c | 2 +- arch/powerpc/kernel/signal_64.c | 9 + arch/powerpc/kernel/syscalls.c | 3 +- arch/powerpc/kernel/traps.c | 18 +- arch/powerpc/kernel/vector.S | 8 +- arch/powerpc/kernel/vmlinux.lds.S | 10 + arch/powerpc/kvm/book3s_hv.c | 3 + arch/powerpc/kvm/book3s_pr.c | 2 + arch/powerpc/lib/Makefile | 2 +- arch/powerpc/lib/feature-fixups.c | 52 +- arch/powerpc/lib/restart_table.c | 30 + arch/powerpc/lib/sstep.c | 4 +- arch/powerpc/math-emu/math.c | 2 +- arch/powerpc/platforms/powernv/opal-call.c | 4 + arch/powerpc/platforms/pseries/hvCall.S | 29 + arch/powerpc/sysdev/fsl_pci.c | 2 +- 39 files changed, 1622 insertions(+), 858 deletions(-) create mode 100644 arch/powerpc/kernel/interrupt_64.S create mode 100644 arch/powerpc/lib/restart_table.c
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 1/17 powerpc/interrupt: Fix CONFIG ifdef typo
From: Christophe Leroy <[email protected]>
CONFIG_PPC_BOOK3S should be CONFIG_PPC_BOOK3S_64. restore_math is a no-op for other configurations.
--- arch/powerpc/kernel/interrupt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/kernel/interrupt.c +1 -1
@@ -303,7 +303,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3, ti_flags =READ_ONCE (current_thread_info ()->flags); } -if (IS_ENABLED (CONFIG_PPC_BOOK3S ) &&IS_ENABLED (CONFIG_PPC_FPU )) { +if (IS_ENABLED (CONFIG_PPC_BOOK3S_64 ) &&IS_ENABLED (CONFIG_PPC_FPU )) {if (IS_ENABLED (CONFIG_PPC_TRANSACTIONAL_MEM ) &&unlikely ((ti_flags & _TIF_RESTORE_TM))) {restore_tm_state (regs);
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 2/17 powerpc: remove interrupt exit helpers unused argument
In reply to Nicholas PigginThe msr argument is not used, remove it.
--- arch/powerpc/include/asm/asm-prototypes.h | 4 ++-- arch/powerpc/kernel/interrupt.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
arch/powerpc/include/asm/asm-prototypes.h +2 -2
@@ -71,8 +71,8 @@ void __init machine_init(u64 dt_ptr);#endif long system_call_exception (long r3,long r4,long r5,long r6,long r7,long r8,unsigned long r0,struct pt_regs *regs);notrace unsigned long syscall_exit_prepare (unsigned long r3,struct pt_regs *regs,long scv); -notrace unsigned long interrupt_exit_user_prepare (struct pt_regs *regs,unsigned long msr); -notrace unsigned long interrupt_exit_kernel_prepare (struct pt_regs *regs,unsigned long msr); +notrace unsigned long interrupt_exit_user_prepare (struct pt_regs *regs); +notrace unsigned long interrupt_exit_kernel_prepare (struct pt_regs *regs);long ppc_fadvise64_64 (int fd,int advice,u32 offset_high,u32 offset_low,u32 len_high,u32 len_low);
arch/powerpc/kernel/interrupt.c +2 -2
@@ -352,7 +352,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,return ret; } -notrace unsigned long interrupt_exit_user_prepare (struct pt_regs *regs,unsigned long msr) +notrace unsigned long interrupt_exit_user_prepare (struct pt_regs *regs) {unsigned long ti_flags;unsigned long flags;
@@ -431,7 +431,7 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsignedvoid preempt_schedule_irq (void ); -notrace unsigned long interrupt_exit_kernel_prepare (struct pt_regs *regs,unsigned long msr) +notrace unsigned long interrupt_exit_kernel_prepare (struct pt_regs *regs) {unsigned long flags;unsigned long ret =0 ;
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 3/17 powerpc/64s: introduce different functions to return from SRR vs HSRR interrupts
In reply to Nicholas PigginThis makes no real difference yet except that HSRR type interrupts will use hrfid to return. This is important for the next patch.
--- arch/powerpc/kernel/entry_64.S | 65 +++++++++++++++++------- arch/powerpc/kernel/exceptions-64e.S | 4 ++ arch/powerpc/kernel/exceptions-64s.S | 76 +++++++++++++++------------- arch/powerpc/kernel/vector.S | 2 +- 4 files changed, 92 insertions(+), 55 deletions(-)
arch/powerpc/kernel/entry_64.S +47 -18
@@ -635,51 +635,57 @@ END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_207S) * touched, no exit work created, then this can be used. */ .balign IFETCH_ALIGN_BYTES - .globl fast_interrupt_return -fast_interrupt_return: -_ASM_NOKPROBE_SYMBOL(fast_interrupt_return) + .globl fast_interrupt_return_srr +fast_interrupt_return_srr: +_ASM_NOKPROBE_SYMBOL(fast_interrupt_return_srr) kuap_check_amr r3, r4 ld r5,_MSR(r1) andi. r0,r5,MSR_PR #ifdef CONFIG_PPC_BOOK3S - bne .Lfast_user_interrupt_return_amr + bne .Lfast_user_interrupt_return_amr_srr kuap_kernel_restore r3, r4 andi. r0,r5,MSR_RI li r3,0 /* 0 return value, no EMULATE_STACK_STORE */ - bne+ .Lfast_kernel_interrupt_return + bne+ .Lfast_kernel_interrupt_return_srr addi r3,r1,STACK_FRAME_OVERHEAD bl unrecoverable_exception b . /* should not get here */ #else - bne .Lfast_user_interrupt_return - b .Lfast_kernel_interrupt_return + bne .Lfast_user_interrupt_return_srr + b .Lfast_kernel_interrupt_return_srr #endif +.macro interrupt_return_macro srr .balign IFETCH_ALIGN_BYTES - .globl interrupt_return -interrupt_return: -_ASM_NOKPROBE_SYMBOL(interrupt_return) + .globl interrupt_return_\srr +interrupt_return_\srr\(): +_ASM_NOKPROBE_SYMBOL(interrupt_return_\srr\()) ld r4,_MSR(r1) andi. r0,r4,MSR_PR - beq .Lkernel_interrupt_return + beq .Lkernel_interrupt_return_\srr addi r3,r1,STACK_FRAME_OVERHEAD bl interrupt_exit_user_prepare cmpdi r3,0 - bne- .Lrestore_nvgprs + bne- .Lrestore_nvgprs_\srr #ifdef CONFIG_PPC_BOOK3S -.Lfast_user_interrupt_return_amr: +.Lfast_user_interrupt_return_amr_\srr\(): kuap_user_restore r3, r4 #endif -.Lfast_user_interrupt_return: +.Lfast_user_interrupt_return_\srr\(): ld r11,_NIP(r1) ld r12,_MSR(r1) BEGIN_FTR_SECTION ld r10,_PPR(r1) mtspr SPRN_PPR,r10 END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR) + .ifc \srr,srr mtspr SPRN_SRR0,r11 mtspr SPRN_SRR1,r12 + .else + mtspr SPRN_HSRR0,r11 + mtspr SPRN_HSRR1,r12 + .endif BEGIN_FTR_SECTION stdcx. r0,0,r1 /* to clear the reservation */
@@ -706,24 +712,33 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_STCX_CHECKS_ADDRESS) REST_GPR(6, r1) REST_GPR(0, r1) REST_GPR(1, r1) + .ifc \srr,srr RFI_TO_USER + .else + HRFI_TO_USER + .endif b . /* prevent speculative execution */ -.Lrestore_nvgprs: +.Lrestore_nvgprs_\srr\(): REST_NVGPRS(r1) - b .Lfast_user_interrupt_return + b .Lfast_user_interrupt_return_\srr .balign IFETCH_ALIGN_BYTES -.Lkernel_interrupt_return: +.Lkernel_interrupt_return_\srr\(): addi r3,r1,STACK_FRAME_OVERHEAD bl interrupt_exit_kernel_prepare -.Lfast_kernel_interrupt_return: +.Lfast_kernel_interrupt_return_\srr\(): cmpdi cr1,r3,0 ld r11,_NIP(r1) ld r12,_MSR(r1) + .ifc \srr,srr mtspr SPRN_SRR0,r11 mtspr SPRN_SRR1,r12 + .else + mtspr SPRN_HSRR0,r11 + mtspr SPRN_HSRR1,r12 + .endif BEGIN_FTR_SECTION stdcx. r0,0,r1 /* to clear the reservation */
@@ -757,7 +772,11 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_STCX_CHECKS_ADDRESS) REST_GPR(6, r1) REST_GPR(0, r1) REST_GPR(1, r1) + .ifc \srr,srr RFI_TO_KERNEL + .else + HRFI_TO_KERNEL + .endif b . /* prevent speculative execution */ 1: /*
@@ -777,8 +796,18 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_STCX_CHECKS_ADDRESS) std r9,0(r1) /* perform store component of stdu */ ld r9,PACA_EXGEN+0(r13) + .ifc \srr,srr RFI_TO_KERNEL + .else + HRFI_TO_KERNEL + .endif b . /* prevent speculative execution */ +.endm + +interrupt_return_macro srr +#ifdef CONFIG_PPC_BOOK3S +interrupt_return_macro hsrr +#endif #ifdef CONFIG_PPC_RTAS /*
arch/powerpc/kernel/exceptions-64e.S +4 -0
@@ -26,6 +26,10 @@ #include <asm/feature-fixups.h> #include <asm/context_tracking.h> +/* 64e interrupt returns always use SRR registers */ +#define fast_interrupt_return fast_interrupt_return_srr +#define interrupt_return interrupt_return_srr + /* XXX This will ultimately add space for a special exception save * structure used to save things like SRR0/SRR1, SPRGs, MAS, etc... * when taking special interrupts. For now we don't support that,
arch/powerpc/kernel/exceptions-64s.S +40 -36
288 lines not on this page · raw message
arch/powerpc/kernel/vector.S +1 -1
@@ -131,7 +131,7 @@ _GLOBAL(load_up_vsx) /* enable use of VSX after return */ oris r12,r12,MSR_VSX@h std r12,_MSR(r1) - b fast_interrupt_return + b fast_interrupt_return_srr #endif /* CONFIG_VSX */
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 4/17 powerpc/64s: avoid reloading (H)SRR registers if they are still valid
In reply to Nicholas PigginWhen an interrupt is taken, the SRR registers are set to return to where it left off. Unless they are modified in the meantime, or the return address or MSR are modified, there is no need to reload these registers when returning from interrupt.
Introduce per-CPU flags that track the validity of SRR and HSRR registers. These are cleared when returning from interrupt, when using the registers for something else (e.g., OPAL calls), when adjusting the return address or MSR of a context, and when context switching (which changes the return address and MSR).
This improves the performance of interrupt returns.
--- arch/powerpc/Kconfig.debug | 5 ++ arch/powerpc/include/asm/hw_irq.h | 10 ++- arch/powerpc/include/asm/interrupt.h | 12 +++ arch/powerpc/include/asm/paca.h | 4 + arch/powerpc/include/asm/ptrace.h | 74 ++++++++++++----- arch/powerpc/kernel/asm-offsets.c | 4 + arch/powerpc/kernel/entry_64.S | 92 ++++++++++++++++++++-- arch/powerpc/kernel/exceptions-64s.S | 27 +++++++ arch/powerpc/kernel/fpu.S | 4 + arch/powerpc/kernel/kgdb.c | 2 +- arch/powerpc/kernel/kprobes-ftrace.c | 2 +- arch/powerpc/kernel/kprobes.c | 10 +-- arch/powerpc/kernel/process.c | 26 +++--- arch/powerpc/kernel/prom_init.c | 3 + arch/powerpc/kernel/rtas.c | 14 +++- arch/powerpc/kernel/signal.c | 2 +- arch/powerpc/kernel/signal_64.c | 9 +++ arch/powerpc/kernel/syscalls.c | 3 +- arch/powerpc/kernel/traps.c | 18 ++--- arch/powerpc/kernel/vector.S | 6 ++ arch/powerpc/kvm/book3s_hv.c | 3 + arch/powerpc/kvm/book3s_pr.c | 2 + arch/powerpc/lib/sstep.c | 4 +- arch/powerpc/math-emu/math.c | 2 +- arch/powerpc/platforms/powernv/opal-call.c | 4 + arch/powerpc/platforms/pseries/hvCall.S | 29 +++++++ arch/powerpc/sysdev/fsl_pci.c | 2 +- 27 files changed, 310 insertions(+), 63 deletions(-)
arch/powerpc/Kconfig.debug +5 -0
@@ -84,6 +84,11 @@ config MSI_BITMAP_SELFTEST config PPC_IRQ_SOFT_MASK_DEBUG bool "Include extra checks for powerpc irq soft masking" + depends on PPC64 + +config PPC_RFI_SRR_DEBUG + bool "Include extra checks for RFI SRR register validity" + depends on PPC_BOOK3S_64 config XMON bool "Include xmon kernel debugger"
arch/powerpc/include/asm/hw_irq.h +9 -1
@@ -389,7 +389,15 @@ static inline bool arch_irq_disabled_regs(struct pt_regs *regs)return !(regs->msr &MSR_EE ); } -static inline void may_hard_irq_enable (void ) { } +static inline bool may_hard_irq_enable (void ) +{ +return false; +} + +static inline void do_hard_irq_enable (void ) +{ +BUILD_BUG (); +}static inline void irq_soft_mask_regs_set_state (struct pt_regs *regs,unsigned long val) {
arch/powerpc/include/asm/interrupt.h +12 -0
@@ -73,6 +73,18 @@#include <asm/kprobes.h> #include <asm/runlatch.h> +#ifdef CONFIG_PPC_BOOK3S_64 +static inline void srr_regs_clobbered (void ) +{ + local_paca->srr_valid =0 ; + local_paca->hsrr_valid =0 ; +} +#else +static inline void srr_regs_clobbered (void ) +{ +} +#endif +static inline void nap_adjust_return (struct pt_regs *regs) {#ifdef CONFIG_PPC_970_NAP
arch/powerpc/include/asm/paca.h +4 -0
@@ -169,6 +169,10 @@ struct paca_struct {u64 saved_msr;/* MSR saved here by enter_rtas */ #ifdef CONFIG_PPC_BOOK3E u16 trap_save;/* Used when bad stack is encountered */ +#endif +#ifdef CONFIG_PPC_BOOK3S_64 +u8 hsrr_valid;/* HSRRs set for HRFID */ +u8 srr_valid;/* SRRs set for RFID */ #endif u8 irq_soft_mask;/* mask for irq soft masking */ u8 irq_happened;/* irq happened while soft-disabled */
arch/powerpc/include/asm/ptrace.h +53 -21
86 lines not on this page · raw message
arch/powerpc/kernel/asm-offsets.c +4 -0
@@ -209,6 +209,10 @@ int main(void)OFFSET (PACATOC , paca_struct, kernel_toc);OFFSET (PACAKBASE , paca_struct, kernelbase);OFFSET (PACAKMSR , paca_struct, kernel_msr); +#ifdef CONFIG_PPC_BOOK3S_64 +OFFSET (PACAHSRR_VALID , paca_struct, hsrr_valid); +OFFSET (PACASRR_VALID , paca_struct, srr_valid); +#endif OFFSET (PACAIRQSOFTMASK , paca_struct, irq_soft_mask);OFFSET (PACAIRQHAPPENED , paca_struct, irq_happened);OFFSET (PACA_FTRACE_ENABLED , paca_struct, ftrace_enabled);
arch/powerpc/kernel/entry_64.S +85 -7
149 lines not on this page · raw message
arch/powerpc/kernel/exceptions-64s.S +27 -0
@@ -567,6 +567,20 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real) std r0,GPR0(r1) /* save r0 in stackframe */ std r10,GPR1(r1) /* save r1 in stackframe */ + /* Mark our [H]SRRs valid for return */ + li r10,1 + .if IHSRR_IF_HVMODE + BEGIN_FTR_SECTION + stb r10,PACAHSRR_VALID(r13) + FTR_SECTION_ELSE + stb r10,PACASRR_VALID(r13) + ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206) + .elseif IHSRR + stb r10,PACAHSRR_VALID(r13) + .else + stb r10,PACASRR_VALID(r13) + .endif + .if ISET_RI li r10,MSR_RI mtmsrd r10,1 /* Set MSR_RI */
@@ -666,10 +680,13 @@ END_FTR_SECTION_IFSET(CPU_FTR_CFAR) .macro EXCEPTION_RESTORE_REGS hsrr=0 /* Move original SRR0 and SRR1 into the respective regs */ ld r9,_MSR(r1) + li r10,0 .if \hsrr mtspr SPRN_HSRR1,r9 + stb r10,PACAHSRR_VALID(r13) .else mtspr SPRN_SRR1,r9 + stb r10,PACASRR_VALID(r13) .endif ld r9,_NIP(r1) .if \hsrr
@@ -1825,6 +1842,8 @@ EXC_COMMON_BEGIN(hdecrementer_common) * * Be careful to avoid touching the kernel stack. */ + li r10,0 + stb r10,PACAHSRR_VALID(r13) ld r10,PACA_EXGEN+EX_CTR(r13) mtctr r10 mtcrf 0x80,r9
@@ -2660,6 +2679,8 @@ BEGIN_FTR_SECTION ld r10,PACA_EXGEN+EX_CFAR(r13) mtspr SPRN_CFAR,r10 END_FTR_SECTION_IFSET(CPU_FTR_CFAR) + li r10,0 + stb r10,PACAHSRR_VALID(r13) ld r10,PACA_EXGEN+EX_R10(r13) ld r11,PACA_EXGEN+EX_R11(r13) ld r12,PACA_EXGEN+EX_R12(r13)
@@ -2828,6 +2849,12 @@ masked_interrupt: ori r11,r11,PACA_IRQ_HARD_DIS stb r11,PACAIRQHAPPENED(r13) 2: /* done */ + li r10,0 + .if \hsrr + stb r10,PACAHSRR_VALID(r13) + .else + stb r10,PACASRR_VALID(r13) + .endif ld r10,PACA_EXGEN+EX_CTR(r13) mtctr r10 mtcrf 0x80,r9
arch/powerpc/kernel/fpu.S +4 -0
@@ -103,6 +103,10 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX) ori r12,r12,MSR_FP or r12,r12,r4 std r12,_MSR(r1) +#ifdef CONFIG_PPC_BOOK3S_64 + li r4,0 + stb r4,PACASRR_VALID(r13) +#endif #endif li r4,1 stb r4,THREAD_LOAD_FP(r5)
arch/powerpc/kernel/kgdb.c +1 -1
@@ -147,7 +147,7 @@ static int kgdb_handle_breakpoint(struct pt_regs *regs)return 0 ;if (*(u32 *)regs->nip ==BREAK_INSTR ) - regs->nip +=BREAK_INSTR_SIZE ; +regs_add_return_ip (regs,BREAK_INSTR_SIZE );return 1 ; }
arch/powerpc/kernel/kprobes-ftrace.c +1 -1
@@ -48,7 +48,7 @@ void kprobe_ftrace_handler(unsigned long nip, unsigned long parent_nip,* Emulate singlestep (and also recover regs->nip) * as if there is a nop */ - regs->nip +=MCOUNT_INSN_SIZE ; +regs_add_return_ip (regs,MCOUNT_INSN_SIZE );if (unlikely (p->post_handler)) { kcb->kprobe_status =KPROBE_HIT_SSDONE ; p->post_handler (p, regs,0 );
arch/powerpc/kernel/kprobes.c +5 -5
@@ -178,7 +178,7 @@ static nokprobe_inline void prepare_singlestep(struct kprobe *p, struct pt_regs* variant as values in regs could play a part in * if the trap is taken or not */ - regs->nip = (unsigned long )p->ainsn.insn; +regs_set_return_ip (regs, (unsigned long )p->ainsn.insn); } static nokprobe_inline void save_previous_kprobe(struct kprobe_ctlblk *kcb)
@@ -415,7 +415,7 @@ static int trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs)* we end up emulating it in kprobe_handler(), which increments the nip * again. */ - regs->nip = orig_ret_address -4 ; +regs_set_return_ip (regs, orig_ret_address -4 ); regs->link = orig_ret_address;return 0 ;
@@ -450,7 +450,7 @@ int kprobe_post_handler(struct pt_regs *regs) }/* Adjust nip to after the single-stepped instruction */ - regs->nip = (unsigned long )cur->addr + len; +regs_set_return_ip (regs, (unsigned long )cur->addr + len); regs->msr |= kcb->kprobe_saved_msr;/*Restore back the original saved kprobes variables and continue. */
@@ -490,7 +490,7 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr)* and allow the page fault handler to continue as a * normal page fault. */ - regs->nip = (unsigned long )cur->addr; +regs_set_return_ip (regs, (unsigned long )cur->addr); regs->msr &= ~MSR_SINGLESTEP ;/* Turn off 'trace' bits */ regs->msr |= kcb->kprobe_saved_msr;if (kcb->kprobe_status ==KPROBE_REENTER )
@@ -523,7 +523,7 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr)* zero, try to fix up. */ if ((entry =search_exception_tables (regs->nip)) !=NULL ) { - regs->nip =extable_fixup (entry); +regs_set_return_ip (regs,extable_fixup (entry));return 1 ; }
arch/powerpc/kernel/process.c +16 -10
@@ -96,7 +96,8 @@ static void check_if_tm_restore_required(struct task_struct *tsk)if (tsk == current && tsk->thread.regs &&MSR_TM_ACTIVE (tsk->thread.regs->msr) && !test_thread_flag (TIF_RESTORE_TM )) { - tsk->thread.ckpt_regs.msr = tsk->thread.regs->msr; +regs_set_return_msr (&tsk->thread.ckpt_regs, + tsk->thread.regs->msr);set_thread_flag (TIF_RESTORE_TM ); } }
@@ -161,7 +162,7 @@ static void __giveup_fpu(struct task_struct *tsk) msr &= ~(MSR_FP |MSR_FE0 |MSR_FE1 );if (cpu_has_feature (CPU_FTR_VSX )) msr &= ~MSR_VSX ; - tsk->thread.regs->msr = msr; +regs_set_return_msr (tsk->thread.regs, msr); }void giveup_fpu (struct task_struct *tsk)
@@ -244,7 +245,7 @@ static void __giveup_altivec(struct task_struct *tsk) msr &= ~MSR_VEC ;if (cpu_has_feature (CPU_FTR_VSX )) msr &= ~MSR_VSX ; - tsk->thread.regs->msr = msr; +regs_set_return_msr (tsk->thread.regs, msr); }void giveup_altivec (struct task_struct *tsk)
@@ -559,7 +560,7 @@ void notrace restore_math(struct pt_regs *regs)msr_check_and_clear (new_msr); - regs->msr |= new_msr | fpexc_mode; + regs_set_return_msr(regs, regs->msr | new_msr | fpexc_mode); } }#endif /* CONFIG_PPC_BOOK3S_64 */
@@ -1287,6 +1288,8 @@ struct task_struct *__switch_to(struct task_struct *prev, }#endif /* CONFIG_PPC_BOOK3S_64 */ +return_ip_or_msr_changed (); +return last; }
@@ -1845,6 +1848,9 @@ void start_thread(struct pt_regs *regs, unsigned long start, unsigned long sp) regs->gpr[2 ] =0 ; regs->msr = MSR_USER32; } + +return_ip_or_msr_changed (); +#endif #ifdef CONFIG_VSX current->thread.used_vsr =0 ;
@@ -1875,7 +1881,6 @@ void start_thread(struct pt_regs *regs, unsigned long start, unsigned long sp) current->thread.tm_tfiar =0 ; current->thread.load_tm =0 ;#endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ - }EXPORT_SYMBOL (start_thread);
@@ -1923,9 +1928,10 @@ int set_fpexc_mode(struct task_struct *tsk, unsigned int val)if (val >PR_FP_EXC_PRECISE )return -EINVAL ; tsk->thread.fpexc_mode =__pack_fe01 (val); -if (regs !=NULL && (regs->msr &MSR_FP ) !=0 ) - regs->msr = (regs->msr & ~(MSR_FE0 |MSR_FE1 )) - | tsk->thread.fpexc_mode; +if (regs !=NULL && (regs->msr &MSR_FP ) !=0 ) { +regs_set_return_msr (regs, (regs->msr & ~(MSR_FE0 |MSR_FE1 )) + | tsk->thread.fpexc_mode); + }return 0 ; }
@@ -1971,9 +1977,9 @@ int set_endian(struct task_struct *tsk, unsigned int val)return -EINVAL ;if (val ==PR_ENDIAN_BIG ) - regs->msr &= ~MSR_LE ; +regs_set_return_msr (regs, regs->msr & ~MSR_LE );else if (val ==PR_ENDIAN_LITTLE || val ==PR_ENDIAN_PPC_LITTLE ) - regs->msr |=MSR_LE ; +regs_set_return_msr (regs, regs->msr |MSR_LE );else return -EINVAL ;
arch/powerpc/kernel/prom_init.c +3 -0
@@ -31,6 +31,7 @@#include <asm/rtas.h> #include <asm/page.h> #include <asm/processor.h> +#include <asm/interrupt.h> #include <asm/irq.h> #include <asm/io.h> #include <asm/smp.h>
@@ -1762,6 +1763,8 @@ static int prom_rtas_hcall(uint64_t args) asmvolatile ("sc 1\n" :"=r" (arg1) :"r" (arg1),"r" (arg2) :); +srr_regs_clobbered (); +return arg1; }
arch/powerpc/kernel/rtas.c +11 -3
@@ -25,6 +25,7 @@#include <linux/reboot.h> #include <linux/syscalls.h> +#include <asm/interrupt.h> #include <asm/prom.h> #include <asm/rtas.h> #include <asm/hvcall.h>
@@ -46,6 +47,13 @@/* This is here deliberately so it's only used in this file */ void enter_rtas (unsigned long ); +static inline void do_enter_rtas (unsigned long args) +{ +enter_rtas (args); + +srr_regs_clobbered ();/* rtas uses SRRs, invalidate */ +} +struct rtas_t rtas = { .lock = __ARCH_SPIN_LOCK_UNLOCKED };
@@ -384,7 +392,7 @@ static char *__fetch_rtas_last_error(char *altbuf) save_args = rtas.args; rtas.args = err_args; -enter_rtas (__pa (&rtas.args)); +do_enter_rtas (__pa (&rtas.args)); err_args = rtas.args; rtas.args = save_args;
@@ -430,7 +438,7 @@ va_rtas_call_unlocked(struct rtas_args *args, int token, int nargs, int nret,for (i =0 ; i < nret; ++i) args->rets[i] =0 ; -enter_rtas (__pa (args)); +do_enter_rtas (__pa (args)); }void rtas_call_unlocked (struct rtas_args *args,int token,int nargs,int nret, ...)
@@ -1138,7 +1146,7 @@ SYSCALL_DEFINE1(rtas, struct rtas_args __user *, uargs) flags =lock_rtas (); rtas.args = args; -enter_rtas (__pa (&rtas.args)); +do_enter_rtas (__pa (&rtas.args)); args = rtas.args;/* A -1 return code indicates that the last command couldn't
arch/powerpc/kernel/signal.c +1 -1
@@ -214,7 +214,7 @@ static void check_syscall_restart(struct pt_regs *regs, struct k_sigaction *ka, regs->gpr[0 ] = __NR_restart_syscall; else regs->gpr[3] = regs->orig_gpr3; - regs->nip -=4 ; +regs_add_return_ip (regs,-4 ); regs->result =0 ; }else {if (trap_is_scv (regs)) {
arch/powerpc/kernel/signal_64.c +9 -0
@@ -720,6 +720,9 @@ SYSCALL_DEFINE3(swapcontext, struct ucontext __user *, old_ctx,/* This returns like rt_sigreturn */ set_thread_flag (TIF_RESTOREALL ); + +return_ip_or_msr_changed (); +return 0 ; efault_out:
@@ -832,6 +835,9 @@ SYSCALL_DEFINE0(rt_sigreturn) goto badframe;set_thread_flag (TIF_RESTOREALL ); + +return_ip_or_msr_changed (); +return 0 ; badframe_block:
@@ -957,12 +963,15 @@ int handle_rt_signal64(struct ksignal *ksig, sigset_t *set,if (err) goto badframe; +return_ip_or_msr_changed (); +return 0 ; badframe_block:user_write_access_end (); badframe:signal_fault (current, regs,"handle_rt_signal64" , frame); +return_ip_or_msr_changed ();return 1 ; }
arch/powerpc/kernel/syscalls.c +2 -1
@@ -114,7 +114,8 @@ SYSCALL_DEFINE0(switch_endian) {struct thread_info *ti; - current->thread.regs->msr ^=MSR_LE ; +regs_set_return_msr (current->thread.regs, + current->thread.regs->msr ^MSR_LE );/* * Set TIF_RESTOREALL so that r3 isn't clobbered on return to
arch/powerpc/kernel/traps.c +9 -9
@@ -1031,7 +1031,7 @@ static void p9_hmi_special_emu(struct pt_regs *regs)#endif /* !__LITTLE_ENDIAN__ */ /* Go to next instruction */ - regs->nip += 4; + regs_add_return_ip(regs, 4); }#endif /* CONFIG_VSX */
@@ -1476,7 +1476,7 @@ static void do_program_check(struct pt_regs *regs)if (!(regs->msr &MSR_PR ) &&/* not user-mode */ report_bug (bugaddr, regs) ==BUG_TRAP_TYPE_WARN ) { - regs->nip +=4 ; +regs_add_return_ip (regs,4 );return ; }_exception (SIGTRAP , regs,TRAP_BRKPT , regs->nip);
@@ -1538,7 +1538,7 @@ static void do_program_check(struct pt_regs *regs)if (reason & (REASON_ILLEGAL |REASON_PRIVILEGED )) {switch (emulate_instruction (regs)) {case 0 : - regs->nip +=4 ; +regs_add_return_ip (regs,4 );emulate_single_step (regs);return ;case -EFAULT :
@@ -1593,7 +1593,7 @@ DEFINE_INTERRUPT_HANDLER(alignment_exception)if (fixed ==1 ) {/* skip over emulated instruction */ - regs->nip +=inst_length (reason); +regs_add_return_ip (regs,inst_length (reason));emulate_single_step (regs);return ; }
@@ -1751,7 +1751,7 @@ DEFINE_INTERRUPT_HANDLER(facility_unavailable_exception)pr_err ("DSCR based mfspr emulation failed\n" );return ; } - regs->nip +=4 ; +regs_add_return_ip (regs,4 );emulate_single_step (regs); }return ;
@@ -2044,7 +2044,7 @@ DEFINE_INTERRUPT_HANDLER(altivec_assist_exception)PPC_WARN_EMULATED (altivec, regs); err =emulate_altivec (regs);if (err ==0 ) { - regs->nip +=4 ;/* skip emulated instruction */ +regs_add_return_ip (regs,4 );/* skip emulated instruction */ emulate_single_step (regs);return ; }
@@ -2109,7 +2109,7 @@ DEFINE_INTERRUPT_HANDLER(SPEFloatingPointException) err =do_spe_mathemu (regs);if (err ==0 ) { - regs->nip +=4 ;/* skip emulated instruction */ +regs_add_return_ip (regs,4 );/* skip emulated instruction */ emulate_single_step (regs);return ; }
@@ -2140,10 +2140,10 @@ DEFINE_INTERRUPT_HANDLER(SPEFloatingPointRoundException)giveup_spe (current);preempt_enable (); - regs->nip -=4 ; +regs_add_return_ip (regs,-4 ); err =speround_handler (regs);if (err ==0 ) { - regs->nip +=4 ;/* skip emulated instruction */ +regs_add_return_ip (regs,4 );/* skip emulated instruction */ emulate_single_step (regs);return ; }
arch/powerpc/kernel/vector.S +6 -0
@@ -73,6 +73,10 @@ _GLOBAL(load_up_altivec) addi r5,r4,THREAD /* Get THREAD */ oris r12,r12,MSR_VEC@h std r12,_MSR(r1) +#ifdef CONFIG_PPC_BOOK3S_64 + li r4,0 + stb r4,PACASRR_VALID(r13) +#endif #endif li r4,1 stb r4,THREAD_LOAD_VEC(r5)
@@ -131,6 +135,8 @@ _GLOBAL(load_up_vsx) /* enable use of VSX after return */ oris r12,r12,MSR_VSX@h std r12,_MSR(r1) + li r4,0 + stb r4,PACASRR_VALID(r13) b fast_interrupt_return_srr #endif /* CONFIG_VSX */
arch/powerpc/kvm/book3s_hv.c +3 -0
@@ -4461,6 +4461,9 @@ static int kvmppc_vcpu_run_hv(struct kvm_vcpu *vcpu) vcpu->arch.state =KVMPPC_VCPU_NOTREADY ;atomic_dec (&kvm->arch.vcpus_running); + +srr_regs_clobbered (); +return r; }
arch/powerpc/kvm/book3s_pr.c +2 -0
@@ -25,6 +25,7 @@#include <asm/cputable.h> #include <asm/cacheflush.h> #include <linux/uaccess.h> +#include <asm/interrupt.h> #include <asm/io.h> #include <asm/kvm_ppc.h> #include <asm/kvm_book3s.h>
@@ -1848,6 +1849,7 @@ static int kvmppc_vcpu_run_pr(struct kvm_vcpu *vcpu)/* Make sure we save the guest TAR/EBB/DSCR state */ kvmppc_giveup_fac (vcpu,FSCR_TAR_LG ); +srr_regs_clobbered (); out: vcpu->mode =OUTSIDE_GUEST_MODE ;return ret;
arch/powerpc/lib/sstep.c +3 -1
@@ -3203,7 +3203,7 @@ void emulate_update_regs(struct pt_regs *regs, struct instruction_op *op)default :WARN_ON_ONCE (1 ); } - regs->nip = next_pc; +regs_set_return_ip (regs, next_pc); }NOKPROBE_SYMBOL (emulate_update_regs);
@@ -3480,6 +3480,8 @@ int emulate_step(struct pt_regs *regs, struct ppc_inst instr)unsigned long val;unsigned long ea; +return_ip_or_msr_changed (); + r =analyse_instr (&op, regs, instr);if (r <0 )return r;
arch/powerpc/math-emu/math.c +1 -1
@@ -453,7 +453,7 @@ do_mathemu(struct pt_regs *regs)break ; } - regs->nip +=4 ; +regs_add_return_ip (regs,4 );return 0 ; illegal:
arch/powerpc/platforms/powernv/opal-call.c +4 -0
@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0 #include <linux/percpu.h> #include <linux/jump_label.h> +#include <asm/interrupt.h> #include <asm/opal-api.h> #include <asm/trace.h> #include <asm/asm-prototypes.h>
@@ -100,6 +101,9 @@ static int64_t opal_call(int64_t a0, int64_t a1, int64_t a2, int64_t a3,bool mmu = (msr & (MSR_IR |MSR_DR ));int64_t ret; +/* OPAL call / firmware may use SRR and/or HSRR */ +srr_regs_clobbered (); + msr &= ~MSR_EE ;if (unlikely (!mmu))
arch/powerpc/platforms/pseries/hvCall.S +29 -0
@@ -108,6 +108,10 @@ _GLOBAL_TOC(plpar_hcall_norets_notrace) mfcr r0 stw r0,8(r1) HVSC /* invoke the hypervisor */ + + li r4,0 + stb r4,PACASRR_VALID(r13) + lwz r0,8(r1) mtcrf 0xff,r0 blr /* return r3 = status */
@@ -120,6 +124,9 @@ _GLOBAL_TOC(plpar_hcall_norets) HCALL_BRANCH(plpar_hcall_norets_trace) HVSC /* invoke the hypervisor */ + li r4,0 + stb r4,PACASRR_VALID(r13) + lwz r0,8(r1) mtcrf 0xff,r0 blr /* return r3 = status */
@@ -129,6 +136,10 @@ plpar_hcall_norets_trace: HCALL_INST_PRECALL(R4) HVSC HCALL_INST_POSTCALL_NORETS + + li r4,0 + stb r4,PACASRR_VALID(r13) + lwz r0,8(r1) mtcrf 0xff,r0 blr
@@ -159,6 +170,9 @@ _GLOBAL_TOC(plpar_hcall) std r6, 16(r12) std r7, 24(r12) + li r4,0 + stb r4,PACASRR_VALID(r13) + lwz r0,8(r1) mtcrf 0xff,r0
@@ -188,6 +202,9 @@ plpar_hcall_trace: HCALL_INST_POSTCALL(r12) + li r4,0 + stb r4,PACASRR_VALID(r13) + lwz r0,8(r1) mtcrf 0xff,r0
@@ -223,6 +240,9 @@ _GLOBAL(plpar_hcall_raw) std r6, 16(r12) std r7, 24(r12) + li r4,0 + stb r4,PACASRR_VALID(r13) + lwz r0,8(r1) mtcrf 0xff,r0
@@ -262,6 +282,9 @@ _GLOBAL_TOC(plpar_hcall9) std r11,56(r12) std r0, 64(r12) + li r4,0 + stb r4,PACASRR_VALID(r13) + lwz r0,8(r1) mtcrf 0xff,r0
@@ -300,6 +323,9 @@ plpar_hcall9_trace: HCALL_INST_POSTCALL(r12) + li r4,0 + stb r4,PACASRR_VALID(r13) + lwz r0,8(r1) mtcrf 0xff,r0
@@ -339,6 +365,9 @@ _GLOBAL(plpar_hcall9_raw) std r11,56(r12) std r0, 64(r12) + li r4,0 + stb r4,PACASRR_VALID(r13) + lwz r0,8(r1) mtcrf 0xff,r0
arch/powerpc/sysdev/fsl_pci.c +1 -1
@@ -1072,7 +1072,7 @@ int fsl_pci_mcheck_exception(struct pt_regs *regs) ret =get_kernel_nofault (inst, (void *)regs->nip);if (!ret &&mcheck_handle_load (regs, inst)) { - regs->nip +=4 ; +regs_add_return_ip (regs,4 );return 1 ; } }
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 5/17 powerpc/64: handle MSR EE and RI in interrupt entry wrapper
In reply to Nicholas PigginSimilarly to the system call change in the previous patch, the mtmsrd to enable RI can be combined with the mtmsrd to enable EE for interrupts which enable the latter, which tends to be the important synchronous interrupts (i.e., page faults).
Do this by enabling EE and RI together at the beginning of the entry wrapper if PACA_IRQ_HARD_DIS is clear, and just enabling RI if it is set (which means something wanted EE=0).
Asynchronous interrupts set PACA_IRQ_HARD_DIS, but synchronous ones leave it unchanged, so by default they always get EE=1 unless they interrupt a caller that has hard disabled. When the sync interrupt later calls interrupt_cond_local_irq_enable(), that will not require another mtmsrd because we already enabled here.
This tends to save one mtmsrd L=1 for synchronous interrupts on 64s. 64e is conceptually unchanged, but it also sets MSR[EE]=1 now in the interrupt wrapper for synchronous interrupts with the same code.
From: Nicholas Piggin <[email protected]>
--- arch/powerpc/include/asm/interrupt.h | 22 ++++++++++++++++++-- arch/powerpc/kernel/exceptions-64s.S | 30 ---------------------------- 2 files changed, 20 insertions(+), 32 deletions(-)
arch/powerpc/include/asm/interrupt.h +20 -2
@@ -126,9 +126,21 @@ static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrup#endif #ifdef CONFIG_PPC64 -if (irq_soft_mask_set_return (IRQS_ALL_DISABLED ) ==IRQS_ENABLED ) +bool trace_enable = false; + +if (IS_ENABLED (CONFIG_TRACE_IRQFLAGS )) { +if (irq_soft_mask_set_return (IRQS_DISABLED ) ==IRQS_ENABLED ) + trace_enable = true; + }else { +irq_soft_mask_set (IRQS_DISABLED ); + } +/* If the interrupt was taken with HARD_DIS set, don't enable MSR[EE] */ +if (local_paca->irq_happened &PACA_IRQ_HARD_DIS ) +__hard_RI_enable (); +else +__hard_irq_enable (); +if (trace_enable)trace_hardirqs_off (); - local_paca->irq_happened |=PACA_IRQ_HARD_DIS ;if (user_mode (regs)) {CT_WARN_ON (ct_state () !=CONTEXT_USER );
@@ -175,6 +187,10 @@ static inline void interrupt_async_enter_prepare(struct pt_regs *regs, struct in__ppc64_runlatch_on ();#endif +#ifdef CONFIG_PPC64 +/* Ensure interrupt_enter_prepare does not enable MSR[EE] */ + local_paca->irq_happened |=PACA_IRQ_HARD_DIS ; +#endif interrupt_enter_prepare (regs, state);irq_enter (); }
@@ -239,6 +255,8 @@ static inline void interrupt_nmi_enter_prepare(struct pt_regs *regs, struct inte regs->softe =IRQS_ALL_DISABLED ; } +__hard_RI_enable (); +/* Don't do any per-CPU operations until interrupt state is fixed */ if (nmi_disables_ftrace (regs)) {
arch/powerpc/kernel/exceptions-64s.S +0 -30
@@ -129,7 +129,6 @@ name: #define IISIDE .L_IISIDE_\name\() /* Uses SRR0/1 not DAR/DSISR */ #define IDAR .L_IDAR_\name\() /* Uses DAR (or SRR0) */ #define IDSISR .L_IDSISR_\name\() /* Uses DSISR (or SRR1) */ -#define ISET_RI .L_ISET_RI_\name\() /* Run common code w/ MSR[RI]=1 */ #define IBRANCH_TO_COMMON .L_IBRANCH_TO_COMMON_\name\() /* ENTRY branch to common */ #define IREALMODE_COMMON .L_IREALMODE_COMMON_\name\() /* Common runs in realmode */ #define IMASK .L_IMASK_\name\() /* IRQ soft-mask bit */
@@ -174,9 +173,6 @@ do_define_int n .ifndef IDSISR IDSISR=0 .endif - .ifndef ISET_RI - ISET_RI=1 - .endif .ifndef IBRANCH_TO_COMMON IBRANCH_TO_COMMON=1 .endif
@@ -581,11 +577,6 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real) stb r10,PACASRR_VALID(r13) .endif - .if ISET_RI - li r10,MSR_RI - mtmsrd r10,1 /* Set MSR_RI */ - .endif - .if ISTACK .if IKUAP kuap_save_amr_and_lock r9, r10, cr1, cr0
@@ -906,11 +897,6 @@ INT_DEFINE_BEGIN(system_reset) IVEC=0x100 IAREA=PACA_EXNMI IVIRT=0 /* no virt entry point */ - /* - * MSR_RI is not enabled, because PACA_EXNMI and nmi stack is - * being used, so a nested NMI exception would corrupt it. - */ - ISET_RI=0 ISTACK=0 IKVM_REAL=1 INT_DEFINE_END(system_reset)
@@ -991,8 +977,6 @@ EXC_COMMON_BEGIN(system_reset_common) lhz r10,PACA_IN_NMI(r13) addi r10,r10,1 sth r10,PACA_IN_NMI(r13) - li r10,MSR_RI - mtmsrd r10,1 mr r10,r1 ld r1,PACA_NMI_EMERG_SP(r13)
@@ -1068,12 +1052,6 @@ INT_DEFINE_BEGIN(machine_check_early) IAREA=PACA_EXMC IVIRT=0 /* no virt entry point */ IREALMODE_COMMON=1 - /* - * MSR_RI is not enabled, because PACA_EXMC is being used, so a - * nested machine check corrupts it. machine_check_common enables - * MSR_RI. - */ - ISET_RI=0 ISTACK=0 IDAR=1 IDSISR=1
@@ -1084,7 +1062,6 @@ INT_DEFINE_BEGIN(machine_check) IVEC=0x200 IAREA=PACA_EXMC IVIRT=0 /* no virt entry point */ - ISET_RI=0 IDAR=1 IDSISR=1 IKVM_SKIP=1
@@ -1155,9 +1132,6 @@ EXC_COMMON_BEGIN(machine_check_early_common) BEGIN_FTR_SECTION bl enable_machine_check END_FTR_SECTION_IFSET(CPU_FTR_HVMODE) - li r10,MSR_RI - mtmsrd r10,1 - addi r3,r1,STACK_FRAME_OVERHEAD bl machine_check_early std r3,RESULT(r1) /* Save result */
@@ -1245,10 +1219,6 @@ EXC_COMMON_BEGIN(machine_check_common) * save area: PACA_EXMC instead of PACA_EXGEN. */ GEN_COMMON machine_check - - /* Enable MSR_RI when finished with PACA_EXMC */ - li r10,MSR_RI - mtmsrd r10,1 addi r3,r1,STACK_FRAME_OVERHEAD bl machine_check_exception b interrupt_return_srr
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 6/17 powerpc/64: move interrupt return asm to interrupt_64.S
In reply to Nicholas PigginThe next patch would like to move interrupt return assembly code to a low location before general text, so move it into its own file and include via head_64.S
--- arch/powerpc/include/asm/head-64.h | 2 +- arch/powerpc/kernel/entry_64.S | 623 ---------------------------- arch/powerpc/kernel/head_64.S | 5 +- arch/powerpc/kernel/interrupt_64.S | 635 +++++++++++++++++++++++++++++ 4 files changed, 640 insertions(+), 625 deletions(-) create mode 100644 arch/powerpc/kernel/interrupt_64.S
arch/powerpc/include/asm/head-64.h +1 -1
@@ -16,7 +16,7 @@ .section ".head.data.\name\()","a",@progbits .endm .macro use_ftsec name - .section ".head.text.\name\()" + .section ".head.text.\name\()","ax",@progbits .endm/*
arch/powerpc/kernel/entry_64.S +0 -623
642 lines not on this page · raw message
arch/powerpc/kernel/head_64.S +4 -1
@@ -194,8 +194,9 @@ CLOSE_FIXED_SECTION(first_256B) /* This value is used to mark exception frames on the stack. */ .section ".toc","aw" +/* This value is used to mark exception frames on the stack. */ exception_marker: - .tc ID_72656773_68657265[TC],0x7265677368657265 + .tc ID_EXC_MARKER[TC],STACK_FRAME_REGS_MARKER .previous /*
@@ -211,6 +212,8 @@ OPEN_TEXT_SECTION(0x100) USE_TEXT_SECTION() +#include "interrupt_64.S" + #ifdef CONFIG_PPC_BOOK3E /* * The booting_thread_hwid holds the thread id we want to boot in cpu
arch/powerpc/kernel/interrupt_64.S new file +635 -0
635 lines not on this page · raw message
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 7/17 powerpc/64s: system call avoid setting MSR[RI] until we set MSR[EE]
In reply to Nicholas PigginThis extends the MSR[RI]=0 window a little further into the system call in order to pair RI and EE enabling with a single mtmsrd.
--- arch/powerpc/kernel/exceptions-64s.S | 2 -- arch/powerpc/kernel/interrupt_64.S | 4 ++-- 2 files changed, 2 insertions(+), 4 deletions(-)
arch/powerpc/kernel/exceptions-64s.S +0 -2
@@ -1942,8 +1942,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_REAL_LE) mtctr r10 bctr .else - li r10,MSR_RI - mtmsrd r10,1 /* Set RI (EE=0) */ #ifdef CONFIG_RELOCATABLE __LOAD_HANDLER(r10, system_call_common) mtctr r10
arch/powerpc/kernel/interrupt_64.S +2 -2
@@ -283,9 +283,9 @@ END_BTB_FLUSH_SECTION * trace_hardirqs_off(). */ li r11,IRQS_ALL_DISABLED - li r12,PACA_IRQ_HARD_DIS + li r12,-1 /* Set MSR_EE and MSR_RI */ stb r11,PACAIRQSOFTMASK(r13) - stb r12,PACAIRQHAPPENED(r13) + mtmsrd r12,1 /* Calling convention has r9 = orig r0, r10 = regs */ mr r9,r0
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 8/17 powerpc/64s: save one more register in the masked interrupt handler
In reply to Nicholas PigginThis frees up one more register (and takes advantage of that to clean things up a little bit).
This register will be used in the following patch.
--- arch/powerpc/kernel/exceptions-64s.S | 34 ++++++++++++++++------------ 1 file changed, 20 insertions(+), 14 deletions(-)
arch/powerpc/kernel/exceptions-64s.S +20 -14
@@ -2758,7 +2758,6 @@ INT_DEFINE_END(soft_nmi) * and run it entirely with interrupts hard disabled. */ EXC_COMMON_BEGIN(soft_nmi_common) - mfspr r11,SPRN_SRR0 mr r10,r1 ld r1,PACAEMERGSP(r13) subi r1,r1,INT_FRAME_SIZE
@@ -2793,19 +2792,24 @@ masked_Hinterrupt: .else masked_interrupt: .endif - lbz r11,PACAIRQHAPPENED(r13) - or r11,r11,r10 - stb r11,PACAIRQHAPPENED(r13) + stw r9,PACA_EXGEN+EX_CCR(r13) + lbz r9,PACAIRQHAPPENED(r13) + or r9,r9,r10 + stb r9,PACAIRQHAPPENED(r13) + + .if ! \hsrr cmpwi r10,PACA_IRQ_DEC bne 1f - lis r10,0x7fff - ori r10,r10,0xffff - mtspr SPRN_DEC,r10 + LOAD_REG_IMMEDIATE(r9, 0x7fffffff) + mtspr SPRN_DEC,r9 #ifdef CONFIG_PPC_WATCHDOG + lwz r9,PACA_EXGEN+EX_CCR(r13) b soft_nmi_common #else b 2f #endif + .endif + 1: andi. r10,r10,PACA_IRQ_MUST_HARD_MASK beq 2f xori r12,r12,MSR_EE /* clear MSR_EE */
@@ -2814,17 +2818,19 @@ masked_interrupt: .else mtspr SPRN_SRR1,r12 .endif - ori r11,r11,PACA_IRQ_HARD_DIS - stb r11,PACAIRQHAPPENED(r13) + ori r9,r9,PACA_IRQ_HARD_DIS + stb r9,PACAIRQHAPPENED(r13) 2: /* done */ - li r10,0 + li r9,0 .if \hsrr - stb r10,PACAHSRR_VALID(r13) + stb r9,PACAHSRR_VALID(r13) .else - stb r10,PACASRR_VALID(r13) + stb r9,PACASRR_VALID(r13) .endif - ld r10,PACA_EXGEN+EX_CTR(r13) - mtctr r10 + + ld r9,PACA_EXGEN+EX_CTR(r13) + mtctr r9 + lwz r9,PACA_EXGEN+EX_CCR(r13) mtcrf 0x80,r9 std r1,PACAR1(r13) ld r9,PACA_EXGEN+EX_R9(r13)
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 9/17 powerpc/64: allow alternate return locations for soft-masked interrupts
In reply to Nicholas PigginThe exception table fixup adjusts a failed page fault's interrupt return location if it was taken at an address specified in the exception table, to a corresponding fixup handler address.
Introduce a variation of that idea which adds a fixup table for NMIs and soft-masked asynchronous interrupts. This will be used to protect certain critical sections that are sensitive to being clobbered by interrupts coming in (due to using the same SPRs and/or irq soft-mask state).
--- arch/powerpc/include/asm/interrupt.h | 13 +++++++++ arch/powerpc/include/asm/ppc_asm.h | 8 ++++++ arch/powerpc/kernel/exceptions-64e.S | 37 +++++++++++++++++++++++-- arch/powerpc/kernel/exceptions-64s.S | 41 ++++++++++++++++++++++++++++ arch/powerpc/kernel/vmlinux.lds.S | 10 +++++++ arch/powerpc/lib/Makefile | 2 +- arch/powerpc/lib/restart_table.c | 30 ++++++++++++++++++++ 7 files changed, 138 insertions(+), 3 deletions(-) create mode 100644 arch/powerpc/lib/restart_table.c
arch/powerpc/include/asm/interrupt.h +13 -0
@@ -73,6 +73,11 @@#include <asm/kprobes.h> #include <asm/runlatch.h> +#ifdef CONFIG_PPC64 +extern char __end_soft_masked[]; +unsigned long search_kernel_restart_table (unsigned long addr); +#endif +#ifdef CONFIG_PPC_BOOK3S_64 static inline void srr_regs_clobbered (void ) {
@@ -287,6 +292,14 @@ static inline void interrupt_nmi_exit_prepare(struct pt_regs *regs, struct inter* new work to do (must use irq_work for that). */ +#ifdef CONFIG_PPC64 +if (arch_irq_disabled_regs (regs)) { +unsigned long rst =search_kernel_restart_table (regs->nip); +if (rst) +regs_set_return_ip (regs, rst); + } +#endif +#ifdef CONFIG_PPC64 if (nmi_disables_ftrace (regs))this_cpu_set_ftrace_enabled (state->ftrace_enabled);
arch/powerpc/include/asm/ppc_asm.h +8 -0
@@ -762,6 +762,14 @@ END_FTR_SECTION_NESTED(CPU_FTR_CELL_TB_BUG, CPU_FTR_CELL_TB_BUG, 96) stringify_in_c(.long (_target) - . ;) \ stringify_in_c(.previous) +#define RESTART_TABLE(_start, _end, _target) \ + stringify_in_c(.section __restart_table,"a";)\ + stringify_in_c(.balign 8;) \ + stringify_in_c(.llong (_start);) \ + stringify_in_c(.llong (_end);) \ + stringify_in_c(.llong (_target);) \ + stringify_in_c(.previous) +#ifdef CONFIG_PPC_FSL_BOOK3E #define BTB_FLUSH (reg) \ lis reg,BUCSR_INIT@h; \
arch/powerpc/kernel/exceptions-64e.S +35 -2
@@ -901,6 +901,28 @@ kernel_dbg_exc: bl unknown_exception b interrupt_return +.macro SEARCH_RESTART_TABLE + LOAD_REG_IMMEDIATE_SYM(r14, r11, __start___restart_table) + LOAD_REG_IMMEDIATE_SYM(r15, r11, __stop___restart_table) +300: + cmpd r14,r15 + beq 302f + ld r11,0(r14) + cmpld r10,r11 + blt 301f + ld r11,8(r14) + cmpld r10,r11 + bge 301f + ld r11,16(r14) + b 303f +301: + addi r14,r14,24 + b 300b +302: + li r11,0 +303: +.endm + /* * An interrupt came in while soft-disabled; We mark paca->irq_happened * accordingly and if the interrupt is level sensitive, we hard disable
@@ -909,6 +931,9 @@ kernel_dbg_exc: */ .macro masked_interrupt_book3e paca_irq full_mask + std r14,PACA_EXGEN+EX_R14(r13) + std r15,PACA_EXGEN+EX_R15(r13) + lbz r10,PACAIRQHAPPENED(r13) .if \full_mask == 1 ori r10,r10,\paca_irq | PACA_IRQ_HARD_DIS
@@ -918,15 +943,23 @@ kernel_dbg_exc: stb r10,PACAIRQHAPPENED(r13) .if \full_mask == 1 - rldicl r10,r11,48,1 /* clear MSR_EE */ - rotldi r11,r10,16 + xori r11,r11,MSR_EE /* clear MSR_EE */ mtspr SPRN_SRR1,r11 .endif + mfspr r10,SPRN_SRR0 + SEARCH_RESTART_TABLE + cmpdi r11,0 + beq 1f + mtspr SPRN_SRR0,r11 /* return to restart address */ +1: + lwz r11,PACA_EXGEN+EX_CR(r13) mtcr r11 ld r10,PACA_EXGEN+EX_R10(r13) ld r11,PACA_EXGEN+EX_R11(r13) + ld r14,PACA_EXGEN+EX_R14(r13) + ld r15,PACA_EXGEN+EX_R15(r13) mfspr r13,SPRN_SPRG_GEN_SCRATCH rfi b .
arch/powerpc/kernel/exceptions-64s.S +41 -0
59 lines not on this page · raw message
arch/powerpc/kernel/vmlinux.lds.S +10 -0
@@ -9,6 +9,14 @@ #define EMITS_PT_NOTE #define RO_EXCEPTION_TABLE_ALIGN 0 +#define RESTART_TABLE(align) \ + . = ALIGN(align); \ + __restart_table : AT(ADDR(__restart_table) - LOAD_OFFSET) { \ + __start___restart_table = .; \ + KEEP(*(__restart_table)) \ + __stop___restart_table = .; \ + } + #include <asm/page.h> #include <asm-generic/vmlinux.lds.h> #include <asm/cache.h>
@@ -124,6 +132,8 @@ SECTIONS RO_DATA(PAGE_SIZE) #ifdef CONFIG_PPC64 + RESTART_TABLE(8) + . = ALIGN(8); __stf_entry_barrier_fixup : AT(ADDR(__stf_entry_barrier_fixup) - LOAD_OFFSET) { __start___stf_entry_barrier_fixup = .;
arch/powerpc/lib/Makefile +1 -1
@@ -42,7 +42,7 @@ obj-$(CONFIG_PPC_BOOK3S_64) += copyuser_power7.o copypage_power7.o \ memcpy_power7.o obj64-y += copypage_64.o copyuser_64.o mem_64.o hweight_64.o \ - memcpy_64.o copy_mc_64.o + memcpy_64.o copy_mc_64.o restart_table.o ifndef CONFIG_PPC_QUEUED_SPINLOCKS obj64-$(CONFIG_SMP) += locks.o
arch/powerpc/lib/restart_table.c new file +30 -0
@@ -0,0 +1,30 @@ +#include <asm/interrupt.h> +#include <asm/kprobes.h> + +struct restart_table_entry { +unsigned long start; +unsigned long end; +unsigned long fixup; +}; + +extern struct restart_table_entry __start___restart_table[]; +extern struct restart_table_entry __stop___restart_table[]; + +/* Given an address, look for it in the kernel exception table */ +unsigned long search_kernel_restart_table (unsigned long addr) +{ +struct restart_table_entry *rte = __start___restart_table; + +while (rte < __stop___restart_table) { +unsigned long start = rte->start; +unsigned long end = rte->end; +unsigned long fixup = rte->fixup; + +if (addr >= start && addr < end) +return fixup; + + rte++; + } +return 0 ; +} +NOKPROBE_SYMBOL (search_kernel_restart_table);
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 10/17 powerpc/64: interrupt soft-enable race fix
In reply to Nicholas PigginPrevent interrupt restore from allowing racing hard interrupts going ahead of previous soft-pending ones, by using the soft-masked restart handler to allow a store to clear the soft-mask while knowing nothing is soft-pending.
This probably doesn't matter much in practice, but it's a simple demonstrator / test case to exercise the restart table logic.
--- arch/powerpc/kernel/irq.c | 95 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+)
arch/powerpc/kernel/irq.c +95 -0
@@ -217,6 +217,100 @@ static inline void replay_soft_interrupts_irqrestore(void)#define replay_soft_interrupts_irqrestore () replay_soft_interrupts()#endif +#ifdef CONFIG_CC_HAS_ASM_GOTO +notrace void arch_local_irq_restore(unsigned long mask) +{ +unsigned char irq_happened; + +/* Write the new soft-enabled value if it is a disable */ +if (mask) { +irq_soft_mask_set (mask); +return ; + } + +/* +* After the stb, interrupts are unmasked and there are no interrupts +* pending replay. The restart sequence makes this atomic with +* respect to soft-masked interrupts. If this was just a simple code +* sequence, a soft-masked interrupt could become pending right after +* the comparison and before the stb. +* +* This allows interrupts to be unmasked without hard disabling, and +* also without new hard interrupts coming in ahead of pending ones. +*/ +asm_volatile_goto ( +"1: \n" +" lbz 9,%0(13) \n" +" cmpwi 9,0 \n" +" bne %l[happened] \n" +" stb 9,%1(13) \n" +"2: \n" +RESTART_TABLE (1b ,2b ,1b ) + : : "i" (offsetof(struct paca_struct, irq_happened)), + "i" (offsetof(struct paca_struct, irq_soft_mask)) + : "cr0", "r9" + : happened); + +if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) +WARN_ON_ONCE (!(mfmsr () &MSR_EE )); + +return ; + +happened: + irq_happened =get_irq_happened (); +if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) +WARN_ON_ONCE (!irq_happened); + +if (irq_happened ==PACA_IRQ_HARD_DIS ) { +if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) +WARN_ON_ONCE (mfmsr () &MSR_EE ); +irq_soft_mask_set (IRQS_ENABLED ); + local_paca->irq_happened =0 ; +__hard_irq_enable (); +return ; + } + +/* Have interrupts to replay, need to hard disable first */ +if (!(irq_happened &PACA_IRQ_HARD_DIS )) { +if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) { +if (!(mfmsr () &MSR_EE )) { +/* +* An interrupt could have come in and cleared +* MSR[EE] and set IRQ_HARD_DIS, so check +* IRQ_HARD_DIS again and warn if it is still +* clear. +*/ + irq_happened =get_irq_happened (); +WARN_ON_ONCE (!(irq_happened &PACA_IRQ_HARD_DIS )); + } + } +__hard_irq_disable (); + local_paca->irq_happened |=PACA_IRQ_HARD_DIS ; + }else { +if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) { +if (WARN_ON_ONCE (mfmsr () &MSR_EE )) +__hard_irq_disable (); + } + } + +/* +* Disable preempt here, so that the below preempt_enable will +* perform resched if required (a replayed interrupt may set +* need_resched). +*/ +preempt_disable (); +irq_soft_mask_set (IRQS_ALL_DISABLED ); +trace_hardirqs_off (); + +replay_soft_interrupts_irqrestore (); + local_paca->irq_happened =0 ; + +trace_hardirqs_on (); +irq_soft_mask_set (IRQS_ENABLED ); +__hard_irq_enable (); +preempt_enable (); +} +#else notrace void arch_local_irq_restore(unsigned long mask) {unsigned char irq_happened;
@@ -288,6 +382,7 @@ notrace void arch_local_irq_restore(unsigned long mask)__hard_irq_enable (); preempt_enable(); } +#endif EXPORT_SYMBOL (arch_local_irq_restore);/*
Signature · 1 line
2.23.0
Re: [PATCH v4 10/17] powerpc/64: interrupt soft-enable race fix
Hi Nick.
Le 17/06/2021 à 17:51, Nicholas Piggin a écrit :
Quoted 18 lines
Prevent interrupt restore from allowing racing hard interrupts going ahead of previous soft-pending ones, by using the soft-masked restart handler to allow a store to clear the soft-mask while knowing nothing is soft-pending.
This probably doesn't matter much in practice, but it's a simple demonstrator / test case to exercise the restart table logic.
Signed-off-by Nicholas Piggin <[email protected]>--- arch/powerpc/kernel/irq.c | 95 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+)arch/powerpc/kernel/irq.c +0
-0
arch/powerpc/kernel/irq.c static inline void replay_soft_interrupts_irqrestore(void) · line 220
Quoted 4 lines
arch/powerpc/kernel/irq.c +1
-0@@ -217,6 +217,100 @@ static inline void replay_soft_interrupts_irqrestore(void)#define replay_soft_interrupts_irqrestore () replay_soft_interrupts()#endif +#ifdef CONFIG_CC_HAS_ASM_GOTO
Wondering why you added that #ifdef CONFIG_CC_HAS_ASM_GOTO ? In 2021 the minimum GCC version was already supporting asm goto and we were already using asm goto unconditionnaly in uaccess.
I just sent a patch to remove it.
Quoted 104 lines
arch/powerpc/kernel/irq.c +94
-0+notrace void arch_local_irq_restore(unsigned long mask) +{ +unsigned char irq_happened; + +/* Write the new soft-enabled value if it is a disable */ +if (mask) { +irq_soft_mask_set (mask); +return ; + } + +/* +* After the stb, interrupts are unmasked and there are no interrupts +* pending replay. The restart sequence makes this atomic with +* respect to soft-masked interrupts. If this was just a simple code +* sequence, a soft-masked interrupt could become pending right after +* the comparison and before the stb. +* +* This allows interrupts to be unmasked without hard disabling, and +* also without new hard interrupts coming in ahead of pending ones. +*/ +asm_volatile_goto ( +"1: \n" +" lbz 9,%0(13) \n" +" cmpwi 9,0 \n" +" bne %l[happened] \n" +" stb 9,%1(13) \n" +"2: \n" +RESTART_TABLE (1b ,2b ,1b ) + : : "i" (offsetof(struct paca_struct, irq_happened)), + "i" (offsetof(struct paca_struct, irq_soft_mask)) + : "cr0", "r9" + : happened); + +if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) +WARN_ON_ONCE (!(mfmsr () &MSR_EE )); + +return ; + +happened: + irq_happened =get_irq_happened (); +if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) +WARN_ON_ONCE (!irq_happened); + +if (irq_happened ==PACA_IRQ_HARD_DIS ) { +if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) +WARN_ON_ONCE (mfmsr () &MSR_EE ); +irq_soft_mask_set (IRQS_ENABLED ); + local_paca->irq_happened =0 ; +__hard_irq_enable (); +return ; + } + +/* Have interrupts to replay, need to hard disable first */ +if (!(irq_happened &PACA_IRQ_HARD_DIS )) { +if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) { +if (!(mfmsr () &MSR_EE )) { +/* +* An interrupt could have come in and cleared +* MSR[EE] and set IRQ_HARD_DIS, so check +* IRQ_HARD_DIS again and warn if it is still +* clear. +*/ + irq_happened =get_irq_happened (); +WARN_ON_ONCE (!(irq_happened &PACA_IRQ_HARD_DIS )); + } + } +__hard_irq_disable (); + local_paca->irq_happened |=PACA_IRQ_HARD_DIS ; + }else { +if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) { +if (WARN_ON_ONCE (mfmsr () &MSR_EE )) +__hard_irq_disable (); + } + } + +/* +* Disable preempt here, so that the below preempt_enable will +* perform resched if required (a replayed interrupt may set +* need_resched). +*/ +preempt_disable (); +irq_soft_mask_set (IRQS_ALL_DISABLED ); +trace_hardirqs_off (); + +replay_soft_interrupts_irqrestore (); + local_paca->irq_happened =0 ; + +trace_hardirqs_on (); +irq_soft_mask_set (IRQS_ENABLED ); +__hard_irq_enable (); +preempt_enable (); +} +#else notrace void arch_local_irq_restore(unsigned long mask) {unsigned char irq_happened;@@ -288,6 +382,7 @@ notrace void arch_local_irq_restore(unsigned long mask)__hard_irq_enable (); preempt_enable(); } +#endif EXPORT_SYMBOL (arch_local_irq_restore);/*
Nicholas Piggin
reviewerPATCH v4 11/17 powerpc/64: treat low kernel text as irqs soft-masked
In reply to Nicholas PigginTreat code below __end_soft_masked as soft-masked for the purpose of alternate return. 64s already mostly does this for scv entry.
This will be used to exit from interrupts without disabling MSR[EE].
--- arch/powerpc/include/asm/interrupt.h | 11 ++++++++--- arch/powerpc/kernel/exceptions-64e.S | 12 +++++++++++- arch/powerpc/kernel/exceptions-64s.S | 19 +++++++++++-------- arch/powerpc/kernel/interrupt_64.S | 6 +++++- 4 files changed, 35 insertions(+), 13 deletions(-)
arch/powerpc/include/asm/interrupt.h +8 -3
@@ -158,8 +158,13 @@ static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrup* CT_WARN_ON comes here via program_check_exception, * so avoid recursion. */ -if (TRAP (regs) !=INTERRUPT_PROGRAM ) +if (TRAP (regs) !=INTERRUPT_PROGRAM ) {CT_WARN_ON (ct_state () !=CONTEXT_KERNEL ); +BUG_ON (regs->nip < (unsigned long )__end_soft_masked); + } +/* Move this under a debugging check */ +if (arch_irq_disabled_regs (regs)) + BUG_ON(search_kernel_restart_table(regs->nip)); }#endif
@@ -254,8 +259,8 @@ static inline void interrupt_nmi_enter_prepare(struct pt_regs *regs, struct inte local_paca->irq_happened |=PACA_IRQ_HARD_DIS ;if (IS_ENABLED (CONFIG_PPC_BOOK3S_64 ) && !(regs->msr &MSR_PR ) && - regs->nip < (unsigned long )__end_interrupts) { -// Kernel code running below __end_interrupts is + regs->nip < (unsigned long )__end_soft_masked) { +// Kernel code running below __end_soft_masked is // implicitly soft-masked. regs->softe =IRQS_ALL_DISABLED ; }
arch/powerpc/kernel/exceptions-64e.S +11 -1
@@ -342,7 +342,17 @@ ret_from_mc_except: #define PROLOG_ADDITION_MASKABLE_GEN(n) \ lbz r10,PACAIRQSOFTMASK(r13); /* are irqs soft-masked? */ \ andi. r10,r10,IRQS_DISABLED; /* yes -> go out of line */ \ - bne masked_interrupt_book3e_##n + bne masked_interrupt_book3e_##n; \ + /* Kernel code below __end_soft_masked is implicitly masked */ \ + andi. r10,r11,MSR_PR; \ + bne 1f; /* user -> not masked */ \ + std r14,PACA_EXGEN+EX_R14(r13); \ + LOAD_REG_IMMEDIATE_SYM(r14, r10, __end_soft_masked); \ + mfspr r10,SPRN_SRR0; \ + cmpld r10,r14; \ + ld r14,PACA_EXGEN+EX_R14(r13); \ + blt masked_interrupt_book3e_##n; \ +1: /* * Additional regs must be re-loaded from paca before EXCEPTION_COMMON* is
arch/powerpc/kernel/exceptions-64s.S +11 -8
@@ -508,10 +508,13 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real) andi. r10,r12,MSR_PR bne 2f - /* Kernel code running below __end_interrupts is implicitly - * soft-masked */ - LOAD_HANDLER(r10, __end_interrupts) + /* + * Kernel code running below __end_soft_masked is implicitly + * soft-masked + */ + LOAD_HANDLER(r10, __end_soft_masked) cmpld r11,r10 + li r10,IMASK blt- 1f
@@ -824,17 +827,17 @@ __start_interrupts: * scv instructions enter the kernel without changing EE, RI, ME, or HV. * In particular, this means we can take a maskable interrupt at any point * in the scv handler, which is unlike any other interrupt. This is solved - * by treating the instruction addresses below __end_interrupts as being + * by treating the instruction addresses below __end_soft_masked as being * soft-masked. * * AIL-0 mode scv exceptions go to 0x17000-0x17fff, but we set AIL-3 and * ensure scv is never executed with relocation off, which means AIL-0 * should never happen. * - * Before leaving the below __end_interrupts text, at least of the following - * must be true: + * Before leaving the following inside-__end_soft_masked text, at least of the + * following must be true: * - MSR[PR]=1 (i.e., return to userspace) - * - MSR_EE|MSR_RI is set (no reentrant exceptions) + * - MSR_EE|MSR_RI is clear (no reentrant exceptions) * - Standard kernel environment is set up (stack, paca, etc) * * Call convention:
@@ -3099,7 +3102,7 @@ kvmppc_skip_Hinterrupt: USE_FIXED_SECTION(virt_trampolines) /* - * All code below __end_interrupts is treated as soft-masked. If + * All code below __end_soft_masked is treated as soft-masked. If * any code runs here with MSR[EE]=1, it must then cope with pending * soft interrupt being raised (i.e., by ensuring it is replayed). *
arch/powerpc/kernel/interrupt_64.S +5 -1
@@ -632,4 +632,8 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_STCX_CHECKS_ADDRESS) interrupt_return_macro srr #ifdef CONFIG_PPC_BOOK3S interrupt_return_macro hsrr -#endif +#endif /* CONFIG_PPC_BOOK3S */ + + .globl __end_soft_masked +__end_soft_masked: +DEFINE_FIXED_SYMBOL(__end_soft_masked)
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 12/17 powerpc/64: use interrupt restart table to speed up return from interrupt
In reply to Nicholas PigginUse the restart table facility to return from interrupt or system calls without disabling MSR[EE] or MSR[RI].
Interrupt return asm is put into the low soft-masked region, to prevent interrupts being processed here, although they are still taken as masked interrupts which causes SRRs to be clobbered, and a pending soft-masked interrupt to require replaying.
The return code uses restart table regions to redirct to a fixup handler rather than continue with the exit, if such an interrupt happens. In this case the interrupt return is redirected to a fixup handler which reloads r1 for the interrupt stack and reloads registers and sets state up to replay the soft-masked interrupt and try the exit again.
Some types of security exit fallback flushes and barriers are currently unable to cope with reentrant interrupts, e.g., because they store some state in the scratch SPR which would be clobbered even by masked interrupts. For now the interrupts-enabled exits are disabled when these flushes are used.
--- arch/powerpc/include/asm/asm-prototypes.h | 5 + arch/powerpc/include/asm/hw_irq.h | 13 +- arch/powerpc/include/asm/interrupt.h | 2 + arch/powerpc/include/asm/paca.h | 3 + arch/powerpc/include/asm/ptrace.h | 1 + arch/powerpc/kernel/asm-offsets.c | 3 + arch/powerpc/kernel/interrupt.c | 393 ++++++++++++++-------- arch/powerpc/kernel/interrupt_64.S | 143 ++++++-- arch/powerpc/lib/feature-fixups.c | 52 ++- 9 files changed, 457 insertions(+), 158 deletions(-)
arch/powerpc/include/asm/asm-prototypes.h +5 -0
@@ -73,6 +73,11 @@ long system_call_exception(long r3, long r4, long r5, long r6, long r7, long r8,notrace unsigned long syscall_exit_prepare (unsigned long r3,struct pt_regs *regs,long scv);notrace unsigned long interrupt_exit_user_prepare (struct pt_regs *regs);notrace unsigned long interrupt_exit_kernel_prepare (struct pt_regs *regs); +#ifdef CONFIG_PPC64 +unsigned long syscall_exit_restart (unsigned long r3,struct pt_regs *regs); +unsigned long interrupt_exit_user_restart (struct pt_regs *regs); +unsigned long interrupt_exit_kernel_restart (struct pt_regs *regs); +#endif long ppc_fadvise64_64 (int fd,int advice,u32 offset_high,u32 offset_low,u32 len_high,u32 len_low);
arch/powerpc/include/asm/hw_irq.h +11 -2
@@ -18,8 +18,17 @@* PACA flags in paca->irq_happened. * * This bits are set when interrupts occur while soft-disabled -* and allow a proper replay. Additionally, PACA_IRQ_HARD_DIS -* is set whenever we manually hard disable. +* and allow a proper replay. +* +* The PACA_IRQ_HARD_DIS is set whenever we hard disable. It is almost +* always in synch with the MSR[EE] state, except: +* - A window in interrupt entry, where hardware disables MSR[EE] and that +* must be "reconciled" with the soft mask state. +* - NMI interrupts that hit in awkward places, until they fix the state. +* - When local irqs are being enabled and state is being fixed up. +* - When returning from an interrupt there are some windows where this +* can become out of synch, but gets fixed before the RFI or before +* executing the next user instruction (see arch/powerpc/kernel/interrupt.c). */ #define PACA_IRQ_HARD_DIS 0x01#define PACA_IRQ_DBELL 0x02
arch/powerpc/include/asm/interrupt.h +2 -0
@@ -79,6 +79,8 @@ unsigned long search_kernel_restart_table(unsigned long addr);#endif #ifdef CONFIG_PPC_BOOK3S_64 +DECLARE_STATIC_KEY_FALSE (interrupt_exit_not_reentrant); +static inline void srr_regs_clobbered (void ) { local_paca->srr_valid =0 ;
arch/powerpc/include/asm/paca.h +3 -0
@@ -167,6 +167,9 @@ struct paca_struct {u64 kstack;/* Saved Kernel stack addr */ u64 saved_r1;/* r1 save for RTAS calls or PM or EE=0 */ u64 saved_msr;/* MSR saved here by enter_rtas */ +#ifdef CONFIG_PPC64 +u64 exit_save_r1;/* Syscall/interrupt R1 save */ +#endif #ifdef CONFIG_PPC_BOOK3E u16 trap_save;/* Used when bad stack is encountered */ #endif
arch/powerpc/include/asm/ptrace.h +1 -0
@@ -52,6 +52,7 @@ struct pt_regsstruct {#ifdef CONFIG_PPC64 unsigned long ppr; +unsigned long exit_result;#endif union {#ifdef CONFIG_PPC_KUAP
arch/powerpc/kernel/asm-offsets.c +3 -0
@@ -286,6 +286,9 @@ int main(void)OFFSET (PACAHWCPUID , paca_struct, hw_cpu_id);OFFSET (PACAKEXECSTATE , paca_struct, kexec_state);OFFSET (PACA_DSCR_DEFAULT , paca_struct, dscr_default); +#ifdef CONFIG_PPC64 +OFFSET (PACA_EXIT_SAVE_R1 , paca_struct, exit_save_r1); +#endif #ifdef CONFIG_PPC_BOOK3E OFFSET (PACA_TRAP_SAVE , paca_struct, trap_save);#endif
arch/powerpc/kernel/interrupt.c +261 -132
465 lines not on this page · raw message
arch/powerpc/kernel/interrupt_64.S +123 -20
235 lines not on this page · raw message
arch/powerpc/lib/feature-fixups.c +48 -4
85 lines not on this page · raw message
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 13/17 powerpc/interrupt: Rename and lightly change syscall_exit_prepare_main()
In reply to Nicholas PigginFrom: Christophe Leroy <[email protected]>
Rename syscall_exit_prepare_main() into interrupt_exit_prepare_main()
Pass it the 'ret' so that it can 'or' it directly instead of oring twice, once inside the function and once outside.
And remove 'r3' parameter which is not used.
--- arch/powerpc/kernel/interrupt.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
arch/powerpc/kernel/interrupt.c +4 -5
@@ -243,11 +243,10 @@ static notrace void booke_load_dbcr0(void)#endif } -static notrace unsigned long syscall_exit_prepare_main (unsigned long r3, -struct pt_regs *regs) +static notrace unsigned long +interrupt_exit_user_prepare_main (unsigned long ret,struct pt_regs *regs) {unsigned long ti_flags; -unsigned long ret =0 ; again: ti_flags =READ_ONCE (current_thread_info ()->flags);
@@ -365,7 +364,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3, }local_irq_disable (); - ret |=syscall_exit_prepare_main (r3, regs); + ret =interrupt_exit_user_prepare_main (ret, regs);#ifdef CONFIG_PPC64 regs->exit_result = ret;
@@ -397,7 +396,7 @@ notrace unsigned long syscall_exit_restart(unsigned long r3, struct pt_regs *regBUG_ON (!user_mode (regs)); - regs->exit_result |=syscall_exit_prepare_main (r3, regs); + regs->exit_result =interrupt_exit_user_prepare_main (regs->exit_result, regs);return regs->exit_result; }
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 14/17 powerpc/interrupt: Refactor interrupt_exit_user_prepare()
In reply to Nicholas PigginFrom: Christophe Leroy <[email protected]>
interrupt_exit_user_prepare() is a superset ofinterrupt_exit_user_prepare_main().
Refactor to avoid code duplication.
--- arch/powerpc/kernel/interrupt.c | 57 ++------------------------------- 1 file changed, 3 insertions(+), 54 deletions(-)
arch/powerpc/kernel/interrupt.c +3 -54
@@ -404,9 +404,7 @@ notrace unsigned long syscall_exit_restart(unsigned long r3, struct pt_regs *regnotrace unsigned long interrupt_exit_user_prepare (struct pt_regs *regs) { -unsigned long ti_flags; -unsigned long flags; -unsigned long ret =0 ; +unsigned long ret;if (!IS_ENABLED (CONFIG_BOOKE ) && !IS_ENABLED (CONFIG_40x))BUG_ON (!(regs->msr &MSR_RI ));
@@ -420,63 +418,14 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs)*/ kuap_assert_locked (); -local_irq_save (flags); - -again: - ti_flags =READ_ONCE (current_thread_info ()->flags); -while (unlikely (ti_flags & (_TIF_USER_WORK_MASK & ~_TIF_RESTORE_TM))) { -local_irq_enable ();/* returning to user: may enable */ -if (ti_flags & _TIF_NEED_RESCHED) { -schedule (); - }else { -if (ti_flags & _TIF_SIGPENDING) - ret |= _TIF_RESTOREALL; -do_notify_resume (regs, ti_flags); - } -local_irq_disable (); - ti_flags =READ_ONCE (current_thread_info ()->flags); - } - -if (IS_ENABLED (CONFIG_PPC_BOOK3S_64 ) &&IS_ENABLED (CONFIG_PPC_FPU )) { -if (IS_ENABLED (CONFIG_PPC_TRANSACTIONAL_MEM ) && -unlikely ((ti_flags & _TIF_RESTORE_TM))) { -restore_tm_state (regs); - }else { -unsigned long mathflags =MSR_FP ; - -if (cpu_has_feature (CPU_FTR_VSX )) - mathflags |=MSR_VEC |MSR_VSX ; -else if (cpu_has_feature (CPU_FTR_ALTIVEC )) - mathflags |=MSR_VEC ; - -/* See above restore_math comment */ -if ((regs->msr & mathflags) != mathflags) -restore_math (regs); - } - } - -if (!prep_irq_for_user_exit ()) { -local_irq_enable (); -local_irq_disable (); - goto again; - } - -booke_load_dbcr0 (); - -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM - local_paca->tm_scratch = regs->msr; -#endif +local_irq_disable (); -account_cpu_user_exit (); + ret =interrupt_exit_user_prepare_main (0 , regs);#ifdef CONFIG_PPC64 regs->exit_result = ret;#endif -/* Restore user access locks last */ -kuap_user_restore (regs); -kuep_unlock (); -return ret; }
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 15/17 powerpc/interrupt: Interchange prep_irq_for_{kernel_enabled/user}_exit()
In reply to Nicholas PigginFrom: Christophe Leroy <[email protected]>
prep_irq_for_user_exit() is a superset of prep_irq_for_kernel_enabled_exit(). In order to allow refactoring in following patch, interchange the two. This will allow prep_irq_for_user_exit() to call a renamed version of prep_irq_for_kernel_enabled_exit().
--- arch/powerpc/kernel/interrupt.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-)
arch/powerpc/kernel/interrupt.c +11 -12
@@ -46,27 +46,28 @@ static inline bool exit_must_hard_disable(void)* This should be called with local irqs disabled, but if they were previously * enabled when the interrupt handler returns (indicating a process-context / * synchronous interrupt) then irqs_enabled should be true. +* +* restartable is true then EE/RI can be left on because interrupts are handled +* with a restart sequence. */ -static notrace __always_inline bool prep_irq_for_user_exit(void) +static notrace __always_inline bool prep_irq_for_kernel_enabled_exit(bool restartable) { -user_enter_irqoff ();/* This must be done with RI=1 because tracing may touch vmaps */ trace_hardirqs_on ();#ifdef CONFIG_PPC32 __hard_EE_RI_disable ();#else -if (exit_must_hard_disable ()) +if (exit_must_hard_disable () || !restartable)__hard_EE_RI_disable ();/* This pattern matches prep_irq_for_idle */ if (unlikely (lazy_irq_pending_nocheck ())) { -if (exit_must_hard_disable ()) { +if (exit_must_hard_disable () || !restartable) { local_paca->irq_happened |=PACA_IRQ_HARD_DIS ;__hard_RI_enable (); }trace_hardirqs_off (); -user_exit_irqoff ();return false; }
@@ -74,28 +75,26 @@ static notrace __always_inline bool prep_irq_for_user_exit(void)return true; } -/* -* restartable is true then EE/RI can be left on because interrupts are handled -* with a restart sequence. -*/ -static notrace __always_inline bool prep_irq_for_kernel_enabled_exit(bool restartable) +static notrace __always_inline bool prep_irq_for_user_exit(void) { +user_enter_irqoff ();/* This must be done with RI=1 because tracing may touch vmaps */ trace_hardirqs_on ();#ifdef CONFIG_PPC32 __hard_EE_RI_disable ();#else -if (exit_must_hard_disable () || !restartable) +if (exit_must_hard_disable ())__hard_EE_RI_disable ();/* This pattern matches prep_irq_for_idle */ if (unlikely (lazy_irq_pending_nocheck ())) { -if (exit_must_hard_disable () || !restartable) { +if (exit_must_hard_disable ()) { local_paca->irq_happened |=PACA_IRQ_HARD_DIS ;__hard_RI_enable (); }trace_hardirqs_off (); +user_exit_irqoff ();return false; }
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 16/17 powerpc/interrupt: Refactor prep_irq_for_{user/kernel_enabled}_exit()
In reply to Nicholas PigginFrom: Christophe Leroy <[email protected]>
prep_irq_for_user_exit() is a superset ofprep_irq_for_kernel_enabled_exit().
Rename prep_irq_for_kernel_enabled_exit() as prep_irq_for_enabled_exit() and have prep_irq_for_user_exit() use it.
--- arch/powerpc/kernel/interrupt.c | 29 +++++++---------------------- 1 file changed, 7 insertions(+), 22 deletions(-)
arch/powerpc/kernel/interrupt.c +7 -22
@@ -50,7 +50,7 @@ static inline bool exit_must_hard_disable(void)* restartable is true then EE/RI can be left on because interrupts are handled * with a restart sequence. */ -static notrace __always_inline bool prep_irq_for_kernel_enabled_exit(bool restartable) +static notrace __always_inline bool prep_irq_for_enabled_exit(bool restartable) {/* This must be done with RI=1 because tracing may touch vmaps */ trace_hardirqs_on ();
@@ -77,29 +77,14 @@ static notrace __always_inline bool prep_irq_for_kernel_enabled_exit(bool restar static notrace __always_inline bool prep_irq_for_user_exit(void) { -user_enter_irqoff (); -/* This must be done with RI=1 because tracing may touch vmaps */ -trace_hardirqs_on (); - -#ifdef CONFIG_PPC32 -__hard_EE_RI_disable (); -#else -if (exit_must_hard_disable ()) -__hard_EE_RI_disable (); +bool ret; -/* This pattern matches prep_irq_for_idle */ -if (unlikely (lazy_irq_pending_nocheck ())) { -if (exit_must_hard_disable ()) { - local_paca->irq_happened |=PACA_IRQ_HARD_DIS ; -__hard_RI_enable (); - } -trace_hardirqs_off (); +user_enter_irqoff (); + ret =prep_irq_for_enabled_exit (true); +if (!ret)user_exit_irqoff (); -return false; - } -#endif -return true; +return ret; }/* Has to run notrace because it is entered not completely "reconciled" */
@@ -469,7 +454,7 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs)* Stack store exit can't be restarted because the interrupt * stack frame might have been clobbered. */ -if (!prep_irq_for_kernel_enabled_exit (unlikely (stack_store))) { +if (!prep_irq_for_enabled_exit (unlikely (stack_store))) {/* * Replay pending soft-masked interrupts now. Don't * just local_irq_enabe(); local_irq_disable(); because
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v4 17/17 powerpc/interrupt: Remove prep_irq_for_user_exit()
In reply to Nicholas PigginFrom: Christophe Leroy <[email protected]>
prep_irq_for_user_exit() has only one caller, squash it inside that caller.
--- arch/powerpc/kernel/interrupt.c | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-)
arch/powerpc/kernel/interrupt.c +3 -13
@@ -75,18 +75,6 @@ static notrace __always_inline bool prep_irq_for_enabled_exit(bool restartable)return true; } -static notrace __always_inline bool prep_irq_for_user_exit(void) -{ -bool ret; - -user_enter_irqoff (); - ret =prep_irq_for_enabled_exit (true); -if (!ret) -user_exit_irqoff (); - -return ret; -} -/* Has to run notrace because it is entered not completely "reconciled" */ notrace long system_call_exception(long r3, long r4, long r5, long r6, long r7, long r8,
@@ -276,7 +264,9 @@ interrupt_exit_user_prepare_main(unsigned long ret, struct pt_regs *regs) } } -if (!prep_irq_for_user_exit ()) { +user_enter_irqoff (); +if (!prep_irq_for_enabled_exit (true)) { +user_exit_irqoff ();local_irq_enable ();local_irq_disable (); goto again;
Signature · 1 line
2.23.0
Quoted 9 lines · Nicholas Piggin
This series attempts to improve the speed of interrupts and system calls in three major ways.
Firstly, the SRR/HSRR registers do not need to be reloaded if they were clobbered for the duration of the interrupt and the return NIP and MSR did not changed. 64e does not implement this part, but it could quite easily.
[...]
Patches 1-4, and 6-17 applied to powerpc/next.
[01/17] powerpc/interrupt: Fix CONFIG ifdef typo
https://git.kernel.org/powerpc/c/9a3ed7adcabce24a85fbe05f54e762b18756ec22 [02/17] powerpc: remove interrupt exit helpers unused argument https://git.kernel.org/powerpc/c/bf9155f1970c4dbf9ec6b87d3688433bd494a4e1 [03/17] powerpc/64s: introduce different functions to return from SRR vs HSRR interrupts https://git.kernel.org/powerpc/c/1df7d5e4baeac74d14c1bee18b2dff9302b3efbc [04/17] powerpc/64s: avoid reloading (H)SRR registers if they are still valid https://git.kernel.org/powerpc/c/59dc5bfca0cb6a29db1a50847684eb5c19f8f400 [06/17] powerpc/64: move interrupt return asm to interrupt_64.S https://git.kernel.org/powerpc/c/e754f4d13e3919aafa485657599907aa63b9a40c [07/17] powerpc/64s: system call avoid setting MSR[RI] until we set MSR[EE] https://git.kernel.org/powerpc/c/dd152f70bdc1b91445b10c65ac874b90c93fb3b5 [08/17] powerpc/64s: save one more register in the masked interrupt handler https://git.kernel.org/powerpc/c/63e40806eea984f770c992120bbfd71b589ea580 [09/17] powerpc/64: allow alternate return locations for soft-masked interrupts https://git.kernel.org/powerpc/c/f23699c93becd746295aaa506537882a46a62219 [10/17] powerpc/64: interrupt soft-enable race fix https://git.kernel.org/powerpc/c/862fa563524b9f92d7e89fe332732bd3421772db [11/17] powerpc/64: treat low kernel text as irqs soft-masked https://git.kernel.org/powerpc/c/9d1988ca87dd90ecf80a0601c7fd13071fbb1a83 [12/17] powerpc/64: use interrupt restart table to speed up return from interrupt https://git.kernel.org/powerpc/c/13799748b957bc5659f97c036224b0f4b42172e2 [13/17] powerpc/interrupt: Rename and lightly change syscall_exit_prepare_main() https://git.kernel.org/powerpc/c/f84aa284947f325c5697d35b92abd2047224f24b [14/17] powerpc/interrupt: Refactor interrupt_exit_user_prepare() https://git.kernel.org/powerpc/c/a214ee8802adb864d175ea6ca4176223bcc11d2b [15/17] powerpc/interrupt: Interchange prep_irq_for_{kernel_enabled/user}_exit() https://git.kernel.org/powerpc/c/99f98f849cf13e5fac532979ccdb77dff07665db [16/17] powerpc/interrupt: Refactor prep_irq_for_{user/kernel_enabled}_exit() https://git.kernel.org/powerpc/c/61eece2d1707796fd45225ea3d20e9289251311c [17/17] powerpc/interrupt: Remove prep_irq_for_user_exit() https://git.kernel.org/powerpc/c/ae58b1c645895c28ca155843db6788d57ea99e11
cheers