powerpc/64s: fast interrupt exit
Subsystems (3)
Nicholas Piggin
reviewercover powerpc/64s: fast interrupt exit
This series attempts to improve the speed of interrupts and system calls in two major ways.
Firstly, the SRR/HSRR registers do not need to be reloaded if they were not used or clobbered fur the duration of the interrupt.
Secondly, an alternate return location facility is added for soft-masked asynchronous interrupts and then that's used to set everything up for return without having to disable MSR RI or EE.
After this series, the entire system call / interrupt handler fast path executes no mtsprs and one mtmsrd to enable interrupts initially, and the system call vectored path doesn't even need to do that.
Thanks,
Nick
Nicholas Piggin (9):
powerpc/64s: syscall real mode entry use mtmsrd rather than rfid powerpc/64s: system call avoid setting MSR[RI] until we set MSR[EE] powerpc/64s: introduce different functions to return from SRR vs HSRR
interrupts
powerpc/64s: avoid reloading (H)SRR registers if they are still valid powerpc/64: move interrupt return asm to interrupt_64.S powerpc/64s: save one more register in the masked interrupt handler powerpc/64s: allow alternate return locations for soft-masked
interrupts
powerpc/64s: interrupt soft-enable race fix
powerpc/64s: use interrupt restart table to speed up return from
interrupt
arch/powerpc/Kconfig.debug | 5 + arch/powerpc/include/asm/asm-prototypes.h | 4 +- arch/powerpc/include/asm/head-64.h | 2 +- arch/powerpc/include/asm/interrupt.h | 18 + arch/powerpc/include/asm/paca.h | 3 + arch/powerpc/include/asm/ppc_asm.h | 8 + arch/powerpc/include/asm/ptrace.h | 28 +- arch/powerpc/kernel/asm-offsets.c | 5 + arch/powerpc/kernel/entry_64.S | 508 --------------- arch/powerpc/kernel/exceptions-64s.S | 180 ++++-- arch/powerpc/kernel/fpu.S | 2 + arch/powerpc/kernel/head_64.S | 5 +- arch/powerpc/kernel/interrupt_64.S | 720 +++++++++++++++++++++ arch/powerpc/kernel/irq.c | 79 ++- arch/powerpc/kernel/kgdb.c | 2 +- arch/powerpc/kernel/kprobes-ftrace.c | 2 +- arch/powerpc/kernel/kprobes.c | 10 +- arch/powerpc/kernel/process.c | 21 +- arch/powerpc/kernel/rtas.c | 13 +- arch/powerpc/kernel/signal.c | 2 +- arch/powerpc/kernel/signal_64.c | 14 + arch/powerpc/kernel/syscall_64.c | 242 ++++--- arch/powerpc/kernel/syscalls.c | 2 + arch/powerpc/kernel/traps.c | 18 +- arch/powerpc/kernel/vector.S | 6 +- arch/powerpc/kernel/vmlinux.lds.S | 10 + arch/powerpc/lib/Makefile | 2 +- arch/powerpc/lib/restart_table.c | 26 + arch/powerpc/lib/sstep.c | 5 +- arch/powerpc/math-emu/math.c | 2 +- arch/powerpc/mm/fault.c | 2 +- arch/powerpc/perf/core-book3s.c | 19 +- arch/powerpc/platforms/powernv/opal-call.c | 3 + arch/powerpc/sysdev/fsl_pci.c | 2 +- 34 files changed, 1244 insertions(+), 726 deletions(-) create mode 100644 arch/powerpc/kernel/interrupt_64.S create mode 100644 arch/powerpc/lib/restart_table.c
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 1/9 powerpc/64s: syscall real mode entry use mtmsrd rather than rfid
Have the real mode system call entry handler branch to the kernel 0xc000... address and then use mtmsrd to enable the MMU, rather than use SRRs and rfid.
Commit 8729c26e675c ("powerpc/64s/exception: Move real to virt switch into the common handler") implemented this style of real mode entry for other interrupt handlers, so this brings system calls into line with them, which is the main motivcation for the change.
This tends to be slightly faster due to avoiding the mtsprs, and it also does not clobber the SRR registers, which becomes important in a subsequent change. The real mode entry points don't tend to be too important for performance these days, but it is possible for a hypervisor to run guests in AIL=0 mode for certian reasons.
--- arch/powerpc/kernel/entry_64.S | 6 ++++++ arch/powerpc/kernel/exceptions-64s.S | 9 +++------ 2 files changed, 9 insertions(+), 6 deletions(-)
arch/powerpc/kernel/entry_64.S +6 -0
@@ -225,6 +225,12 @@ _ASM_NOKPROBE_SYMBOL(system_call_vectored_emulate) b system_call_vectored_common #endif + .balign IFETCH_ALIGN_BYTES + .globl system_call_common_real +system_call_common_real: + ld r10,PACAKMSR(r13) /* get MSR value for kernel */ + mtmsrd r10 + .balign IFETCH_ALIGN_BYTES .globl system_call_common system_call_common:
arch/powerpc/kernel/exceptions-64s.S +3 -6
@@ -1901,12 +1901,9 @@ END_FTR_SECTION_IFSET(CPU_FTR_REAL_LE) HMT_MEDIUM .if ! \virt - __LOAD_HANDLER(r10, system_call_common) - mtspr SPRN_SRR0,r10 - ld r10,PACAKMSR(r13) - mtspr SPRN_SRR1,r10 - RFI_TO_KERNEL - b . /* prevent speculative execution */ + __LOAD_HANDLER(r10, system_call_common_real) + mtctr r10 + bctr .else li r10,MSR_RI mtmsrd r10,1 /* Set RI (EE=0) */
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 2/9 powerpc/64s: system call avoid setting MSR[RI] until we set MSR[EE]
In reply to Nicholas PigginThis extends the MSR[RI]=0 window a little further into the system call in order to pair RI and EE enabling with a single mtmsrd.
XXX: May need to make this radix-only opt. - might take SLB or HPT faults on the context tracking, etc.
--- arch/powerpc/kernel/exceptions-64s.S | 2 -- 1 file changed, 2 deletions(-)
arch/powerpc/kernel/exceptions-64s.S +0 -2
@@ -1905,8 +1905,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_REAL_LE) mtctr r10 bctr .else - li r10,MSR_RI - mtmsrd r10,1 /* Set RI (EE=0) */ #ifdef CONFIG_RELOCATABLE __LOAD_HANDLER(r10, system_call_common) mtctr r10
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 3/9 powerpc/64s: introduce different functions to return from SRR vs HSRR interrupts
In reply to Nicholas PigginThis makes no real difference yet except that HSRR type interrupts will use hrfid to return. This is important for the next patch.
--- arch/powerpc/kernel/entry_64.S | 58 +++++++++++++++------ arch/powerpc/kernel/exceptions-64s.S | 78 +++++++++++++++------------- arch/powerpc/kernel/vector.S | 2 +- 3 files changed, 85 insertions(+), 53 deletions(-)
arch/powerpc/kernel/entry_64.S +43 -15
@@ -648,47 +648,54 @@ END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_207S) blr #ifdef CONFIG_PPC_BOOK3S + /* * If MSR EE/RI was never enabled, IRQs not reconciled, NVGPRs not * touched, no exit work created, then this can be used. */ .balign IFETCH_ALIGN_BYTES - .globl fast_interrupt_return -fast_interrupt_return: -_ASM_NOKPROBE_SYMBOL(fast_interrupt_return) + .globl fast_interrupt_return_srr +fast_interrupt_return_srr: +_ASM_NOKPROBE_SYMBOL(fast_interrupt_return_srr) kuap_check_amr r3, r4 ld r5,_MSR(r1) andi. r0,r5,MSR_PR - bne .Lfast_user_interrupt_return + bne .Lfast_user_interrupt_return_srr kuap_restore_amr r3, r4 andi. r0,r5,MSR_RI li r3,0 /* 0 return value, no EMULATE_STACK_STORE */ - bne+ .Lfast_kernel_interrupt_return + bne+ .Lfast_kernel_interrupt_return_srr addi r3,r1,STACK_FRAME_OVERHEAD bl unrecoverable_exception b . /* should not get here */ +.macro interrupt_return_macro srr .balign IFETCH_ALIGN_BYTES - .globl interrupt_return -interrupt_return: -_ASM_NOKPROBE_SYMBOL(interrupt_return) + .globl interrupt_return_\srr +interrupt_return_\srr\(): +_ASM_NOKPROBE_SYMBOL(interrupt_return_\srr\()) ld r4,_MSR(r1) andi. r0,r4,MSR_PR - beq .Lkernel_interrupt_return + beq .Lkernel_interrupt_return_\srr addi r3,r1,STACK_FRAME_OVERHEAD bl interrupt_exit_user_prepare cmpdi r3,0 - bne- .Lrestore_nvgprs + bne- .Lrestore_nvgprs_\srr -.Lfast_user_interrupt_return: +.Lfast_user_interrupt_return_\srr: ld r11,_NIP(r1) ld r12,_MSR(r1) BEGIN_FTR_SECTION ld r10,_PPR(r1) mtspr SPRN_PPR,r10 END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR) + .ifc \srr,srr mtspr SPRN_SRR0,r11 mtspr SPRN_SRR1,r12 + .else + mtspr SPRN_HSRR0,r11 + mtspr SPRN_HSRR1,r12 + .endif BEGIN_FTR_SECTION stdcx. r0,0,r1 /* to clear the reservation */
@@ -715,24 +722,33 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_STCX_CHECKS_ADDRESS) REST_GPR(6, r1) REST_GPR(0, r1) REST_GPR(1, r1) + .ifc \srr,srr RFI_TO_USER + .else + HRFI_TO_USER + .endif b . /* prevent speculative execution */ -.Lrestore_nvgprs: +.Lrestore_nvgprs_\srr\(): REST_NVGPRS(r1) - b .Lfast_user_interrupt_return + b .Lfast_user_interrupt_return_\srr .balign IFETCH_ALIGN_BYTES -.Lkernel_interrupt_return: +.Lkernel_interrupt_return_\srr\(): addi r3,r1,STACK_FRAME_OVERHEAD bl interrupt_exit_kernel_prepare -.Lfast_kernel_interrupt_return: +.Lfast_kernel_interrupt_return_\srr\(): cmpdi cr1,r3,0 ld r11,_NIP(r1) ld r12,_MSR(r1) + .ifc \srr,srr mtspr SPRN_SRR0,r11 mtspr SPRN_SRR1,r12 + .else + mtspr SPRN_HSRR0,r11 + mtspr SPRN_HSRR1,r12 + .endif BEGIN_FTR_SECTION stdcx. r0,0,r1 /* to clear the reservation */
@@ -766,7 +782,11 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_STCX_CHECKS_ADDRESS) REST_GPR(6, r1) REST_GPR(0, r1) REST_GPR(1, r1) + .ifc \srr,srr RFI_TO_KERNEL + .else + HRFI_TO_KERNEL + .endif b . /* prevent speculative execution */ 1: /*
@@ -786,8 +806,16 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_STCX_CHECKS_ADDRESS) std r9,0(r1) /* perform store component of stdu */ ld r9,PACA_EXGEN+0(r13) + .ifc \srr,srr RFI_TO_KERNEL + .else + HRFI_TO_KERNEL + .endif b . /* prevent speculative execution */ +.endm + +interrupt_return_macro srr +interrupt_return_macro hsrr #endif /* CONFIG_PPC_BOOK3S */ #ifdef CONFIG_PPC_RTAS
arch/powerpc/kernel/exceptions-64s.S +41 -37
288 lines not on this page · raw message
arch/powerpc/kernel/vector.S +1 -1
@@ -133,7 +133,7 @@ _GLOBAL(load_up_vsx) /* enable use of VSX after return */ oris r12,r12,MSR_VSX@h std r12,_MSR(r1) - b fast_interrupt_return + b fast_interrupt_return_srr #endif /* CONFIG_VSX */
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 4/9 powerpc/64s: avoid reloading (H)SRR registers if they are still valid
In reply to Nicholas PigginWhen an interrupt is taken, the SRR registers are set to return to where it left off. Unless they are modified in the meantime, or the return address or MSR are modified, there is no need to reload these registers when returning from interrupt.
Introduce per-CPU flags that track the validity of SRR and HSRR registers, clear them when returning from interrupt, using the registers for something else (e.g., OPAL calls), or adjusting return address or MSR.
This improves the performance of interrupt returns.
XXX: 64e build breaks
XXX: may not need to invalidate both hsrr and srr all the time
--- arch/powerpc/Kconfig.debug | 5 ++ arch/powerpc/include/asm/paca.h | 2 + arch/powerpc/include/asm/ptrace.h | 15 +++++ arch/powerpc/kernel/asm-offsets.c | 2 + arch/powerpc/kernel/entry_64.S | 75 +++++++++++++++++++--- arch/powerpc/kernel/exceptions-64s.S | 27 ++++++++ arch/powerpc/kernel/fpu.S | 2 + arch/powerpc/kernel/kgdb.c | 2 +- arch/powerpc/kernel/kprobes-ftrace.c | 2 +- arch/powerpc/kernel/kprobes.c | 10 +-- arch/powerpc/kernel/process.c | 21 +++++- arch/powerpc/kernel/rtas.c | 13 +++- arch/powerpc/kernel/signal.c | 2 +- arch/powerpc/kernel/signal_64.c | 14 ++++ arch/powerpc/kernel/syscalls.c | 2 + arch/powerpc/kernel/traps.c | 18 +++--- arch/powerpc/kernel/vector.S | 4 ++ arch/powerpc/lib/sstep.c | 5 +- arch/powerpc/math-emu/math.c | 2 +- arch/powerpc/mm/fault.c | 2 +- arch/powerpc/platforms/powernv/opal-call.c | 3 + arch/powerpc/sysdev/fsl_pci.c | 2 +- 22 files changed, 197 insertions(+), 33 deletions(-)
arch/powerpc/Kconfig.debug +5 -0
@@ -84,6 +84,11 @@ config MSI_BITMAP_SELFTEST config PPC_IRQ_SOFT_MASK_DEBUG bool "Include extra checks for powerpc irq soft masking" + depends on PPC64 + +config PPC_RFI_SRR_DEBUG + bool "Include extra checks for RFI SRR register validity" + depends on PPC_BOOK3S_64 config XMON bool "Include xmon kernel debugger"
arch/powerpc/include/asm/paca.h +2 -0
@@ -170,6 +170,8 @@ struct paca_struct {#ifdef CONFIG_PPC_BOOK3E u16 trap_save;/* Used when bad stack is encountered */ #endif +u8 hsrr_valid;/* HSRRs set for HRFID */ +u8 srr_valid;/* SRRs set for RFID */ u8 irq_soft_mask;/* mask for irq soft masking */ u8 irq_happened;/* irq happened while soft-disabled */ u8 irq_work_pending;/* IRQ_WORK interrupt while soft-disable */
arch/powerpc/include/asm/ptrace.h +15 -0
@@ -110,6 +110,7 @@ struct pt_regs#endif /* __powerpc64__ */ #ifndef __ASSEMBLY__ +#include <asm/paca.h> static inline unsigned long instruction_pointer (struct pt_regs *regs) {
@@ -160,6 +161,20 @@ static inline void regs_set_return_value(struct pt_regs *regs, unsigned long rc) regs->gpr[3 ] = rc; } +static inline void regs_set_return_ip (struct pt_regs *regs,unsigned long ip) +{ + regs->nip = ip; +#ifdef CONFIG_PPC_BOOK3S + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ; +#endif +} + +static inline void regs_add_return_ip (struct pt_regs *regs,long offset) +{ +regs_set_return_ip (regs, regs->nip + offset); +} +#ifdef __powerpc64__#define user_mode (regs) ((((regs)->msr) >> MSR_PR_LG) & 0x1)#else
arch/powerpc/kernel/asm-offsets.c +2 -0
@@ -210,6 +210,8 @@ int main(void)OFFSET (PACATOC , paca_struct, kernel_toc);OFFSET (PACAKBASE , paca_struct, kernelbase);OFFSET (PACAKMSR , paca_struct, kernel_msr); +OFFSET (PACAHSRR_VALID , paca_struct, hsrr_valid); +OFFSET (PACASRR_VALID , paca_struct, srr_valid);OFFSET (PACAIRQSOFTMASK , paca_struct, irq_soft_mask);OFFSET (PACAIRQHAPPENED , paca_struct, irq_happened);OFFSET (PACA_FTRACE_ENABLED , paca_struct, ftrace_enabled);
arch/powerpc/kernel/entry_64.S +67 -8
138 lines not on this page · raw message
arch/powerpc/kernel/exceptions-64s.S +27 -0
@@ -567,6 +567,20 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real) std r0,GPR0(r1) /* save r0 in stackframe */ std r10,GPR1(r1) /* save r1 in stackframe */ + /* Mark our [H]SRRs valid for return */ + li r10,1 + .if IHSRR_IF_HVMODE + BEGIN_FTR_SECTION + stb r10,PACAHSRR_VALID(r13) + FTR_SECTION_ELSE + stb r10,PACASRR_VALID(r13) + ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206) + .elseif IHSRR + stb r10,PACAHSRR_VALID(r13) + .else + stb r10,PACASRR_VALID(r13) + .endif + .if ISET_RI li r10,MSR_RI mtmsrd r10,1 /* Set MSR_RI */
@@ -666,10 +680,13 @@ END_FTR_SECTION_IFSET(CPU_FTR_CFAR) .macro EXCEPTION_RESTORE_REGS hsrr=0 /* Move original SRR0 and SRR1 into the respective regs */ ld r9,_MSR(r1) + li r10,0 .if \hsrr mtspr SPRN_HSRR1,r9 + stb r10,PACAHSRR_VALID(r13) .else mtspr SPRN_SRR1,r9 + stb r10,PACASRR_VALID(r13) .endif ld r9,_NIP(r1) .if \hsrr
@@ -1781,6 +1798,8 @@ EXC_COMMON_BEGIN(hdecrementer_common) * * Be careful to avoid touching the kernel stack. */ + li r10,0 + stb r10,PACAHSRR_VALID(r13) ld r10,PACA_EXGEN+EX_CTR(r13) mtctr r10 mtcrf 0x80,r9
@@ -2611,6 +2630,8 @@ BEGIN_FTR_SECTION ld r10,PACA_EXGEN+EX_CFAR(r13) mtspr SPRN_CFAR,r10 END_FTR_SECTION_IFSET(CPU_FTR_CFAR) + li r10,0 + stb r10,PACAHSRR_VALID(r13) ld r10,PACA_EXGEN+EX_R10(r13) ld r11,PACA_EXGEN+EX_R11(r13) ld r12,PACA_EXGEN+EX_R12(r13)
@@ -2783,6 +2804,12 @@ masked_interrupt: ori r11,r11,PACA_IRQ_HARD_DIS stb r11,PACAIRQHAPPENED(r13) 2: /* done */ + li r10,0 + .if \hsrr + stb r10,PACAHSRR_VALID(r13) + .else + stb r10,PACASRR_VALID(r13) + .endif ld r10,PACA_EXGEN+EX_CTR(r13) mtctr r10 mtcrf 0x80,r9
arch/powerpc/kernel/fpu.S +2 -0
@@ -105,6 +105,8 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX) ori r12,r12,MSR_FP or r12,r12,r4 std r12,_MSR(r1) + li r4,0 + stb r4,PACASRR_VALID(r13) #endif li r4,1 stb r4,THREAD_LOAD_FP(r5)
arch/powerpc/kernel/kgdb.c +1 -1
@@ -147,7 +147,7 @@ static int kgdb_handle_breakpoint(struct pt_regs *regs)return 0 ;if (*(u32 *)regs->nip ==BREAK_INSTR ) - regs->nip +=BREAK_INSTR_SIZE ; +regs_add_return_ip (regs,BREAK_INSTR_SIZE );return 1 ; }
arch/powerpc/kernel/kprobes-ftrace.c +1 -1
@@ -40,7 +40,7 @@ void kprobe_ftrace_handler(unsigned long nip, unsigned long parent_nip,* Emulate singlestep (and also recover regs->nip) * as if there is a nop */ - regs->nip +=MCOUNT_INSN_SIZE ; +regs_add_return_ip (regs,MCOUNT_INSN_SIZE );if (unlikely (p->post_handler)) { kcb->kprobe_status =KPROBE_HIT_SSDONE ; p->post_handler (p, regs,0 );
arch/powerpc/kernel/kprobes.c +5 -5
@@ -178,7 +178,7 @@ static nokprobe_inline void prepare_singlestep(struct kprobe *p, struct pt_regs* variant as values in regs could play a part in * if the trap is taken or not */ - regs->nip = (unsigned long )p->ainsn.insn; +regs_set_return_ip (regs, (unsigned long )p->ainsn.insn); } static nokprobe_inline void save_previous_kprobe(struct kprobe_ctlblk *kcb)
@@ -415,7 +415,7 @@ static int trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs)* we end up emulating it in kprobe_handler(), which increments the nip * again. */ - regs->nip = orig_ret_address -4 ; +regs_set_return_ip (regs, orig_ret_address -4 ); regs->link = orig_ret_address;return 0 ;
@@ -450,7 +450,7 @@ int kprobe_post_handler(struct pt_regs *regs) }/* Adjust nip to after the single-stepped instruction */ - regs->nip = (unsigned long )cur->addr + len; +regs_set_return_ip (regs, (unsigned long )cur->addr + len); regs->msr |= kcb->kprobe_saved_msr;/*Restore back the original saved kprobes variables and continue. */
@@ -490,7 +490,7 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr)* and allow the page fault handler to continue as a * normal page fault. */ - regs->nip = (unsigned long )cur->addr; +regs_set_return_ip (regs, (unsigned long )cur->addr); regs->msr &= ~MSR_SINGLESTEP ;/* Turn off 'trace' bits */ regs->msr |= kcb->kprobe_saved_msr;if (kcb->kprobe_status ==KPROBE_REENTER )
@@ -523,7 +523,7 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr)* zero, try to fix up. */ if ((entry =search_exception_tables (regs->nip)) !=NULL ) { - regs->nip =extable_fixup (entry); +regs_set_return_ip (regs,extable_fixup (entry));return 1 ; }
arch/powerpc/kernel/process.c +20 -1
@@ -97,6 +97,8 @@ static void check_if_tm_restore_required(struct task_struct *tsk) !test_thread_flag(TIF_RESTORE_TM)) { tsk->thread.ckpt_regs.msr = tsk->thread.regs->msr;set_thread_flag (TIF_RESTORE_TM ); + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ; } }
@@ -161,6 +163,8 @@ static void __giveup_fpu(struct task_struct *tsk)if (cpu_has_feature (CPU_FTR_VSX )) msr &= ~MSR_VSX ; tsk->thread.regs->msr = msr; + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ; }void giveup_fpu (struct task_struct *tsk)
@@ -244,6 +248,8 @@ static void __giveup_altivec(struct task_struct *tsk)if (cpu_has_feature (CPU_FTR_VSX )) msr &= ~MSR_VSX ; tsk->thread.regs->msr = msr; + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ; }void giveup_altivec (struct task_struct *tsk)
@@ -559,6 +565,8 @@ void notrace restore_math(struct pt_regs *regs)msr_check_and_clear (new_msr); regs->msr |= new_msr | fpexc_mode; + local_paca->hsrr_valid =0 ; + local_paca->srr_valid = 0; } }#endif /* CONFIG_PPC_BOOK3S_64 */
@@ -1295,6 +1303,8 @@ struct task_struct *__switch_to(struct task_struct *prev, atomic_read(¤t->mm->context.vas_windows))) asm volatile(PPC_CP_ABORT); } + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ;#endif /* CONFIG_PPC_BOOK3S_64 */ return last;
@@ -1878,6 +1888,9 @@ void start_thread(struct pt_regs *regs, unsigned long start, unsigned long sp)#endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ thread_pkey_regs_init (¤t->thread); + + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ; }EXPORT_SYMBOL (start_thread);
@@ -1925,9 +1938,12 @@ int set_fpexc_mode(struct task_struct *tsk, unsigned int val)if (val >PR_FP_EXC_PRECISE )return -EINVAL ; tsk->thread.fpexc_mode =__pack_fe01 (val); -if (regs !=NULL && (regs->msr &MSR_FP ) !=0 ) +if (regs !=NULL && (regs->msr &MSR_FP ) !=0 ) { regs->msr = (regs->msr & ~(MSR_FE0 |MSR_FE1 )) | tsk->thread.fpexc_mode; + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ; + }return 0 ; }
@@ -1979,6 +1995,9 @@ int set_endian(struct task_struct *tsk, unsigned int val)else return -EINVAL ; + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ; +return 0 ; }
arch/powerpc/kernel/rtas.c +10 -3
@@ -46,6 +46,13 @@/* This is here deliberately so it's only used in this file */ void enter_rtas (unsigned long ); +static inline void do_enter_rtas (unsigned long args) +{ +enter_rtas (args); + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ; +} +struct rtas_t rtas = { .lock = __ARCH_SPIN_LOCK_UNLOCKED };
@@ -384,7 +391,7 @@ static char *__fetch_rtas_last_error(char *altbuf) save_args = rtas.args; rtas.args = err_args; -enter_rtas (__pa (&rtas.args)); +do_enter_rtas (__pa (&rtas.args)); err_args = rtas.args; rtas.args = save_args;
@@ -430,7 +437,7 @@ va_rtas_call_unlocked(struct rtas_args *args, int token, int nargs, int nret,for (i =0 ; i < nret; ++i) args->rets[i] =0 ; -enter_rtas (__pa (args)); +do_enter_rtas (__pa (args)); }void rtas_call_unlocked (struct rtas_args *args,int token,int nargs,int nret, ...)
@@ -1198,7 +1205,7 @@ SYSCALL_DEFINE1(rtas, struct rtas_args __user *, uargs) flags =lock_rtas (); rtas.args = args; -enter_rtas (__pa (&rtas.args)); +do_enter_rtas (__pa (&rtas.args)); args = rtas.args;/* A -1 return code indicates that the last command couldn't
arch/powerpc/kernel/signal.c +1 -1
@@ -242,7 +242,7 @@ static void check_syscall_restart(struct pt_regs *regs, struct k_sigaction *ka, regs->gpr[0 ] = __NR_restart_syscall; else regs->gpr[3] = regs->orig_gpr3; - regs->nip -=4 ; +regs_add_return_ip (regs, -4 ); regs->result =0 ; }else {if (trap_is_scv (regs)) {
arch/powerpc/kernel/signal_64.c +14 -0
@@ -693,6 +693,10 @@ SYSCALL_DEFINE3(swapcontext, struct ucontext __user *, old_ctx,/* This returns like rt_sigreturn */ set_thread_flag (TIF_RESTOREALL ); + + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ; +return 0 ; }
@@ -798,6 +802,10 @@ SYSCALL_DEFINE0(rt_sigreturn) goto badframe;set_thread_flag (TIF_RESTOREALL ); + + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ; +return 0 ; badframe:
@@ -878,6 +886,7 @@ int handle_rt_signal64(struct ksignal *ksig, sigset_t *set, err |=put_user (regs->gpr[1 ], (unsigned long __user *)newsp);/* Set up "regs" so we "return" to the signal handler. */ +/* XXX: use set return IP */ if (is_elf2_task ()) { regs->ctr = (unsigned long ) ksig->ka.sa.sa_handler; regs->gpr[12 ] = regs->ctr;
@@ -910,6 +919,9 @@ int handle_rt_signal64(struct ksignal *ksig, sigset_t *set,if (err) goto badframe; + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ; +return 0 ; badframe:
@@ -917,6 +929,8 @@ int handle_rt_signal64(struct ksignal *ksig, sigset_t *set,printk_ratelimited (regs->msr &MSR_64BIT ? fmt64 : fmt32, tsk->comm, tsk->pid,"setup_rt_frame" , (long )frame, regs->nip, regs->link); + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ;return 1 ; }
arch/powerpc/kernel/syscalls.c +2 -0
@@ -123,6 +123,8 @@ SYSCALL_DEFINE0(switch_endian)struct thread_info *ti; current->thread.regs->msr ^=MSR_LE ; + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ;/* * Set TIF_RESTOREALL so that r3 isn't clobbered on return to
arch/powerpc/kernel/traps.c +9 -9
@@ -1049,7 +1049,7 @@ static void p9_hmi_special_emu(struct pt_regs *regs)#endif /* !__LITTLE_ENDIAN__ */ /* Go to next instruction */ - regs->nip += 4; + regs_add_return_ip(regs, 4); }#endif /* CONFIG_VSX */
@@ -1487,7 +1487,7 @@ DEFINE_INTERRUPT_HANDLER(program_check_exception)if (!(regs->msr &MSR_PR ) &&/* not user-mode */ report_bug (bugaddr, regs) ==BUG_TRAP_TYPE_WARN ) { - regs->nip +=4 ; +regs_add_return_ip (regs,4 );return ; }_exception (SIGTRAP , regs,TRAP_BRKPT , regs->nip);
@@ -1549,7 +1549,7 @@ DEFINE_INTERRUPT_HANDLER(program_check_exception)if (reason & (REASON_ILLEGAL |REASON_PRIVILEGED )) {switch (emulate_instruction (regs)) {case 0 : - regs->nip +=4 ; +regs_add_return_ip (regs,4 );emulate_single_step (regs);return ;case -EFAULT :
@@ -1600,7 +1600,7 @@ DEFINE_INTERRUPT_HANDLER(alignment_exception)if (fixed ==1 ) {/* skip over emulated instruction */ - regs->nip +=inst_length (reason); +regs_add_return_ip (regs,inst_length (reason));emulate_single_step (regs);return ; }
@@ -1767,7 +1767,7 @@ DEFINE_INTERRUPT_HANDLER(facility_unavailable_exception)pr_err ("DSCR based mfspr emulation failed\n" );return ; } - regs->nip +=4 ; +regs_add_return_ip (regs,4 );emulate_single_step (regs); }return ;
@@ -2030,7 +2030,7 @@ DEFINE_INTERRUPT_HANDLER(altivec_assist_exception)PPC_WARN_EMULATED (altivec, regs); err =emulate_altivec (regs);if (err ==0 ) { - regs->nip +=4 ;/* skip emulated instruction */ +regs_add_return_ip (regs,4 );/* skip emulated instruction */ emulate_single_step (regs);return ; }
@@ -2094,7 +2094,7 @@ void SPEFloatingPointException(struct pt_regs *regs) err =do_spe_mathemu (regs);if (err ==0 ) { - regs->nip +=4 ;/* skip emulated instruction */ +regs_add_return_ip (regs,4 );/* skip emulated instruction */ emulate_single_step (regs);return ; }
@@ -2125,10 +2125,10 @@ void SPEFloatingPointRoundException(struct pt_regs *regs)giveup_spe (current);preempt_enable (); - regs->nip -=4 ; +regs_add_return_ip (regs, -4 ); err =speround_handler (regs);if (err ==0 ) { - regs->nip +=4 ;/* skip emulated instruction */ +regs_add_return_ip (regs,4 );/* skip emulated instruction */ emulate_single_step (regs);return ; }
arch/powerpc/kernel/vector.S +4 -0
@@ -75,6 +75,8 @@ _GLOBAL(load_up_altivec) addi r5,r4,THREAD /* Get THREAD */ oris r12,r12,MSR_VEC@h std r12,_MSR(r1) + li r4,0 + stb r4,PACASRR_VALID(r13) #endif li r4,1 stb r4,THREAD_LOAD_VEC(r5)
@@ -133,6 +135,8 @@ _GLOBAL(load_up_vsx) /* enable use of VSX after return */ oris r12,r12,MSR_VSX@h std r12,_MSR(r1) + li r4,0 + stb r4,PACASRR_VALID(r13) b fast_interrupt_return_srr #endif /* CONFIG_VSX */
arch/powerpc/lib/sstep.c +4 -1
@@ -3033,7 +3033,7 @@ void emulate_update_regs(struct pt_regs *regs, struct instruction_op *op)default :WARN_ON_ONCE (1 ); } - regs->nip = next_pc; +regs_set_return_ip (regs, next_pc); }NOKPROBE_SYMBOL (emulate_update_regs);
@@ -3310,6 +3310,9 @@ int emulate_step(struct pt_regs *regs, struct ppc_inst instr)unsigned long val;unsigned long ea; + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ; + r =analyse_instr (&op, regs, instr);if (r <0 )return r;
arch/powerpc/math-emu/math.c +1 -1
@@ -453,7 +453,7 @@ do_mathemu(struct pt_regs *regs)break ; } - regs->nip +=4 ; +regs_add_return_ip (regs,4 );return 0 ; illegal:
arch/powerpc/mm/fault.c +1 -1
@@ -615,7 +615,7 @@ void bad_page_fault(struct pt_regs *regs, int sig)/* Are we prepared to handle this fault? */ if ((entry =search_exception_tables (regs->nip)) !=NULL ) { - regs->nip =extable_fixup (entry); +regs_set_return_ip (regs,extable_fixup (entry));return ; }
arch/powerpc/platforms/powernv/opal-call.c +3 -0
@@ -100,6 +100,9 @@ static int64_t opal_call(int64_t a0, int64_t a1, int64_t a2, int64_t a3,bool mmu = (msr & (MSR_IR |MSR_DR ));int64_t ret; + local_paca->hsrr_valid =0 ; + local_paca->srr_valid =0 ; + msr &= ~MSR_EE ;if (unlikely (!mmu))
arch/powerpc/sysdev/fsl_pci.c +1 -1
@@ -1072,7 +1072,7 @@ int fsl_pci_mcheck_exception(struct pt_regs *regs) ret =get_kernel_nofault (inst, (void *)regs->nip);if (!ret &&mcheck_handle_load (regs, inst)) { - regs->nip +=4 ; +regs_add_return_ip (regs,4 );return 1 ; } }
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 5/9 powerpc/64: move interrupt return asm to interrupt_64.S
In reply to Nicholas PigginThe next patch would like to move interrupt return assembly code to a low location before general text, so move it into its own file and include via head_64.S
--- arch/powerpc/include/asm/head-64.h | 2 +- arch/powerpc/kernel/entry_64.S | 601 ---------------------------- arch/powerpc/kernel/head_64.S | 5 +- arch/powerpc/kernel/interrupt_64.S | 608 +++++++++++++++++++++++++++++ 4 files changed, 613 insertions(+), 603 deletions(-) create mode 100644 arch/powerpc/kernel/interrupt_64.S
arch/powerpc/include/asm/head-64.h +1 -1
@@ -16,7 +16,7 @@ .section ".head.data.\name\()","a",@progbits .endm .macro use_ftsec name - .section ".head.text.\name\()" + .section ".head.text.\name\()","ax",@progbits .endm/*
arch/powerpc/kernel/entry_64.S +0 -601
620 lines not on this page · raw message
arch/powerpc/kernel/head_64.S +4 -1
@@ -189,8 +189,9 @@ CLOSE_FIXED_SECTION(first_256B) /* This value is used to mark exception frames on the stack. */ .section ".toc","aw" +/* This value is used to mark exception frames on the stack. */ exception_marker: - .tc ID_72656773_68657265[TC],0x7265677368657265 + .tc ID_EXC_MARKER[TC],STACK_FRAME_REGS_MARKER .previous /*
@@ -206,6 +207,8 @@ OPEN_TEXT_SECTION(0x100) USE_TEXT_SECTION() +#include "interrupt_64.S" + #ifdef CONFIG_PPC_BOOK3E /* * The booting_thread_hwid holds the thread id we want to boot in cpu
arch/powerpc/kernel/interrupt_64.S new file +608 -0
608 lines not on this page · raw message
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 6/9 powerpc/64s: save one more register in the masked interrupt handler
In reply to Nicholas PigginThis frees up one more register (and takes advantage of that to clean things up a little bit).
This register will be used in the following patch.
--- arch/powerpc/kernel/exceptions-64s.S | 34 ++++++++++++++++------------ 1 file changed, 20 insertions(+), 14 deletions(-)
arch/powerpc/kernel/exceptions-64s.S +20 -14
@@ -2745,7 +2745,6 @@ INT_DEFINE_END(soft_nmi) * and run it entirely with interrupts hard disabled. */ EXC_COMMON_BEGIN(soft_nmi_common) - mfspr r11,SPRN_SRR0 mr r10,r1 ld r1,PACAEMERGSP(r13) subi r1,r1,INT_FRAME_SIZE
@@ -2780,19 +2779,24 @@ masked_Hinterrupt: .else masked_interrupt: .endif - lbz r11,PACAIRQHAPPENED(r13) - or r11,r11,r10 - stb r11,PACAIRQHAPPENED(r13) + stw r9,PACA_EXGEN+EX_CCR(r13) + lbz r9,PACAIRQHAPPENED(r13) + or r9,r9,r10 + stb r9,PACAIRQHAPPENED(r13) + + .if ! \hsrr cmpwi r10,PACA_IRQ_DEC bne 1f - lis r10,0x7fff - ori r10,r10,0xffff - mtspr SPRN_DEC,r10 + LOAD_REG_IMMEDIATE(r9, 0x7fffffff) + mtspr SPRN_DEC,r9 #ifdef CONFIG_PPC_WATCHDOG + lwz r9,PACA_EXGEN+EX_CCR(r13) b soft_nmi_common #else b 2f #endif + .endif + 1: andi. r10,r10,PACA_IRQ_MUST_HARD_MASK beq 2f xori r12,r12,MSR_EE /* clear MSR_EE */
@@ -2801,17 +2805,19 @@ masked_interrupt: .else mtspr SPRN_SRR1,r12 .endif - ori r11,r11,PACA_IRQ_HARD_DIS - stb r11,PACAIRQHAPPENED(r13) + ori r9,r9,PACA_IRQ_HARD_DIS + stb r9,PACAIRQHAPPENED(r13) 2: /* done */ - li r10,0 + li r9,0 .if \hsrr - stb r10,PACAHSRR_VALID(r13) + stb r9,PACAHSRR_VALID(r13) .else - stb r10,PACASRR_VALID(r13) + stb r9,PACASRR_VALID(r13) .endif - ld r10,PACA_EXGEN+EX_CTR(r13) - mtctr r10 + + ld r9,PACA_EXGEN+EX_CTR(r13) + mtctr r9 + lwz r9,PACA_EXGEN+EX_CCR(r13) mtcrf 0x80,r9 std r1,PACAR1(r13) ld r9,PACA_EXGEN+EX_R9(r13)
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 7/9 powerpc/64s: allow alternate return locations for soft-masked interrupts
In reply to Nicholas PigginThis is a variation of the exception table code which adjusts a failed page fault return location if it was taken at an address specified in an exception table, to a corresponding fixup handler address.
This patch adds a similar masked interrupt restart table that is checked when when an asynchronous interrupt is taken while soft-masked.
--- arch/powerpc/include/asm/interrupt.h | 18 ++++++++++++++ arch/powerpc/include/asm/ppc_asm.h | 8 +++++++ arch/powerpc/kernel/exceptions-64s.S | 36 +++++++++++++++++++++++++++- arch/powerpc/kernel/interrupt_64.S | 3 +++ arch/powerpc/kernel/vmlinux.lds.S | 10 ++++++++ arch/powerpc/lib/Makefile | 2 +- arch/powerpc/lib/restart_table.c | 26 ++++++++++++++++++++ arch/powerpc/perf/core-book3s.c | 19 ++++++++++++--- 8 files changed, 117 insertions(+), 5 deletions(-) create mode 100644 arch/powerpc/lib/restart_table.c
arch/powerpc/include/asm/interrupt.h +18 -0
@@ -18,6 +18,11 @@ static inline void nap_adjust_return(struct pt_regs *regs)#endif } +#ifdef CONFIG_PPC_BOOK3S_64 +extern char __end_soft_masked[]; +unsigned long search_kernel_restart_table (unsigned long addr); +#endif +struct interrupt_state {#ifdef CONFIG_PPC_BOOK3E_64 enum ctx_state ctx_state;
@@ -47,6 +52,9 @@ static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrup*/ if (TRAP (regs) !=0x700 )CT_WARN_ON (ct_state () !=CONTEXT_KERNEL ); + BUG_ON(regs->nip < (unsigned long)__end_soft_masked)); +if (arch_irq_disabled_regs (regs)) + BUG_ON(search_kernel_restart_table(regs->nip)); }#endif
@@ -126,6 +134,8 @@ static inline void interrupt_nmi_enter_prepare(struct pt_regs *regs, struct inte {#ifdef CONFIG_PPC64 #ifdef CONFIG_PPC_BOOK3S_64 +if (!(regs->msr &MSR_PR ) && regs->nip < (unsigned long )__end_soft_masked) + regs->softe =IRQS_ALL_DISABLED ; state->irq_soft_mask = local_paca->irq_soft_mask; state->irq_happened = local_paca->irq_happened;
@@ -163,6 +173,14 @@ static inline void interrupt_nmi_exit_prepare(struct pt_regs *regs, struct internap_adjust_return (regs); +#ifdef CONFIG_PPC_BOOK3S_64 +if (arch_irq_disabled_regs (regs)) { +unsigned long rst =search_kernel_restart_table (regs->nip); +if (rst) +regs_set_return_ip (regs, rst); + } +#endif +#ifdef CONFIG_PPC64 this_cpu_set_ftrace_enabled (state->ftrace_enabled);
arch/powerpc/include/asm/ppc_asm.h +8 -0
@@ -791,6 +791,14 @@ END_FTR_SECTION_NESTED(CPU_FTR_CELL_TB_BUG, CPU_FTR_CELL_TB_BUG, 96) stringify_in_c(.long (_target) - . ;) \ stringify_in_c(.previous) +#define RESTART_TABLE(_start, _end, _target) \ + stringify_in_c(.section __restart_table,"a";)\ + stringify_in_c(.balign 8;) \ + stringify_in_c(.llong (_start);) \ + stringify_in_c(.llong (_end);) \ + stringify_in_c(.llong (_target);) \ + stringify_in_c(.previous) +#ifdef CONFIG_PPC_FSL_BOOK3E #define BTB_FLUSH (reg) \ lis reg,BUCSR_INIT@h; \
arch/powerpc/kernel/exceptions-64s.S +35 -1
@@ -514,8 +514,9 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real) /* Kernel code running below __end_interrupts is implicitly * soft-masked */ - LOAD_HANDLER(r10, __end_interrupts) + LOAD_HANDLER(r10, __end_soft_masked) cmpld r11,r10 + li r10,IMASK blt- 1f
@@ -673,6 +674,28 @@ END_FTR_SECTION_IFSET(CPU_FTR_CFAR) __GEN_COMMON_BODY \name .endm +.macro SEARCH_RESTART_TABLE + LOAD_REG_IMMEDIATE_SYM(r9, r12, __start___restart_table) + LOAD_REG_IMMEDIATE_SYM(r10, r12, __stop___restart_table) +300: + cmpd r9,r10 + beq 302f + ld r12,0(r9) + cmpld r11,r12 + blt 301f + ld r12,8(r9) + cmpld r11,r12 + bge 301f + ld r12,16(r9) + b 303f +301: + addi r9,r9,24 + b 300b +302: + li r12,0 +303: +.endm + /* * Restore all registers including H/SRR0/1 saved in a stack frame of a * standard exception.
@@ -2758,6 +2781,7 @@ EXC_COMMON_BEGIN(soft_nmi_common) mtmsrd r9,1 kuap_restore_amr r9, r10 + EXCEPTION_RESTORE_REGS hsrr=0 RFI_TO_KERNEL
@@ -2815,6 +2839,16 @@ masked_interrupt: stb r9,PACASRR_VALID(r13) .endif + SEARCH_RESTART_TABLE + cmpdi r12,0 + beq 3f + .if \hsrr + mtspr SPRN_HSRR0,r12 + .else + mtspr SPRN_SRR0,r12 + .endif +3: + ld r9,PACA_EXGEN+EX_CTR(r13) mtctr r9 lwz r9,PACA_EXGEN+EX_CCR(r13)
arch/powerpc/kernel/interrupt_64.S +3 -0
@@ -605,4 +605,7 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_STCX_CHECKS_ADDRESS) interrupt_return_macro srr interrupt_return_macro hsrr + .globl __end_soft_masked +__end_soft_masked: +DEFINE_FIXED_SYMBOL(__end_soft_masked) #endif /* CONFIG_PPC_BOOK3S */
arch/powerpc/kernel/vmlinux.lds.S +10 -0
@@ -9,6 +9,14 @@ #define EMITS_PT_NOTE #define RO_EXCEPTION_TABLE_ALIGN 0 +#define RESTART_TABLE(align) \ + . = ALIGN(align); \ + __restart_table : AT(ADDR(__restart_table) - LOAD_OFFSET) { \ + __start___restart_table = .; \ + KEEP(*(__restart_table)) \ + __stop___restart_table = .; \ + } + #include <asm/page.h> #include <asm-generic/vmlinux.lds.h> #include <asm/cache.h>
@@ -124,6 +132,8 @@ SECTIONS RO_DATA(PAGE_SIZE) #ifdef CONFIG_PPC64 + RESTART_TABLE(8) + . = ALIGN(8); __stf_entry_barrier_fixup : AT(ADDR(__stf_entry_barrier_fixup) - LOAD_OFFSET) { __start___stf_entry_barrier_fixup = .;
arch/powerpc/lib/Makefile +1 -1
@@ -36,7 +36,7 @@ extra-$(CONFIG_PPC64) += crtsavres.o endif obj-$(CONFIG_PPC_BOOK3S_64) += copyuser_power7.o copypage_power7.o \ - memcpy_power7.o + memcpy_power7.o restart_table.o obj64-y += copypage_64.o copyuser_64.o mem_64.o hweight_64.o \ memcpy_64.o copy_mc_64.o
arch/powerpc/lib/restart_table.c new file +26 -0
@@ -0,0 +1,26 @@ +struct restart_table_entry { +unsigned long start; +unsigned long end; +unsigned long fixup; +}; + +extern struct restart_table_entry __start___restart_table[]; +extern struct restart_table_entry __stop___restart_table[]; + +/* Given an address, look for it in the kernel exception table */ +unsigned long search_kernel_restart_table (unsigned long addr) +{ +struct restart_table_entry *rte = __start___restart_table; + +while (rte < __stop___restart_table) { +unsigned long start = rte->start; +unsigned long end = rte->end; +unsigned long fixup = rte->fixup; + +if (addr >= start && addr < end) +return fixup; + + rte++; + } +return 0 ; +}
arch/powerpc/perf/core-book3s.c +16 -3
@@ -332,9 +332,15 @@ static inline void perf_read_regs(struct pt_regs *regs)* If interrupts were soft-disabled when a PMU interrupt occurs, treat * it as an NMI. */ +extern char __end_soft_masked[];static inline int perf_intr_is_nmi (struct pt_regs *regs) { -return (regs->softe &IRQS_DISABLED ); +if (regs->softe &IRQS_DISABLED ) +return true; + +if (!(regs->msr &MSR_PR ) && regs->nip < (unsigned long )__end_soft_masked) +return true; +return false; }/*
@@ -2214,6 +2220,8 @@ static bool pmc_overflow(unsigned long val)return false; } +unsigned long search_kernel_restart_table (unsigned long addr); +/* * Performance monitor interrupt stuff */
@@ -2301,10 +2309,15 @@ static void __perf_event_interrupt(struct pt_regs *regs)*/ write_mmcr0 (cpuhw, cpuhw->mmcr.mmcr0); -if (nmi) +if (nmi) { +unsigned long rst =search_kernel_restart_table (regs->nip); +if (rst) +regs_set_return_ip (regs, rst); +nmi_exit (); -else + }else {irq_exit (); + } }static void perf_event_interrupt (struct pt_regs *regs)
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 8/9 powerpc/64s: interrupt soft-enable race fix
In reply to Nicholas PigginPrevent interrupt restore from allowing racing hard interrupts going ahead of previous soft-pending ones, by using the soft-masked restart handler to allow a store to clear the soft-mask while knowing nothing is soft-pending.
This probably doesn't matter much in practice, but it's a simple demonstrator / test case to exercise the restart table logic.
XXX: 64e can't do this unless it adds restart table support
--- arch/powerpc/kernel/irq.c | 79 +++++++++++++++++++++++++-------------- 1 file changed, 50 insertions(+), 29 deletions(-)
arch/powerpc/kernel/irq.c +50 -29
@@ -317,52 +317,73 @@ notrace void arch_local_irq_restore(unsigned long mask)unsigned char irq_happened;/* Write the new soft-enabled value */ -irq_soft_mask_set (mask); -if (mask) +if (mask) { +irq_soft_mask_set (mask);return ; + }/* -* From this point onward, we can take interrupts, preempt, -* etc... unless we got hard-disabled. We check if an event -* happened. If none happened, we know we can just return. -* -* We may have preempted before the check below, in which case -* we are checking the "new" CPU instead of the old one. This -* is only a problem if an event happened on the "old" CPU. +* After the stb, interrupts are unmasked and there are no interrupts +* pending replay. The restart sequence makes this atomic with +* respect to soft-masked interrupts. If this was just a simple code +* sequence, a soft-masked interrupt could become pending right after +* the comparison and before the stb. * -* External interrupt events will have caused interrupts to -* be hard-disabled, so there is no problem, we -* cannot have preempted. +* This allows interrupts to be unmasked without hard disabling, and +* also without new hard interrupts coming in ahead of pending ones. */ +asm_volatile_goto ( +"1: \n" +" lbz 9,%0(13) \n" +" cmpwi 9,0 \n" +" bne %l[happened] \n" +" stb 9,%1(13) \n" +"2: \n" +RESTART_TABLE (1b ,2b ,1b ) + : : "i" (offsetof(struct paca_struct, irq_happened)), + "i" (offsetof(struct paca_struct, irq_soft_mask)) + : "cr0", "r9" + : happened); + +if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) +WARN_ON_ONCE (!(mfmsr () &MSR_EE )); + +return ; + +happened: irq_happened =get_irq_happened (); -if (!irq_happened) { +if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) +WARN_ON_ONCE (!irq_happened); + +if (irq_happened ==PACA_IRQ_HARD_DIS ) {if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) -WARN_ON_ONCE (!(mfmsr () &MSR_EE )); +WARN_ON_ONCE (mfmsr () &MSR_EE ); +irq_soft_mask_set (IRQS_ENABLED ); + local_paca->irq_happened =0 ; +__hard_irq_enable ();return ; } -/* We need to hard disable to replay. */ +/* Have interrupts to replay, need to hard disable first */ if (!(irq_happened &PACA_IRQ_HARD_DIS )) { -if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) -WARN_ON_ONCE (!(mfmsr () &MSR_EE )); +if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) { +if (!(mfmsr () &MSR_EE )) { +/* +* An interrupt could have come in and cleared +* MSR[EE] and set IRQ_HARD_DIS, so check +* IRQ_HARD_DIS again and warn if it is still +* clear. +*/ + irq_happened =get_irq_happened (); +WARN_ON_ONCE (!(irq_happened &PACA_IRQ_HARD_DIS )); + } + }__hard_irq_disable (); }else { -/* -* We should already be hard disabled here. We had bugs -* where that wasn't the case so let's dbl check it and -* warn if we are wrong. Only do that when IRQ tracing -* is enabled as mfmsr() can be costly. -*/ if (IS_ENABLED (CONFIG_PPC_IRQ_SOFT_MASK_DEBUG )) {if (WARN_ON_ONCE (mfmsr () &MSR_EE ))__hard_irq_disable (); } - -if (irq_happened ==PACA_IRQ_HARD_DIS ) { - local_paca->irq_happened =0 ; -__hard_irq_enable (); -return ; - } }/*
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 9/9 powerpc/64s: use interrupt restart table to speed up return from interrupt
In reply to Nicholas PigginThe restart table facility is used to return from interrupt without disabling MSR[EE] or MSR[RI].
Interrupt return code is put into the low soft-masked region, and critical code that has return SRRs set, soft-masked state set with no soft-pending interrupts to replay, and no exit work, is put into alternate return sections. r1 is saved in the paca. If an interrupt hits at this point, our fixup location re-loads r1 and from there we can get to regs, critical things are re-loaded, and we go around and do the exit sequence again.
--- arch/powerpc/include/asm/asm-prototypes.h | 4 +- arch/powerpc/include/asm/paca.h | 1 + arch/powerpc/include/asm/ptrace.h | 13 +- arch/powerpc/kernel/asm-offsets.c | 3 + arch/powerpc/kernel/interrupt_64.S | 119 ++++++++++- arch/powerpc/kernel/syscall_64.c | 242 +++++++++++++--------- 6 files changed, 277 insertions(+), 105 deletions(-)
arch/powerpc/include/asm/asm-prototypes.h +2 -2
@@ -71,8 +71,8 @@ void __init machine_init(u64 dt_ptr);#endif long system_call_exception (long r3,long r4,long r5,long r6,long r7,long r8,unsigned long r0,struct pt_regs *regs);notrace unsigned long syscall_exit_prepare (unsigned long r3,struct pt_regs *regs,long scv); -notrace unsigned long interrupt_exit_user_prepare (struct pt_regs *regs,unsigned long msr); -notrace unsigned long interrupt_exit_kernel_prepare (struct pt_regs *regs,unsigned long msr); +notrace unsigned long interrupt_exit_user_prepare (struct pt_regs *regs); +notrace unsigned long interrupt_exit_kernel_prepare (struct pt_regs *regs);long ppc_fadvise64_64 (int fd,int advice,u32 offset_high,u32 offset_low,u32 len_high,u32 len_low);
arch/powerpc/include/asm/paca.h +1 -0
@@ -167,6 +167,7 @@ struct paca_struct {u64 kstack;/* Saved Kernel stack addr */ u64 saved_r1;/* r1 save for RTAS calls or PM or EE=0 */ u64 saved_msr;/* MSR saved here by enter_rtas */ +u64 exit_save_r1;/* Syscall/interrupt R1 save */ #ifdef CONFIG_PPC_BOOK3E u16 trap_save;/* Used when bad stack is encountered */ #endif
arch/powerpc/include/asm/ptrace.h +11 -2
@@ -50,14 +50,23 @@ struct pt_regsunion {struct { -#ifdef CONFIG_PPC64 +#ifdef CONFIG_PPC_BOOK3S_64 unsigned long ppr; +unsigned long exit_result;#endif #ifdef CONFIG_PPC_KUAP unsigned long kuap;#endif }; -unsigned long __pad[2 ];/* Maintain 16 byte interrupt stack alignment */ + +/* Maintain 16 byte interrupt stack alignment */ +#ifdef CONFIG_PPC_KUAP +#ifdef CONFIG_PPC_BOOK3S_64 +unsigned long __pad[4 ]; +#else +unsigned long __pad[2 ]; +#endif +#endif }; };#endif
arch/powerpc/kernel/asm-offsets.c +3 -0
@@ -290,6 +290,7 @@ int main(void)OFFSET (ACCOUNT_STARTTIME_USER , paca_struct, accounting.starttime_user);OFFSET (ACCOUNT_USER_TIME , paca_struct, accounting.utime);OFFSET (ACCOUNT_SYSTEM_TIME , paca_struct, accounting.stime); +OFFSET (PACA_EXIT_SAVE_R1 , paca_struct, exit_save_r1);#ifdef CONFIG_PPC_BOOK3E OFFSET (PACA_TRAP_SAVE , paca_struct, trap_save);#endif
@@ -353,7 +354,9 @@ int main(void)STACK_PT_REGS_OFFSET (_ESR, dsisr);#else /* CONFIG_PPC64 */ STACK_PT_REGS_OFFSET (SOFTE , softe); +#ifdef CONFIG_PPC_BOOK3S_64 STACK_PT_REGS_OFFSET (_PPR, ppr); +#endif #endif /* CONFIG_PPC64 */ #ifdef CONFIG_PPC_KUAP
arch/powerpc/kernel/interrupt_64.S +114 -5
193 lines not on this page · raw message
arch/powerpc/kernel/syscall_64.c +146 -96
350 lines not on this page · raw message
Signature · 1 line
2.23.0
Le 06/11/2020 à 16:59, Nicholas Piggin a écrit :
Quoted 6 lines
This series attempts to improve the speed of interrupts and system calls in two major ways.
Firstly, the SRR/HSRR registers do not need to be reloaded if they were not used or clobbered fur the duration of the interrupt.
Quoted 7 lines
Secondly, an alternate return location facility is added for soft-masked asynchronous interrupts and then that's used to set everything up for return without having to disable MSR RI or EE.
After this series, the entire system call / interrupt handler fast path executes no mtsprs and one mtmsrd to enable interrupts initially, and the system call vectored path doesn't even need to do that.
Interesting series.
Unfortunately, can't be done on PPC32 (at least on non bookE), because it would mean mapping kernel at 0 instead of 0xC0000000. Not sure libc would like it, and anyway it would be an issue for catching NULL pointer dereferencing, unless we use page tables instead of BATs to map kernel mem, which would be serious performance cut.
Christophe
Quoted 56 lines
Thanks,
NickNicholas Piggin (9):
powerpc/64s: syscall real mode entry use mtmsrd rather than rfid powerpc/64s: system call avoid setting MSR[RI] until we set MSR[EE] powerpc/64s: introduce different functions to return from SRR vs HSRR
interrupts
powerpc/64s: avoid reloading (H)SRR registers if they are still valid powerpc/64: move interrupt return asm tointerrupt_64.Spowerpc/64s: save one more register in the masked interrupt handler powerpc/64s: allow alternate return locations for soft-masked
interrupts
powerpc/64s: interrupt soft-enable race fix
powerpc/64s: use interrupt restart table to speed up return from
interruptarch/powerpc/Kconfig.debug | 5 + arch/powerpc/include/asm/asm-prototypes.h | 4 +- arch/powerpc/include/asm/head-64.h | 2 +- arch/powerpc/include/asm/interrupt.h | 18 + arch/powerpc/include/asm/paca.h | 3 + arch/powerpc/include/asm/ppc_asm.h | 8 + arch/powerpc/include/asm/ptrace.h | 28 +- arch/powerpc/kernel/asm-offsets.c | 5 + arch/powerpc/kernel/entry_64.S | 508 --------------- arch/powerpc/kernel/exceptions-64s.S | 180 ++++-- arch/powerpc/kernel/fpu.S | 2 + arch/powerpc/kernel/head_64.S | 5 +- arch/powerpc/kernel/interrupt_64.S | 720 +++++++++++++++++++++ arch/powerpc/kernel/irq.c | 79 ++- arch/powerpc/kernel/kgdb.c | 2 +- arch/powerpc/kernel/kprobes-ftrace.c | 2 +- arch/powerpc/kernel/kprobes.c | 10 +- arch/powerpc/kernel/process.c | 21 +- arch/powerpc/kernel/rtas.c | 13 +- arch/powerpc/kernel/signal.c | 2 +- arch/powerpc/kernel/signal_64.c | 14 + arch/powerpc/kernel/syscall_64.c | 242 ++++--- arch/powerpc/kernel/syscalls.c | 2 + arch/powerpc/kernel/traps.c | 18 +- arch/powerpc/kernel/vector.S | 6 +- arch/powerpc/kernel/vmlinux.lds.S | 10 + arch/powerpc/lib/Makefile | 2 +- arch/powerpc/lib/restart_table.c | 26 + arch/powerpc/lib/sstep.c | 5 +- arch/powerpc/math-emu/math.c | 2 +- arch/powerpc/mm/fault.c | 2 +- arch/powerpc/perf/core-book3s.c | 19 +- arch/powerpc/platforms/powernv/opal-call.c | 3 + arch/powerpc/sysdev/fsl_pci.c | 2 +- 34 files changed, 1244 insertions(+), 726 deletions(-)create mode 100644
arch/powerpc/kernel/interrupt_64.Screate mode 100644arch/powerpc/lib/restart_table.c
Nicholas Piggin
reviewerExcerpts from Christophe Leroy's message of November 7, 2020 8:35 pm:
Quoted 17 lines
Le 06/11/2020 à 16:59, Nicholas Piggin a écrit :
Quoted 6 lines
This series attempts to improve the speed of interrupts and system calls in two major ways.
Firstly, the SRR/HSRR registers do not need to be reloaded if they were not used or clobbered fur the duration of the interrupt.
Quoted 7 lines
Secondly, an alternate return location facility is added for soft-masked asynchronous interrupts and then that's used to set everything up for return without having to disable MSR RI or EE.
After this series, the entire system call / interrupt handler fast path executes no mtsprs and one mtmsrd to enable interrupts initially, and the system call vectored path doesn't even need to do that.
Quoted 6 lines
Interesting series.
Unfortunately, can't be done on PPC32 (at least on non bookE), because it would mean mapping kernel at 0 instead of
0xC0000000. Not sure libc would like it, and anyway it would be an issue for catching NULL pointer dereferencing, unless we use page tables instead of BATs to map kernel mem, which would be serious performance cut.
Hmm, why would you have to map at 0?
PPC32 doesn't have soft mask interrupts, but you could still test all MSR[PR]=0 interrupts to see if they land inside some region to see if they hit in the restart table I think?
Could PPC32 skip the SRR reload at least? That's simpler.
Thanks,
Nick
Le 10/11/2020 à 09:49, Nicholas Piggin a écrit :
Quoted 18 lines
Excerpts from Christophe Leroy's message of November 7, 2020 8:35 pm:
Quoted 17 lines
Le 06/11/2020 à 16:59, Nicholas Piggin a écrit :
Quoted 13 lines
This series attempts to improve the speed of interrupts and system calls in two major ways.
Firstly, the SRR/HSRR registers do not need to be reloaded if they were not used or clobbered fur the duration of the interrupt.
Secondly, an alternate return location facility is added for soft-masked asynchronous interrupts and then that's used to set everything up for return without having to disable MSR RI or EE.
After this series, the entire system call / interrupt handler fast path executes no mtsprs and one mtmsrd to enable interrupts initially, and the system call vectored path doesn't even need to do that.
Quoted 8 lines
Quoted 6 lines
Interesting series.
Unfortunately, can't be done on PPC32 (at least on non bookE), because it would mean mapping kernel at 0 instead of
0xC0000000. Not sure libc would like it, and anyway it would be an issue for catching NULL pointer dereferencing, unless we use page tables instead of BATs to map kernel mem, which would be serious performance cut.Hmm, why would you have to map at 0?
In real mode, physical mem is at 0. If we want to switch from real to virtual mode by just writing to MSR, then we need virtuel addresses match with real addresses ?
We could play with chip selects to put RAM at 0xc0000000, but then we'd have a problem with exception as they have to be at 0. Or we could play with MSR[IP] and get the exceptions at 0xfff00000, but that would not be so easy I guess.
Quoted 4 lines
PPC32 doesn't have soft mask interrupts, but you could still test all MSR[PR]=0 interrupts to see if they land inside some region to see if they hit in the restart table I think?
Yes and this is already what is done at exit for the ones that handle MSR[RI] I think.
Quoted 2 lines
Could PPC32 skip the SRR reload at least? That's simpler.
I think that would only be possible if real addresses where matching virtual addresses, or am I missing something ?
Christophe
Nicholas Piggin
reviewerExcerpts from Christophe Leroy's message of November 10, 2020 9:31 pm:
Quoted 28 lines
Le 10/11/2020 à 09:49, Nicholas Piggin a écrit :
Quoted 25 lines
Excerpts from Christophe Leroy's message of November 7, 2020 8:35 pm:
Quoted 23 lines
Le 06/11/2020 à 16:59, Nicholas Piggin a écrit :
Quoted 6 lines
This series attempts to improve the speed of interrupts and system calls in two major ways.
Firstly, the SRR/HSRR registers do not need to be reloaded if they were not used or clobbered fur the duration of the interrupt.
Quoted 7 lines
Secondly, an alternate return location facility is added for soft-masked asynchronous interrupts and then that's used to set everything up for return without having to disable MSR RI or EE.
After this series, the entire system call / interrupt handler fast path executes no mtsprs and one mtmsrd to enable interrupts initially, and the system call vectored path doesn't even need to do that.
Interesting series.
Unfortunately, can't be done on PPC32 (at least on non bookE), because it would mean mapping kernel at 0 instead of
0xC0000000. Not sure libc would like it, and anyway it would be an issue for catching NULL pointer dereferencing, unless we use page tables instead of BATs to map kernel mem, which would be serious performance cut.
Quoted 4 lines
Quoted 1 line
Hmm, why would you have to map at 0?
In real mode, physical mem is at 0. If we want to switch from real to virtual mode by just writing to MSR, then we need virtuel addresses match with real addresses ?
Ah that's what I missed.
64s real mode masks out the top 2 bits of the address which is how that works. But I don't usually think about that path anyway because most iterrupts arrive with MMU on.
Quoted 10 lines
We could play with chip selects to put RAM at
0xc0000000, but then we'd have a problem with exception as they have to be at 0. Or we could play with MSR[IP] and get the exceptions at0xfff00000, but that would not be so easy I guess.Quoted 4 lines
PPC32 doesn't have soft mask interrupts, but you could still test all MSR[PR]=0 interrupts to see if they land inside some region to see if they hit in the restart table I think?
Yes and this is already what is done at exit for the ones that handle MSR[RI] I think.
Interesting, I'll have to check that out.
Quoted 5 lines
Quoted 2 lines
Could PPC32 skip the SRR reload at least? That's simpler.
I think that would only be possible if real addresses where matching virtual addresses, or am I missing something ?
No you're right, I was missing something.
Thanks,
Nick