Skip to contentSkip to conversation

powerpc/64: fast interrupt exits

PATCH Superseded Nicholas Piggin 2 participants 17 messages Jun 10, 2021 – Jun 16, 2021

Subsystems (4)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Nicholas Piggin

author reviewer
#0 lore raw

v3 cover powerpc/64: fast interrupt exits

This series attempts to improve the speed of interrupts and system calls in three major ways.

Firstly, the SRR/HSRR registers do not need to be reloaded if they were clobbered for the duration of the interrupt and the return NIP and MSR did not changed. 64e does not implement this part, but it could quite easily.

Secondly, mtmsrd and mtspr are reduced by various means, this is mostly specific to 64s.

Lastly, an alternate interrupt return location facility is added for soft-masked asynchronous interrupts, and then that's used to set everything up for return without having to disable MSR RI or EE.

After this series, the entire system call / interrupt handler fast path can execute with no mtsprs and one mtmsrd to enable interrupts initially, and the system call vectored path doesn't even need to do that. This gives a decent performance benefit. On POWER9 with a powernv_defconfig without VIRT_CPU_ACCOUNTING_NATIVE, no meltdown workarounds, gettid sc system call goes from 481 -> 344 cycles, gettid scv 345->299 cycles, and page fault 1225->1064 cycles.

Changes since v2:
- KVM PR compile fix
- Missed SRR clobbers from hcalls causing pseries crashes
- Add program check recursion test to interrupt entry assert, which prevented pseries crash bug being cleanly reported.

Changes since v1:
- Rebased to upstream
- Reordered paches, updated comments and changelogs.
- Clean ups, fixed ppc32 compile breakage.
- Fixed missing srr invalidations in KVM guest entry/exit.
- Dropped the security fallback rework from the start of the series because it's quite complicated to change. Instead for now the incompatible fallbacks just disable the interrupts-enabled exits.

Thanks,
Nick

Nicholas Piggin (11):
powerpc: remove interrupt exit helpers unused argument powerpc/64s: introduce different functions to return from SRR vs HSRR
interrupts
powerpc/64s: avoid reloading (H)SRR registers if they are still valid powerpc/64: handle MSR EE and RI in interrupt entry wrapper powerpc/64: move interrupt return asm to interrupt_64.S powerpc/64s: system call avoid setting MSR[RI] until we set MSR[EE] powerpc/64s: save one more register in the masked interrupt handler powerpc/64: allow alternate return locations for soft-masked
interrupts
powerpc/64: interrupt soft-enable race fix
powerpc/64: treat low kernel text as irqs soft-masked powerpc/64: use interrupt restart table to speed up return from
interrupt

 arch/powerpc/Kconfig.debug                 |   5 +
 arch/powerpc/include/asm/asm-prototypes.h  |   4 +-
 arch/powerpc/include/asm/head-64.h         |   2 +-
 arch/powerpc/include/asm/hw_irq.h          |  23 +-
 arch/powerpc/include/asm/interrupt.h       |  60 +-
 arch/powerpc/include/asm/paca.h            |   7 +
 arch/powerpc/include/asm/ppc_asm.h         |   8 +
 arch/powerpc/include/asm/ptrace.h          |  75 ++-
 arch/powerpc/kernel/asm-offsets.c          |   7 +
 arch/powerpc/kernel/entry_64.S             | 516 --------------
 arch/powerpc/kernel/exceptions-64e.S       |  53 +-
 arch/powerpc/kernel/exceptions-64s.S       | 219 +++---
 arch/powerpc/kernel/fpu.S                  |   2 +
 arch/powerpc/kernel/head_64.S              |   5 +-
 arch/powerpc/kernel/interrupt.c            | 389 +++++++----
 arch/powerpc/kernel/interrupt_64.S         | 749 +++++++++++++++++++++
 arch/powerpc/kernel/irq.c                  |  95 +++
 arch/powerpc/kernel/kgdb.c                 |   2 +-
 arch/powerpc/kernel/kprobes-ftrace.c       |   2 +-
 arch/powerpc/kernel/kprobes.c              |  10 +-
 arch/powerpc/kernel/process.c              |  26 +-
 arch/powerpc/kernel/prom_init.c            |   3 +
 arch/powerpc/kernel/rtas.c                 |  14 +-
 arch/powerpc/kernel/signal.c               |   2 +-
 arch/powerpc/kernel/signal_64.c            |   9 +
 arch/powerpc/kernel/syscalls.c             |   3 +-
 arch/powerpc/kernel/traps.c                |  18 +-
 arch/powerpc/kernel/vector.S               |   6 +-
 arch/powerpc/kernel/vmlinux.lds.S          |  10 +
 arch/powerpc/kvm/book3s_hv.c               |   3 +
 arch/powerpc/kvm/book3s_pr.c               |   2 +
 arch/powerpc/lib/Makefile                  |   2 +-
 arch/powerpc/lib/feature-fixups.c          |  38 +-
 arch/powerpc/lib/restart_table.c           |  29 +
 arch/powerpc/lib/sstep.c                   |   4 +-
 arch/powerpc/math-emu/math.c               |   2 +-
 arch/powerpc/platforms/powernv/opal-call.c |   4 +
 arch/powerpc/platforms/pseries/hvCall.S    |  29 +
 arch/powerpc/sysdev/fsl_pci.c              |   2 +-
 39 files changed, 1631 insertions(+), 808 deletions(-)
 create mode 100644 arch/powerpc/kernel/interrupt_64.S
 create mode 100644 arch/powerpc/lib/restart_table.c
Signature · 1 line
2.23.0
author reviewer
#1 lore raw

PATCH v3 1/11 powerpc: remove interrupt exit helpers unused argument

The msr argument is not used, remove it.

---
 arch/powerpc/include/asm/asm-prototypes.h | 4 ++--
 arch/powerpc/kernel/interrupt.c           | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
arch/powerpc/include/asm/asm-prototypes.h +2 -2
index 1c7b75834e04..95492655462e 100644
@@ -71,8 +71,8 @@ void __init machine_init(u64 dt_ptr);
 #endif
 long system_call_exception(long r3, long r4, long r5, long r6, long r7, long r8, unsigned long r0, struct pt_regs *regs);
 notrace unsigned long syscall_exit_prepare(unsigned long r3, struct pt_regs *regs, long scv);
-notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned long msr);
-notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsigned long msr);
+notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs);
+notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs);
 
 long ppc_fadvise64_64(int fd, int advice, u32 offset_high, u32 offset_low,
 		      u32 len_high, u32 len_low);
arch/powerpc/kernel/interrupt.c +2 -2
index e0938ba298f2..38ae7057d6c2 100644
@@ -352,7 +352,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
 	return ret;
 }
 
-notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned long msr)
+notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs)
 {
 	unsigned long ti_flags;
 	unsigned long flags;
@@ -431,7 +431,7 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned
 
 void preempt_schedule_irq(void);
 
-notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsigned long msr)
+notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs)
 {
 	unsigned long flags;
 	unsigned long ret = 0;
Signature · 1 line
2.23.0
author reviewer
#2 lore raw

PATCH v3 2/11 powerpc/64s: introduce different functions to return from SRR vs HSRR interrupts

In reply to Nicholas Piggin

This makes no real difference yet except that HSRR type interrupts will use hrfid to return. This is important for the next patch.

---
 arch/powerpc/kernel/entry_64.S       | 65 +++++++++++++++++-------
 arch/powerpc/kernel/exceptions-64e.S |  4 ++
 arch/powerpc/kernel/exceptions-64s.S | 76 +++++++++++++++-------------
 arch/powerpc/kernel/vector.S         |  2 +-
 4 files changed, 92 insertions(+), 55 deletions(-)
arch/powerpc/kernel/entry_64.S +47 -18
index 03727308d8cc..03a45a88b4b8 100644
@@ -635,51 +635,57 @@ END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_207S)
 	 * touched, no exit work created, then this can be used.
 	 */
 	.balign IFETCH_ALIGN_BYTES
-	.globl fast_interrupt_return
-fast_interrupt_return:
-_ASM_NOKPROBE_SYMBOL(fast_interrupt_return)
+	.globl fast_interrupt_return_srr
+fast_interrupt_return_srr:
+_ASM_NOKPROBE_SYMBOL(fast_interrupt_return_srr)
 	kuap_check_amr r3, r4
 	ld	r5,_MSR(r1)
 	andi.	r0,r5,MSR_PR
 #ifdef CONFIG_PPC_BOOK3S
-	bne	.Lfast_user_interrupt_return_amr
+	bne	.Lfast_user_interrupt_return_amr_srr
 	kuap_kernel_restore r3, r4
 	andi.	r0,r5,MSR_RI
 	li	r3,0 /* 0 return value, no EMULATE_STACK_STORE */
-	bne+	.Lfast_kernel_interrupt_return
+	bne+	.Lfast_kernel_interrupt_return_srr
 	addi	r3,r1,STACK_FRAME_OVERHEAD
 	bl	unrecoverable_exception
 	b	. /* should not get here */
 #else
-	bne	.Lfast_user_interrupt_return
-	b	.Lfast_kernel_interrupt_return
+	bne	.Lfast_user_interrupt_return_srr
+	b	.Lfast_kernel_interrupt_return_srr
 #endif
 
+.macro interrupt_return_macro srr
 	.balign IFETCH_ALIGN_BYTES
-	.globl interrupt_return
-interrupt_return:
-_ASM_NOKPROBE_SYMBOL(interrupt_return)
+	.globl interrupt_return_\srr
+interrupt_return_\srr\():
+_ASM_NOKPROBE_SYMBOL(interrupt_return_\srr\())
 	ld	r4,_MSR(r1)
 	andi.	r0,r4,MSR_PR
-	beq	.Lkernel_interrupt_return
+	beq	.Lkernel_interrupt_return_\srr
 	addi	r3,r1,STACK_FRAME_OVERHEAD
 	bl	interrupt_exit_user_prepare
 	cmpdi	r3,0
-	bne-	.Lrestore_nvgprs
+	bne-	.Lrestore_nvgprs_\srr
 
 #ifdef CONFIG_PPC_BOOK3S
-.Lfast_user_interrupt_return_amr:
+.Lfast_user_interrupt_return_amr_\srr\():
 	kuap_user_restore r3, r4
 #endif
-.Lfast_user_interrupt_return:
+.Lfast_user_interrupt_return_\srr\():
 	ld	r11,_NIP(r1)
 	ld	r12,_MSR(r1)
 BEGIN_FTR_SECTION
 	ld	r10,_PPR(r1)
 	mtspr	SPRN_PPR,r10
 END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
+	.ifc \srr,srr
 	mtspr	SPRN_SRR0,r11
 	mtspr	SPRN_SRR1,r12
+	.else
+	mtspr	SPRN_HSRR0,r11
+	mtspr	SPRN_HSRR1,r12
+	.endif
 
 BEGIN_FTR_SECTION
 	stdcx.	r0,0,r1		/* to clear the reservation */
@@ -706,24 +712,33 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_STCX_CHECKS_ADDRESS)
 	REST_GPR(6, r1)
 	REST_GPR(0, r1)
 	REST_GPR(1, r1)
+	.ifc \srr,srr
 	RFI_TO_USER
+	.else
+	HRFI_TO_USER
+	.endif
 	b	.	/* prevent speculative execution */
 
-.Lrestore_nvgprs:
+.Lrestore_nvgprs_\srr\():
 	REST_NVGPRS(r1)
-	b	.Lfast_user_interrupt_return
+	b	.Lfast_user_interrupt_return_\srr
 
 	.balign IFETCH_ALIGN_BYTES
-.Lkernel_interrupt_return:
+.Lkernel_interrupt_return_\srr\():
 	addi	r3,r1,STACK_FRAME_OVERHEAD
 	bl	interrupt_exit_kernel_prepare
 
-.Lfast_kernel_interrupt_return:
+.Lfast_kernel_interrupt_return_\srr\():
 	cmpdi	cr1,r3,0
 	ld	r11,_NIP(r1)
 	ld	r12,_MSR(r1)
+	.ifc \srr,srr
 	mtspr	SPRN_SRR0,r11
 	mtspr	SPRN_SRR1,r12
+	.else
+	mtspr	SPRN_HSRR0,r11
+	mtspr	SPRN_HSRR1,r12
+	.endif
 
 BEGIN_FTR_SECTION
 	stdcx.	r0,0,r1		/* to clear the reservation */
@@ -757,7 +772,11 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_STCX_CHECKS_ADDRESS)
 	REST_GPR(6, r1)
 	REST_GPR(0, r1)
 	REST_GPR(1, r1)
+	.ifc \srr,srr
 	RFI_TO_KERNEL
+	.else
+	HRFI_TO_KERNEL
+	.endif
 	b	.	/* prevent speculative execution */
 
 1:	/*
@@ -777,8 +796,18 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_STCX_CHECKS_ADDRESS)
 	std	r9,0(r1) /* perform store component of stdu */
 	ld	r9,PACA_EXGEN+0(r13)
 
+	.ifc \srr,srr
 	RFI_TO_KERNEL
+	.else
+	HRFI_TO_KERNEL
+	.endif
 	b	.	/* prevent speculative execution */
+.endm
+
+interrupt_return_macro srr
+#ifdef CONFIG_PPC_BOOK3S
+interrupt_return_macro hsrr
+#endif
 
 #ifdef CONFIG_PPC_RTAS
 /*
arch/powerpc/kernel/exceptions-64e.S +4 -0
index f1ae710274bc..b35c97c7082f 100644
@@ -26,6 +26,10 @@
 #include <asm/feature-fixups.h>
 #include <asm/context_tracking.h>
 
+/* 64e interrupt returns always use SRR registers */
+#define fast_interrupt_return fast_interrupt_return_srr
+#define interrupt_return interrupt_return_srr
+
 /* XXX This will ultimately add space for a special exception save
  *     structure used to save things like SRR0/SRR1, SPRGs, MAS, etc...
  *     when taking special interrupts. For now we don't support that,
arch/powerpc/kernel/exceptions-64s.S +40 -36

288 lines not on this page · raw message

arch/powerpc/kernel/vector.S +1 -1
index f5a52f444e36..54dbefcb4cde 100644
@@ -131,7 +131,7 @@ _GLOBAL(load_up_vsx)
 	/* enable use of VSX after return */
 	oris	r12,r12,MSR_VSX@h
 	std	r12,_MSR(r1)
-	b	fast_interrupt_return
+	b	fast_interrupt_return_srr
 
 #endif /* CONFIG_VSX */
 
Signature · 1 line
2.23.0
author reviewer
#3 lore raw

PATCH v3 3/11 powerpc/64s: avoid reloading (H)SRR registers if they are still valid

In reply to Nicholas Piggin

When an interrupt is taken, the SRR registers are set to return to where it left off. Unless they are modified in the meantime, or the return address or MSR are modified, there is no need to reload these registers when returning from interrupt.

Introduce per-CPU flags that track the validity of SRR and HSRR registers. These are cleared when returning from interrupt, when using the registers for something else (e.g., OPAL calls), when adjusting the return address or MSR of a context, and when context switching (which changes the return address and MSR).

This improves the performance of interrupt returns.

---
 arch/powerpc/Kconfig.debug                 |  5 ++
 arch/powerpc/include/asm/hw_irq.h          | 10 ++-
 arch/powerpc/include/asm/interrupt.h       | 12 +++
 arch/powerpc/include/asm/paca.h            |  4 +
 arch/powerpc/include/asm/ptrace.h          | 74 ++++++++++++-----
 arch/powerpc/kernel/asm-offsets.c          |  4 +
 arch/powerpc/kernel/entry_64.S             | 92 ++++++++++++++++++++--
 arch/powerpc/kernel/exceptions-64s.S       | 27 +++++++
 arch/powerpc/kernel/fpu.S                  |  2 +
 arch/powerpc/kernel/kgdb.c                 |  2 +-
 arch/powerpc/kernel/kprobes-ftrace.c       |  2 +-
 arch/powerpc/kernel/kprobes.c              | 10 +--
 arch/powerpc/kernel/process.c              | 26 +++---
 arch/powerpc/kernel/prom_init.c            |  3 +
 arch/powerpc/kernel/rtas.c                 | 14 +++-
 arch/powerpc/kernel/signal.c               |  2 +-
 arch/powerpc/kernel/signal_64.c            |  9 +++
 arch/powerpc/kernel/syscalls.c             |  3 +-
 arch/powerpc/kernel/traps.c                | 18 ++---
 arch/powerpc/kernel/vector.S               |  4 +
 arch/powerpc/kvm/book3s_hv.c               |  3 +
 arch/powerpc/kvm/book3s_pr.c               |  2 +
 arch/powerpc/lib/sstep.c                   |  4 +-
 arch/powerpc/math-emu/math.c               |  2 +-
 arch/powerpc/platforms/powernv/opal-call.c |  4 +
 arch/powerpc/platforms/pseries/hvCall.S    | 29 +++++++
 arch/powerpc/sysdev/fsl_pci.c              |  2 +-
 27 files changed, 306 insertions(+), 63 deletions(-)
arch/powerpc/Kconfig.debug +5 -0
index 6342f9da4545..205cd77f321f 100644
@@ -84,6 +84,11 @@ config MSI_BITMAP_SELFTEST
 
 config PPC_IRQ_SOFT_MASK_DEBUG
 	bool "Include extra checks for powerpc irq soft masking"
+	depends on PPC64
+
+config PPC_RFI_SRR_DEBUG
+	bool "Include extra checks for RFI SRR register validity"
+	depends on PPC_BOOK3S_64
 
 config XMON
 	bool "Include xmon kernel debugger"
arch/powerpc/include/asm/hw_irq.h +9 -1
index 56a98936a6a9..19bcef666cf6 100644
@@ -389,7 +389,15 @@ static inline bool arch_irq_disabled_regs(struct pt_regs *regs)
 	return !(regs->msr & MSR_EE);
 }
 
-static inline void may_hard_irq_enable(void) { }
+static inline bool may_hard_irq_enable(void)
+{
+	return false;
+}
+
+static inline void do_hard_irq_enable(void)
+{
+	BUILD_BUG();
+}
 
 static inline void irq_soft_mask_regs_set_state(struct pt_regs *regs, unsigned long val)
 {
arch/powerpc/include/asm/interrupt.h +12 -0
index 59f704408d65..6e9d18838d56 100644
@@ -73,6 +73,18 @@
 #include <asm/kprobes.h>
 #include <asm/runlatch.h>
 
+#ifdef CONFIG_PPC_BOOK3S_64
+static inline void srr_regs_clobbered(void)
+{
+	local_paca->srr_valid = 0;
+	local_paca->hsrr_valid = 0;
+}
+#else
+static inline void srr_regs_clobbered(void)
+{
+}
+#endif
+
 static inline void nap_adjust_return(struct pt_regs *regs)
 {
 #ifdef CONFIG_PPC_970_NAP
arch/powerpc/include/asm/paca.h +4 -0
index ec18ac818e3a..dfc984b0e640 100644
@@ -169,6 +169,10 @@ struct paca_struct {
 	u64 saved_msr;			/* MSR saved here by enter_rtas */
 #ifdef CONFIG_PPC_BOOK3E
 	u16 trap_save;			/* Used when bad stack is encountered */
+#endif
+#ifdef CONFIG_PPC_BOOK3S_64
+	u8 hsrr_valid;			/* HSRRs set for HRFID */
+	u8 srr_valid;			/* SRRs set for RFID */
 #endif
 	u8 irq_soft_mask;		/* mask for irq soft masking */
 	u8 irq_happened;		/* irq happened while soft-disabled */
arch/powerpc/include/asm/ptrace.h +53 -21

86 lines not on this page · raw message

arch/powerpc/kernel/asm-offsets.c +4 -0
index 28af4efb4587..1c2b869d9d52 100644
@@ -209,6 +209,10 @@ int main(void)
 	OFFSET(PACATOC, paca_struct, kernel_toc);
 	OFFSET(PACAKBASE, paca_struct, kernelbase);
 	OFFSET(PACAKMSR, paca_struct, kernel_msr);
+#ifdef CONFIG_PPC_BOOK3S
+	OFFSET(PACAHSRR_VALID, paca_struct, hsrr_valid);
+	OFFSET(PACASRR_VALID, paca_struct, srr_valid);
+#endif
 	OFFSET(PACAIRQSOFTMASK, paca_struct, irq_soft_mask);
 	OFFSET(PACAIRQHAPPENED, paca_struct, irq_happened);
 	OFFSET(PACA_FTRACE_ENABLED, paca_struct, ftrace_enabled);
arch/powerpc/kernel/entry_64.S +85 -7

149 lines not on this page · raw message

arch/powerpc/kernel/exceptions-64s.S +27 -0
index 3a63feedae0b..5c18a2a3058d 100644
@@ -567,6 +567,20 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real)
 	std	r0,GPR0(r1)		/* save r0 in stackframe	*/
 	std	r10,GPR1(r1)		/* save r1 in stackframe	*/
 
+	/* Mark our [H]SRRs valid for return */
+	li	r10,1
+	.if IHSRR_IF_HVMODE
+	BEGIN_FTR_SECTION
+	stb	r10,PACAHSRR_VALID(r13)
+	FTR_SECTION_ELSE
+	stb	r10,PACASRR_VALID(r13)
+	ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206)
+	.elseif IHSRR
+	stb	r10,PACAHSRR_VALID(r13)
+	.else
+	stb	r10,PACASRR_VALID(r13)
+	.endif
+
 	.if ISET_RI
 	li	r10,MSR_RI
 	mtmsrd	r10,1			/* Set MSR_RI */
@@ -666,10 +680,13 @@ END_FTR_SECTION_IFSET(CPU_FTR_CFAR)
 .macro EXCEPTION_RESTORE_REGS hsrr=0
 	/* Move original SRR0 and SRR1 into the respective regs */
 	ld	r9,_MSR(r1)
+	li	r10,0
 	.if \hsrr
 	mtspr	SPRN_HSRR1,r9
+	stb	r10,PACAHSRR_VALID(r13)
 	.else
 	mtspr	SPRN_SRR1,r9
+	stb	r10,PACASRR_VALID(r13)
 	.endif
 	ld	r9,_NIP(r1)
 	.if \hsrr
@@ -1825,6 +1842,8 @@ EXC_COMMON_BEGIN(hdecrementer_common)
 	 *
 	 * Be careful to avoid touching the kernel stack.
 	 */
+	li	r10,0
+	stb	r10,PACAHSRR_VALID(r13)
 	ld	r10,PACA_EXGEN+EX_CTR(r13)
 	mtctr	r10
 	mtcrf	0x80,r9
@@ -2660,6 +2679,8 @@ BEGIN_FTR_SECTION
 	ld	r10,PACA_EXGEN+EX_CFAR(r13)
 	mtspr	SPRN_CFAR,r10
 END_FTR_SECTION_IFSET(CPU_FTR_CFAR)
+	li	r10,0
+	stb	r10,PACAHSRR_VALID(r13)
 	ld	r10,PACA_EXGEN+EX_R10(r13)
 	ld	r11,PACA_EXGEN+EX_R11(r13)
 	ld	r12,PACA_EXGEN+EX_R12(r13)
@@ -2828,6 +2849,12 @@ masked_interrupt:
 	ori	r11,r11,PACA_IRQ_HARD_DIS
 	stb	r11,PACAIRQHAPPENED(r13)
 2:	/* done */
+	li	r10,0
+	.if \hsrr
+	stb	r10,PACAHSRR_VALID(r13)
+	.else
+	stb	r10,PACASRR_VALID(r13)
+	.endif
 	ld	r10,PACA_EXGEN+EX_CTR(r13)
 	mtctr	r10
 	mtcrf	0x80,r9
arch/powerpc/kernel/fpu.S +2 -0
index 2c57ece6671c..44526c157bff 100644
@@ -103,6 +103,8 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX)
 	ori	r12,r12,MSR_FP
 	or	r12,r12,r4
 	std	r12,_MSR(r1)
+	li	r4,0
+	stb	r4,PACASRR_VALID(r13)
 #endif
 	li	r4,1
 	stb	r4,THREAD_LOAD_FP(r5)
arch/powerpc/kernel/kgdb.c +1 -1
index 7dd2ad3603ad..26163497d69e 100644
@@ -147,7 +147,7 @@ static int kgdb_handle_breakpoint(struct pt_regs *regs)
 		return 0;
 
 	if (*(u32 *)regs->nip == BREAK_INSTR)
-		regs->nip += BREAK_INSTR_SIZE;
+		regs_add_return_ip(regs, BREAK_INSTR_SIZE);
 
 	return 1;
 }
arch/powerpc/kernel/kprobes-ftrace.c +1 -1
index 660138f6c4b2..a4965a32628a 100644
@@ -48,7 +48,7 @@ void kprobe_ftrace_handler(unsigned long nip, unsigned long parent_nip,
 			 * Emulate singlestep (and also recover regs->nip)
 			 * as if there is a nop
 			 */
-			regs->nip += MCOUNT_INSN_SIZE;
+			regs_add_return_ip(regs, MCOUNT_INSN_SIZE);
 			if (unlikely(p->post_handler)) {
 				kcb->kprobe_status = KPROBE_HIT_SSDONE;
 				p->post_handler(p, regs, 0);
arch/powerpc/kernel/kprobes.c +5 -5
index 01ab2163659e..8165ed71ab51 100644
@@ -178,7 +178,7 @@ static nokprobe_inline void prepare_singlestep(struct kprobe *p, struct pt_regs
 	 * variant as values in regs could play a part in
 	 * if the trap is taken or not
 	 */
-	regs->nip = (unsigned long)p->ainsn.insn;
+	regs_set_return_ip(regs, (unsigned long)p->ainsn.insn);
 }
 
 static nokprobe_inline void save_previous_kprobe(struct kprobe_ctlblk *kcb)
@@ -415,7 +415,7 @@ static int trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs)
 	 * we end up emulating it in kprobe_handler(), which increments the nip
 	 * again.
 	 */
-	regs->nip = orig_ret_address - 4;
+	regs_set_return_ip(regs, orig_ret_address - 4);
 	regs->link = orig_ret_address;
 
 	return 0;
@@ -450,7 +450,7 @@ int kprobe_post_handler(struct pt_regs *regs)
 	}
 
 	/* Adjust nip to after the single-stepped instruction */
-	regs->nip = (unsigned long)cur->addr + len;
+	regs_set_return_ip(regs, (unsigned long)cur->addr + len);
 	regs->msr |= kcb->kprobe_saved_msr;
 
 	/*Restore back the original saved kprobes variables and continue. */
@@ -490,7 +490,7 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr)
 		 * and allow the page fault handler to continue as a
 		 * normal page fault.
 		 */
-		regs->nip = (unsigned long)cur->addr;
+		regs_set_return_ip(regs, (unsigned long)cur->addr);
 		regs->msr &= ~MSR_SINGLESTEP; /* Turn off 'trace' bits */
 		regs->msr |= kcb->kprobe_saved_msr;
 		if (kcb->kprobe_status == KPROBE_REENTER)
@@ -523,7 +523,7 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr)
 		 * zero, try to fix up.
 		 */
 		if ((entry = search_exception_tables(regs->nip)) != NULL) {
-			regs->nip = extable_fixup(entry);
+			regs_set_return_ip(regs, extable_fixup(entry));
 			return 1;
 		}
 
arch/powerpc/kernel/process.c +16 -10
index 89e34aa273e2..fea5c68daeef 100644
@@ -96,7 +96,8 @@ static void check_if_tm_restore_required(struct task_struct *tsk)
 	if (tsk == current && tsk->thread.regs &&
 	    MSR_TM_ACTIVE(tsk->thread.regs->msr) &&
 	    !test_thread_flag(TIF_RESTORE_TM)) {
-		tsk->thread.ckpt_regs.msr = tsk->thread.regs->msr;
+		regs_set_return_msr(&tsk->thread.ckpt_regs,
+						tsk->thread.regs->msr);
 		set_thread_flag(TIF_RESTORE_TM);
 	}
 }
@@ -161,7 +162,7 @@ static void __giveup_fpu(struct task_struct *tsk)
 	msr &= ~(MSR_FP|MSR_FE0|MSR_FE1);
 	if (cpu_has_feature(CPU_FTR_VSX))
 		msr &= ~MSR_VSX;
-	tsk->thread.regs->msr = msr;
+	regs_set_return_msr(tsk->thread.regs, msr);
 }
 
 void giveup_fpu(struct task_struct *tsk)
@@ -244,7 +245,7 @@ static void __giveup_altivec(struct task_struct *tsk)
 	msr &= ~MSR_VEC;
 	if (cpu_has_feature(CPU_FTR_VSX))
 		msr &= ~MSR_VSX;
-	tsk->thread.regs->msr = msr;
+	regs_set_return_msr(tsk->thread.regs, msr);
 }
 
 void giveup_altivec(struct task_struct *tsk)
@@ -559,7 +560,7 @@ void notrace restore_math(struct pt_regs *regs)
 
 		msr_check_and_clear(new_msr);
 
-		regs->msr |= new_msr | fpexc_mode;
+		regs_set_return_msr(regs, regs->msr | new_msr | fpexc_mode);
 	}
 }
 #endif /* CONFIG_PPC_BOOK3S_64 */
@@ -1287,6 +1288,8 @@ struct task_struct *__switch_to(struct task_struct *prev,
 	}
 #endif /* CONFIG_PPC_BOOK3S_64 */
 
+	return_ip_or_msr_changed();
+
 	return last;
 }
 
@@ -1845,6 +1848,9 @@ void start_thread(struct pt_regs *regs, unsigned long start, unsigned long sp)
 		regs->gpr[2] = 0;
 		regs->msr = MSR_USER32;
 	}
+
+	return_ip_or_msr_changed();
+
 #endif
 #ifdef CONFIG_VSX
 	current->thread.used_vsr = 0;
@@ -1875,7 +1881,6 @@ void start_thread(struct pt_regs *regs, unsigned long start, unsigned long sp)
 	current->thread.tm_tfiar = 0;
 	current->thread.load_tm = 0;
 #endif /* CONFIG_PPC_TRANSACTIONAL_MEM */
-
 }
 EXPORT_SYMBOL(start_thread);
 
@@ -1923,9 +1928,10 @@ int set_fpexc_mode(struct task_struct *tsk, unsigned int val)
 	if (val > PR_FP_EXC_PRECISE)
 		return -EINVAL;
 	tsk->thread.fpexc_mode = __pack_fe01(val);
-	if (regs != NULL && (regs->msr & MSR_FP) != 0)
-		regs->msr = (regs->msr & ~(MSR_FE0|MSR_FE1))
-			| tsk->thread.fpexc_mode;
+	if (regs != NULL && (regs->msr & MSR_FP) != 0) {
+		regs_set_return_msr(regs, (regs->msr & ~(MSR_FE0|MSR_FE1))
+						| tsk->thread.fpexc_mode);
+	}
 	return 0;
 }
 
@@ -1971,9 +1977,9 @@ int set_endian(struct task_struct *tsk, unsigned int val)
 		return -EINVAL;
 
 	if (val == PR_ENDIAN_BIG)
-		regs->msr &= ~MSR_LE;
+		regs_set_return_msr(regs, regs->msr & ~MSR_LE);
 	else if (val == PR_ENDIAN_LITTLE || val == PR_ENDIAN_PPC_LITTLE)
-		regs->msr |= MSR_LE;
+		regs_set_return_msr(regs, regs->msr | MSR_LE);
 	else
 		return -EINVAL;
 
arch/powerpc/kernel/prom_init.c +3 -0
index 41ed7e33d897..f4774614e7eb 100644
@@ -31,6 +31,7 @@
 #include <asm/rtas.h>
 #include <asm/page.h>
 #include <asm/processor.h>
+#include <asm/interrupt.h>
 #include <asm/irq.h>
 #include <asm/io.h>
 #include <asm/smp.h>
@@ -1762,6 +1763,8 @@ static int prom_rtas_hcall(uint64_t args)
 	asm volatile("sc 1\n" : "=r" (arg1) :
 			"r" (arg1),
 			"r" (arg2) :);
+	srr_regs_clobbered();
+
 	return arg1;
 }
 
arch/powerpc/kernel/rtas.c +11 -3
index 6bada744402b..99f2cce635fb 100644
@@ -25,6 +25,7 @@
 #include <linux/reboot.h>
 #include <linux/syscalls.h>
 
+#include <asm/interrupt.h>
 #include <asm/prom.h>
 #include <asm/rtas.h>
 #include <asm/hvcall.h>
@@ -46,6 +47,13 @@
 /* This is here deliberately so it's only used in this file */
 void enter_rtas(unsigned long);
 
+static inline void do_enter_rtas(unsigned long args)
+{
+	enter_rtas(args);
+
+	srr_regs_clobbered(); /* rtas uses SRRs, invalidate */
+}
+
 struct rtas_t rtas = {
 	.lock = __ARCH_SPIN_LOCK_UNLOCKED
 };
@@ -384,7 +392,7 @@ static char *__fetch_rtas_last_error(char *altbuf)
 	save_args = rtas.args;
 	rtas.args = err_args;
 
-	enter_rtas(__pa(&rtas.args));
+	do_enter_rtas(__pa(&rtas.args));
 
 	err_args = rtas.args;
 	rtas.args = save_args;
@@ -430,7 +438,7 @@ va_rtas_call_unlocked(struct rtas_args *args, int token, int nargs, int nret,
 	for (i = 0; i < nret; ++i)
 		args->rets[i] = 0;
 
-	enter_rtas(__pa(args));
+	do_enter_rtas(__pa(args));
 }
 
 void rtas_call_unlocked(struct rtas_args *args, int token, int nargs, int nret, ...)
@@ -1138,7 +1146,7 @@ SYSCALL_DEFINE1(rtas, struct rtas_args __user *, uargs)
 	flags = lock_rtas();
 
 	rtas.args = args;
-	enter_rtas(__pa(&rtas.args));
+	do_enter_rtas(__pa(&rtas.args));
 	args = rtas.args;
 
 	/* A -1 return code indicates that the last command couldn't
arch/powerpc/kernel/signal.c +1 -1
index 9ded046edb0e..3f851fa8f5f1 100644
@@ -214,7 +214,7 @@ static void check_syscall_restart(struct pt_regs *regs, struct k_sigaction *ka,
 			regs->gpr[0] = __NR_restart_syscall;
 		else
 			regs->gpr[3] = regs->orig_gpr3;
-		regs->nip -= 4;
+		regs_add_return_ip(regs, -4);
 		regs->result = 0;
 	} else {
 		if (trap_is_scv(regs)) {
arch/powerpc/kernel/signal_64.c +9 -0
index dca66481d0c2..4a62faefba0f 100644
@@ -720,6 +720,9 @@ SYSCALL_DEFINE3(swapcontext, struct ucontext __user *, old_ctx,
 
 	/* This returns like rt_sigreturn */
 	set_thread_flag(TIF_RESTOREALL);
+
+	return_ip_or_msr_changed();
+
 	return 0;
 
 efault_out:
@@ -832,6 +835,9 @@ SYSCALL_DEFINE0(rt_sigreturn)
 		goto badframe;
 
 	set_thread_flag(TIF_RESTOREALL);
+
+	return_ip_or_msr_changed();
+
 	return 0;
 
 badframe_block:
@@ -957,12 +963,15 @@ int handle_rt_signal64(struct ksignal *ksig, sigset_t *set,
 	if (err)
 		goto badframe;
 
+	return_ip_or_msr_changed();
+
 	return 0;
 
 badframe_block:
 	user_write_access_end();
 badframe:
 	signal_fault(current, regs, "handle_rt_signal64", frame);
+	return_ip_or_msr_changed();
 
 	return 1;
 }
arch/powerpc/kernel/syscalls.c +2 -1
index a552c9e68d7e..bf4ae0f0e36c 100644
@@ -114,7 +114,8 @@ SYSCALL_DEFINE0(switch_endian)
 {
 	struct thread_info *ti;
 
-	current->thread.regs->msr ^= MSR_LE;
+	regs_set_return_msr(current->thread.regs,
+				current->thread.regs->msr ^ MSR_LE);
 
 	/*
 	 * Set TIF_RESTOREALL so that r3 isn't clobbered on return to
arch/powerpc/kernel/traps.c +9 -9
index b4ab95c9e94a..e413f9908664 100644
@@ -1031,7 +1031,7 @@ static void p9_hmi_special_emu(struct pt_regs *regs)
 #endif /* !__LITTLE_ENDIAN__ */
 
 	/* Go to next instruction */
-	regs->nip += 4;
+	regs_add_return_ip(regs, 4);
 }
 #endif /* CONFIG_VSX */
 
@@ -1476,7 +1476,7 @@ static void do_program_check(struct pt_regs *regs)
 
 		if (!(regs->msr & MSR_PR) &&  /* not user-mode */
 		    report_bug(bugaddr, regs) == BUG_TRAP_TYPE_WARN) {
-			regs->nip += 4;
+			regs_add_return_ip(regs, 4);
 			return;
 		}
 		_exception(SIGTRAP, regs, TRAP_BRKPT, regs->nip);
@@ -1538,7 +1538,7 @@ static void do_program_check(struct pt_regs *regs)
 	if (reason & (REASON_ILLEGAL | REASON_PRIVILEGED)) {
 		switch (emulate_instruction(regs)) {
 		case 0:
-			regs->nip += 4;
+			regs_add_return_ip(regs, 4);
 			emulate_single_step(regs);
 			return;
 		case -EFAULT:
@@ -1593,7 +1593,7 @@ DEFINE_INTERRUPT_HANDLER(alignment_exception)
 
 	if (fixed == 1) {
 		/* skip over emulated instruction */
-		regs->nip += inst_length(reason);
+		regs_add_return_ip(regs, inst_length(reason));
 		emulate_single_step(regs);
 		return;
 	}
@@ -1751,7 +1751,7 @@ DEFINE_INTERRUPT_HANDLER(facility_unavailable_exception)
 				pr_err("DSCR based mfspr emulation failed\n");
 				return;
 			}
-			regs->nip += 4;
+			regs_add_return_ip(regs, 4);
 			emulate_single_step(regs);
 		}
 		return;
@@ -2044,7 +2044,7 @@ DEFINE_INTERRUPT_HANDLER(altivec_assist_exception)
 	PPC_WARN_EMULATED(altivec, regs);
 	err = emulate_altivec(regs);
 	if (err == 0) {
-		regs->nip += 4;		/* skip emulated instruction */
+		regs_add_return_ip(regs, 4); /* skip emulated instruction */
 		emulate_single_step(regs);
 		return;
 	}
@@ -2109,7 +2109,7 @@ DEFINE_INTERRUPT_HANDLER(SPEFloatingPointException)
 
 	err = do_spe_mathemu(regs);
 	if (err == 0) {
-		regs->nip += 4;		/* skip emulated instruction */
+		regs_add_return_ip(regs, 4); /* skip emulated instruction */
 		emulate_single_step(regs);
 		return;
 	}
@@ -2140,10 +2140,10 @@ DEFINE_INTERRUPT_HANDLER(SPEFloatingPointRoundException)
 		giveup_spe(current);
 	preempt_enable();
 
-	regs->nip -= 4;
+	regs_add_return_ip(regs, -4);
 	err = speround_handler(regs);
 	if (err == 0) {
-		regs->nip += 4;		/* skip emulated instruction */
+		regs_add_return_ip(regs, 4); /* skip emulated instruction */
 		emulate_single_step(regs);
 		return;
 	}
arch/powerpc/kernel/vector.S +4 -0
index 54dbefcb4cde..02f8925c7919 100644
@@ -73,6 +73,8 @@ _GLOBAL(load_up_altivec)
 	addi	r5,r4,THREAD		/* Get THREAD */
 	oris	r12,r12,MSR_VEC@h
 	std	r12,_MSR(r1)
+	li	r4,0
+	stb	r4,PACASRR_VALID(r13)
 #endif
 	li	r4,1
 	stb	r4,THREAD_LOAD_VEC(r5)
@@ -131,6 +133,8 @@ _GLOBAL(load_up_vsx)
 	/* enable use of VSX after return */
 	oris	r12,r12,MSR_VSX@h
 	std	r12,_MSR(r1)
+	li	r4,0
+	stb	r4,PACASRR_VALID(r13)
 	b	fast_interrupt_return_srr
 
 #endif /* CONFIG_VSX */
arch/powerpc/kvm/book3s_hv.c +3 -0
index 28a80d240b76..efd95b867210 100644
@@ -4461,6 +4461,9 @@ static int kvmppc_vcpu_run_hv(struct kvm_vcpu *vcpu)
 
 	vcpu->arch.state = KVMPPC_VCPU_NOTREADY;
 	atomic_dec(&kvm->arch.vcpus_running);
+
+	srr_regs_clobbered();
+
 	return r;
 }
 
arch/powerpc/kvm/book3s_pr.c +2 -0
index d7733b07f489..1ed5ceee73eb 100644
@@ -25,6 +25,7 @@
 #include <asm/cputable.h>
 #include <asm/cacheflush.h>
 #include <linux/uaccess.h>
+#include <asm/interrupt.h>
 #include <asm/io.h>
 #include <asm/kvm_ppc.h>
 #include <asm/kvm_book3s.h>
@@ -1848,6 +1849,7 @@ static int kvmppc_vcpu_run_pr(struct kvm_vcpu *vcpu)
 	/* Make sure we save the guest TAR/EBB/DSCR state */
 	kvmppc_giveup_fac(vcpu, FSCR_TAR_LG);
 
+	srr_regs_clobbered();
 out:
 	vcpu->mode = OUTSIDE_GUEST_MODE;
 	return ret;
arch/powerpc/lib/sstep.c +3 -1
index 45bda2520755..90fa3878299a 100644
@@ -3203,7 +3203,7 @@ void emulate_update_regs(struct pt_regs *regs, struct instruction_op *op)
 	default:
 		WARN_ON_ONCE(1);
 	}
-	regs->nip = next_pc;
+	regs_set_return_ip(regs, next_pc);
 }
 NOKPROBE_SYMBOL(emulate_update_regs);
 
@@ -3480,6 +3480,8 @@ int emulate_step(struct pt_regs *regs, struct ppc_inst instr)
 	unsigned long val;
 	unsigned long ea;
 
+	return_ip_or_msr_changed();
+
 	r = analyse_instr(&op, regs, instr);
 	if (r < 0)
 		return r;
arch/powerpc/math-emu/math.c +1 -1
index 327165f26ca6..36761bd00f38 100644
@@ -453,7 +453,7 @@ do_mathemu(struct pt_regs *regs)
 		break;
 	}
 
-	regs->nip += 4;
+	regs_add_return_ip(regs, 4);
 	return 0;
 
 illegal:
arch/powerpc/platforms/powernv/opal-call.c +4 -0
index 01401e3da7ca..f812c74c61e5 100644
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 #include <linux/percpu.h>
 #include <linux/jump_label.h>
+#include <asm/interrupt.h>
 #include <asm/opal-api.h>
 #include <asm/trace.h>
 #include <asm/asm-prototypes.h>
@@ -100,6 +101,9 @@ static int64_t opal_call(int64_t a0, int64_t a1, int64_t a2, int64_t a3,
 	bool mmu = (msr & (MSR_IR|MSR_DR));
 	int64_t ret;
 
+	/* OPAL call / firmware may use SRR and/or HSRR */
+	srr_regs_clobbered();
+
 	msr &= ~MSR_EE;
 
 	if (unlikely(!mmu))
arch/powerpc/platforms/pseries/hvCall.S +29 -0
index 8a2b8d64265b..ab9fc6506861 100644
@@ -108,6 +108,10 @@ _GLOBAL_TOC(plpar_hcall_norets_notrace)
 	mfcr	r0
 	stw	r0,8(r1)
 	HVSC				/* invoke the hypervisor */
+
+	li	r4,0
+	stb	r4,PACASRR_VALID(r13)
+
 	lwz	r0,8(r1)
 	mtcrf	0xff,r0
 	blr				/* return r3 = status */
@@ -120,6 +124,9 @@ _GLOBAL_TOC(plpar_hcall_norets)
 	HCALL_BRANCH(plpar_hcall_norets_trace)
 	HVSC				/* invoke the hypervisor */
 
+	li	r4,0
+	stb	r4,PACASRR_VALID(r13)
+
 	lwz	r0,8(r1)
 	mtcrf	0xff,r0
 	blr				/* return r3 = status */
@@ -129,6 +136,10 @@ plpar_hcall_norets_trace:
 	HCALL_INST_PRECALL(R4)
 	HVSC
 	HCALL_INST_POSTCALL_NORETS
+
+	li	r4,0
+	stb	r4,PACASRR_VALID(r13)
+
 	lwz	r0,8(r1)
 	mtcrf	0xff,r0
 	blr
@@ -159,6 +170,9 @@ _GLOBAL_TOC(plpar_hcall)
 	std	r6, 16(r12)
 	std	r7, 24(r12)
 
+	li	r4,0
+	stb	r4,PACASRR_VALID(r13)
+
 	lwz	r0,8(r1)
 	mtcrf	0xff,r0
 
@@ -188,6 +202,9 @@ plpar_hcall_trace:
 
 	HCALL_INST_POSTCALL(r12)
 
+	li	r4,0
+	stb	r4,PACASRR_VALID(r13)
+
 	lwz	r0,8(r1)
 	mtcrf	0xff,r0
 
@@ -223,6 +240,9 @@ _GLOBAL(plpar_hcall_raw)
 	std	r6, 16(r12)
 	std	r7, 24(r12)
 
+	li	r4,0
+	stb	r4,PACASRR_VALID(r13)
+
 	lwz	r0,8(r1)
 	mtcrf	0xff,r0
 
@@ -262,6 +282,9 @@ _GLOBAL_TOC(plpar_hcall9)
 	std	r11,56(r12)
 	std	r0, 64(r12)
 
+	li	r4,0
+	stb	r4,PACASRR_VALID(r13)
+
 	lwz	r0,8(r1)
 	mtcrf	0xff,r0
 
@@ -300,6 +323,9 @@ plpar_hcall9_trace:
 
 	HCALL_INST_POSTCALL(r12)
 
+	li	r4,0
+	stb	r4,PACASRR_VALID(r13)
+
 	lwz	r0,8(r1)
 	mtcrf	0xff,r0
 
@@ -339,6 +365,9 @@ _GLOBAL(plpar_hcall9_raw)
 	std	r11,56(r12)
 	std	r0, 64(r12)
 
+	li	r4,0
+	stb	r4,PACASRR_VALID(r13)
+
 	lwz	r0,8(r1)
 	mtcrf	0xff,r0
 
arch/powerpc/sysdev/fsl_pci.c +1 -1
index 69af73765783..b8f76f3fd994 100644
@@ -1072,7 +1072,7 @@ int fsl_pci_mcheck_exception(struct pt_regs *regs)
 			ret = get_kernel_nofault(inst, (void *)regs->nip);
 
 		if (!ret && mcheck_handle_load(regs, inst)) {
-			regs->nip += 4;
+			regs_add_return_ip(regs, 4);
 			return 1;
 		}
 	}
Signature · 1 line
2.23.0
author reviewer
#5 lore raw

PATCH v3 4/11 powerpc/64: handle MSR EE and RI in interrupt entry wrapper

In reply to Nicholas Piggin

Similarly to the system call change in the previous patch, the mtmsrd to enable RI can be combined with the mtmsrd to enable EE for interrupts which enable the latter, which tends to be the important synchronous interrupts (i.e., page faults).

Do this by enabling EE and RI together at the beginning of the entry wrapper if PACA_IRQ_HARD_DIS is clear, and just enabling RI if it is set (which means something wanted EE=0).

Asynchronous interrupts set PACA_IRQ_HARD_DIS, but synchronous ones leave it unchanged, so by default they always get EE=1 unless they interrupt a caller that has hard disabled. When the sync interrupt later calls interrupt_cond_local_irq_enable(), that will not require another mtmsrd because we already enabled here.

This tends to save one mtmsrd L=1 for synchronous interrupts on 64s. 64e is conceptually unchanged, but it also sets MSR[EE]=1 now in the interrupt wrapper for synchronous interrupts with the same code.

From: Nicholas Piggin <[email protected]>

---
 arch/powerpc/include/asm/interrupt.h | 22 ++++++++++++++++++--
 arch/powerpc/kernel/exceptions-64s.S | 30 ----------------------------
 2 files changed, 20 insertions(+), 32 deletions(-)
arch/powerpc/include/asm/interrupt.h +20 -2
index 6e9d18838d56..b9c510187b58 100644
@@ -126,9 +126,21 @@ static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrup
 #endif
 
 #ifdef CONFIG_PPC64
-	if (irq_soft_mask_set_return(IRQS_ALL_DISABLED) == IRQS_ENABLED)
+	bool trace_enable = false;
+
+	if (IS_ENABLED(CONFIG_TRACE_IRQFLAGS)) {
+		if (irq_soft_mask_set_return(IRQS_DISABLED) == IRQS_ENABLED)
+			trace_enable = true;
+	} else {
+		irq_soft_mask_set(IRQS_DISABLED);
+	}
+	/* If the interrupt was taken with HARD_DIS set, don't enable MSR[EE] */
+	if (local_paca->irq_happened & PACA_IRQ_HARD_DIS)
+		__hard_RI_enable();
+	else
+		__hard_irq_enable();
+	if (trace_enable)
 		trace_hardirqs_off();
-	local_paca->irq_happened |= PACA_IRQ_HARD_DIS;
 
 	if (user_mode(regs)) {
 		CT_WARN_ON(ct_state() != CONTEXT_USER);
@@ -175,6 +187,10 @@ static inline void interrupt_async_enter_prepare(struct pt_regs *regs, struct in
 		__ppc64_runlatch_on();
 #endif
 
+#ifdef CONFIG_PPC64
+	/* Ensure interrupt_enter_prepare does not enable MSR[EE] */
+	local_paca->irq_happened |= PACA_IRQ_HARD_DIS;
+#endif
 	interrupt_enter_prepare(regs, state);
 	irq_enter();
 }
@@ -239,6 +255,8 @@ static inline void interrupt_nmi_enter_prepare(struct pt_regs *regs, struct inte
 		regs->softe = IRQS_ALL_DISABLED;
 	}
 
+	__hard_RI_enable();
+
 	/* Don't do any per-CPU operations until interrupt state is fixed */
 
 	if (nmi_disables_ftrace(regs)) {
arch/powerpc/kernel/exceptions-64s.S +0 -30
index 5c18a2a3058d..a2ae14d0600e 100644
@@ -129,7 +129,6 @@ name:
 #define IISIDE		.L_IISIDE_\name\()	/* Uses SRR0/1 not DAR/DSISR */
 #define IDAR		.L_IDAR_\name\()	/* Uses DAR (or SRR0) */
 #define IDSISR		.L_IDSISR_\name\()	/* Uses DSISR (or SRR1) */
-#define ISET_RI		.L_ISET_RI_\name\()	/* Run common code w/ MSR[RI]=1 */
 #define IBRANCH_TO_COMMON	.L_IBRANCH_TO_COMMON_\name\() /* ENTRY branch to common */
 #define IREALMODE_COMMON	.L_IREALMODE_COMMON_\name\() /* Common runs in realmode */
 #define IMASK		.L_IMASK_\name\()	/* IRQ soft-mask bit */
@@ -174,9 +173,6 @@ do_define_int n
 	.ifndef IDSISR
 		IDSISR=0
 	.endif
-	.ifndef ISET_RI
-		ISET_RI=1
-	.endif
 	.ifndef IBRANCH_TO_COMMON
 		IBRANCH_TO_COMMON=1
 	.endif
@@ -581,11 +577,6 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real)
 	stb	r10,PACASRR_VALID(r13)
 	.endif
 
-	.if ISET_RI
-	li	r10,MSR_RI
-	mtmsrd	r10,1			/* Set MSR_RI */
-	.endif
-
 	.if ISTACK
 	.if IKUAP
 	kuap_save_amr_and_lock r9, r10, cr1, cr0
@@ -906,11 +897,6 @@ INT_DEFINE_BEGIN(system_reset)
 	IVEC=0x100
 	IAREA=PACA_EXNMI
 	IVIRT=0 /* no virt entry point */
-	/*
-	 * MSR_RI is not enabled, because PACA_EXNMI and nmi stack is
-	 * being used, so a nested NMI exception would corrupt it.
-	 */
-	ISET_RI=0
 	ISTACK=0
 	IKVM_REAL=1
 INT_DEFINE_END(system_reset)
@@ -991,8 +977,6 @@ EXC_COMMON_BEGIN(system_reset_common)
 	lhz	r10,PACA_IN_NMI(r13)
 	addi	r10,r10,1
 	sth	r10,PACA_IN_NMI(r13)
-	li	r10,MSR_RI
-	mtmsrd 	r10,1
 
 	mr	r10,r1
 	ld	r1,PACA_NMI_EMERG_SP(r13)
@@ -1068,12 +1052,6 @@ INT_DEFINE_BEGIN(machine_check_early)
 	IAREA=PACA_EXMC
 	IVIRT=0 /* no virt entry point */
 	IREALMODE_COMMON=1
-	/*
-	 * MSR_RI is not enabled, because PACA_EXMC is being used, so a
-	 * nested machine check corrupts it. machine_check_common enables
-	 * MSR_RI.
-	 */
-	ISET_RI=0
 	ISTACK=0
 	IDAR=1
 	IDSISR=1
@@ -1084,7 +1062,6 @@ INT_DEFINE_BEGIN(machine_check)
 	IVEC=0x200
 	IAREA=PACA_EXMC
 	IVIRT=0 /* no virt entry point */
-	ISET_RI=0
 	IDAR=1
 	IDSISR=1
 	IKVM_SKIP=1
@@ -1155,9 +1132,6 @@ EXC_COMMON_BEGIN(machine_check_early_common)
 BEGIN_FTR_SECTION
 	bl	enable_machine_check
 END_FTR_SECTION_IFSET(CPU_FTR_HVMODE)
-	li	r10,MSR_RI
-	mtmsrd	r10,1
-
 	addi	r3,r1,STACK_FRAME_OVERHEAD
 	bl	machine_check_early
 	std	r3,RESULT(r1)	/* Save result */
@@ -1245,10 +1219,6 @@ EXC_COMMON_BEGIN(machine_check_common)
 	 * save area: PACA_EXMC instead of PACA_EXGEN.
 	 */
 	GEN_COMMON machine_check
-
-	/* Enable MSR_RI when finished with PACA_EXMC */
-	li	r10,MSR_RI
-	mtmsrd 	r10,1
 	addi	r3,r1,STACK_FRAME_OVERHEAD
 	bl	machine_check_exception
 	b	interrupt_return_srr
Signature · 1 line
2.23.0
author reviewer
#6 lore raw

PATCH v3 5/11 powerpc/64: move interrupt return asm to interrupt_64.S

In reply to Nicholas Piggin

The next patch would like to move interrupt return assembly code to a low location before general text, so move it into its own file and include via head_64.S

---
 arch/powerpc/include/asm/head-64.h |   2 +-
 arch/powerpc/kernel/entry_64.S     | 623 ----------------------------
 arch/powerpc/kernel/head_64.S      |   5 +-
 arch/powerpc/kernel/interrupt_64.S | 636 +++++++++++++++++++++++++++++
 4 files changed, 641 insertions(+), 625 deletions(-)
 create mode 100644 arch/powerpc/kernel/interrupt_64.S
arch/powerpc/include/asm/head-64.h +1 -1
index 4cb9efa2eb21..242204e12993 100644
@@ -16,7 +16,7 @@
 	.section ".head.data.\name\()","a",@progbits
 .endm
 .macro use_ftsec name
-	.section ".head.text.\name\()"
+	.section ".head.text.\name\()","ax",@progbits
 .endm
 
 /*
arch/powerpc/kernel/entry_64.S +0 -623

642 lines not on this page · raw message

arch/powerpc/kernel/head_64.S +4 -1
index ece7f97bafff..d49c25daf1c0 100644
@@ -194,8 +194,9 @@ CLOSE_FIXED_SECTION(first_256B)
 
 /* This value is used to mark exception frames on the stack. */
 	.section ".toc","aw"
+/* This value is used to mark exception frames on the stack. */
 exception_marker:
-	.tc	ID_72656773_68657265[TC],0x7265677368657265
+	.tc	ID_EXC_MARKER[TC],STACK_FRAME_REGS_MARKER
 	.previous
 
 /*
@@ -211,6 +212,8 @@ OPEN_TEXT_SECTION(0x100)
 
 USE_TEXT_SECTION()
 
+#include "interrupt_64.S"
+
 #ifdef CONFIG_PPC_BOOK3E
 /*
  * The booting_thread_hwid holds the thread id we want to boot in cpu
arch/powerpc/kernel/interrupt_64.S new file +636 -0

636 lines not on this page · raw message

Signature · 1 line
2.23.0
author reviewer
#7 lore raw

PATCH v3 6/11 powerpc/64s: system call avoid setting MSR[RI] until we set MSR[EE]

In reply to Nicholas Piggin

This extends the MSR[RI]=0 window a little further into the system call in order to pair RI and EE enabling with a single mtmsrd.

---
 arch/powerpc/kernel/exceptions-64s.S | 2 --
 arch/powerpc/kernel/interrupt_64.S   | 4 ++--
 2 files changed, 2 insertions(+), 4 deletions(-)
arch/powerpc/kernel/exceptions-64s.S +0 -2
index a2ae14d0600e..b6e1c46c97d0 100644
@@ -1942,8 +1942,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_REAL_LE)
 	mtctr	r10
 	bctr
 	.else
-	li	r10,MSR_RI
-	mtmsrd 	r10,1			/* Set RI (EE=0) */
 #ifdef CONFIG_RELOCATABLE
 	__LOAD_HANDLER(r10, system_call_common)
 	mtctr	r10
arch/powerpc/kernel/interrupt_64.S +2 -2
index 83826775d239..3038c831fc5f 100644
@@ -284,9 +284,9 @@ END_BTB_FLUSH_SECTION
 	 * trace_hardirqs_off().
 	 */
 	li	r11,IRQS_ALL_DISABLED
-	li	r12,PACA_IRQ_HARD_DIS
+	li	r12,-1 /* Set MSR_EE and MSR_RI */
 	stb	r11,PACAIRQSOFTMASK(r13)
-	stb	r12,PACAIRQHAPPENED(r13)
+	mtmsrd	r12,1
 
 	/* Calling convention has r9 = orig r0, r10 = regs */
 	mr	r9,r0
Signature · 1 line
2.23.0
author reviewer
#8 lore raw

PATCH v3 7/11 powerpc/64s: save one more register in the masked interrupt handler

In reply to Nicholas Piggin

This frees up one more register (and takes advantage of that to clean things up a little bit).

This register will be used in the following patch.

---
 arch/powerpc/kernel/exceptions-64s.S | 34 ++++++++++++++++------------
 1 file changed, 20 insertions(+), 14 deletions(-)
arch/powerpc/kernel/exceptions-64s.S +20 -14
index b6e1c46c97d0..0ba8c2387aac 100644
@@ -2758,7 +2758,6 @@ INT_DEFINE_END(soft_nmi)
  * and run it entirely with interrupts hard disabled.
  */
 EXC_COMMON_BEGIN(soft_nmi_common)
-	mfspr	r11,SPRN_SRR0
 	mr	r10,r1
 	ld	r1,PACAEMERGSP(r13)
 	subi	r1,r1,INT_FRAME_SIZE
@@ -2793,19 +2792,24 @@ masked_Hinterrupt:
 	.else
 masked_interrupt:
 	.endif
-	lbz	r11,PACAIRQHAPPENED(r13)
-	or	r11,r11,r10
-	stb	r11,PACAIRQHAPPENED(r13)
+	stw	r9,PACA_EXGEN+EX_CCR(r13)
+	lbz	r9,PACAIRQHAPPENED(r13)
+	or	r9,r9,r10
+	stb	r9,PACAIRQHAPPENED(r13)
+
+	.if ! \hsrr
 	cmpwi	r10,PACA_IRQ_DEC
 	bne	1f
-	lis	r10,0x7fff
-	ori	r10,r10,0xffff
-	mtspr	SPRN_DEC,r10
+	LOAD_REG_IMMEDIATE(r9, 0x7fffffff)
+	mtspr	SPRN_DEC,r9
 #ifdef CONFIG_PPC_WATCHDOG
+	lwz	r9,PACA_EXGEN+EX_CCR(r13)
 	b	soft_nmi_common
 #else
 	b	2f
 #endif
+	.endif
+
 1:	andi.	r10,r10,PACA_IRQ_MUST_HARD_MASK
 	beq	2f
 	xori	r12,r12,MSR_EE	/* clear MSR_EE */
@@ -2814,17 +2818,19 @@ masked_interrupt:
 	.else
 	mtspr	SPRN_SRR1,r12
 	.endif
-	ori	r11,r11,PACA_IRQ_HARD_DIS
-	stb	r11,PACAIRQHAPPENED(r13)
+	ori	r9,r9,PACA_IRQ_HARD_DIS
+	stb	r9,PACAIRQHAPPENED(r13)
 2:	/* done */
-	li	r10,0
+	li	r9,0
 	.if \hsrr
-	stb	r10,PACAHSRR_VALID(r13)
+	stb	r9,PACAHSRR_VALID(r13)
 	.else
-	stb	r10,PACASRR_VALID(r13)
+	stb	r9,PACASRR_VALID(r13)
 	.endif
-	ld	r10,PACA_EXGEN+EX_CTR(r13)
-	mtctr	r10
+
+	ld	r9,PACA_EXGEN+EX_CTR(r13)
+	mtctr	r9
+	lwz	r9,PACA_EXGEN+EX_CCR(r13)
 	mtcrf	0x80,r9
 	std	r1,PACAR1(r13)
 	ld	r9,PACA_EXGEN+EX_R9(r13)
Signature · 1 line
2.23.0
author reviewer
#9 lore raw

PATCH v3 8/11 powerpc/64: allow alternate return locations for soft-masked interrupts

In reply to Nicholas Piggin

The exception table fixup adjusts a failed page fault's interrupt return location if it was taken at an address specified in the exception table, to a corresponding fixup handler address.

Introduce a variation of that idea which adds a fixup table for NMIs and soft-masked asynchronous interrupts. This will be used to protect certain critical sections that are sensitive to being clobbered by interrupts coming in (due to using the same SPRs and/or irq soft-mask state).

---
 arch/powerpc/include/asm/interrupt.h | 13 +++++++++
 arch/powerpc/include/asm/ppc_asm.h   |  8 ++++++
 arch/powerpc/kernel/exceptions-64e.S | 37 +++++++++++++++++++++++--
 arch/powerpc/kernel/exceptions-64s.S | 41 ++++++++++++++++++++++++++++
 arch/powerpc/kernel/vmlinux.lds.S    | 10 +++++++
 arch/powerpc/lib/Makefile            |  2 +-
 arch/powerpc/lib/restart_table.c     | 29 ++++++++++++++++++++
 7 files changed, 137 insertions(+), 3 deletions(-)
 create mode 100644 arch/powerpc/lib/restart_table.c
arch/powerpc/include/asm/interrupt.h +13 -0
index b9c510187b58..49d9a6fd1bb9 100644
@@ -73,6 +73,11 @@
 #include <asm/kprobes.h>
 #include <asm/runlatch.h>
 
+#ifdef CONFIG_PPC64
+extern char __end_soft_masked[];
+unsigned long search_kernel_restart_table(unsigned long addr);
+#endif
+
 #ifdef CONFIG_PPC_BOOK3S_64
 static inline void srr_regs_clobbered(void)
 {
@@ -287,6 +292,14 @@ static inline void interrupt_nmi_exit_prepare(struct pt_regs *regs, struct inter
 	 * new work to do (must use irq_work for that).
 	 */
 
+#ifdef CONFIG_PPC64
+	if (arch_irq_disabled_regs(regs)) {
+		unsigned long rst = search_kernel_restart_table(regs->nip);
+		if (rst)
+			regs_set_return_ip(regs, rst);
+	}
+#endif
+
 #ifdef CONFIG_PPC64
 	if (nmi_disables_ftrace(regs))
 		this_cpu_set_ftrace_enabled(state->ftrace_enabled);
arch/powerpc/include/asm/ppc_asm.h +8 -0
index d6739d700f0a..c9c2c36c1f8f 100644
@@ -762,6 +762,14 @@ END_FTR_SECTION_NESTED(CPU_FTR_CELL_TB_BUG, CPU_FTR_CELL_TB_BUG, 96)
 	stringify_in_c(.long (_target) - . ;)	\
 	stringify_in_c(.previous)
 
+#define RESTART_TABLE(_start, _end, _target)	\
+	stringify_in_c(.section __restart_table,"a";)\
+	stringify_in_c(.balign 8;)		\
+	stringify_in_c(.llong (_start);)	\
+	stringify_in_c(.llong (_end);)		\
+	stringify_in_c(.llong (_target);)	\
+	stringify_in_c(.previous)
+
 #ifdef CONFIG_PPC_FSL_BOOK3E
 #define BTB_FLUSH(reg)			\
 	lis reg,BUCSR_INIT@h;		\
arch/powerpc/kernel/exceptions-64e.S +35 -2
index b35c97c7082f..1b79f8a75298 100644
@@ -901,6 +901,28 @@ kernel_dbg_exc:
 	bl	unknown_exception
 	b	interrupt_return
 
+.macro SEARCH_RESTART_TABLE
+	LOAD_REG_IMMEDIATE_SYM(r14, r11, __start___restart_table)
+	LOAD_REG_IMMEDIATE_SYM(r15, r11, __stop___restart_table)
+300:
+	cmpd	r14,r15
+	beq	302f
+	ld	r11,0(r14)
+	cmpld	r10,r11
+	blt	301f
+	ld	r11,8(r14)
+	cmpld	r10,r11
+	bge	301f
+	ld	r11,16(r14)
+	b	303f
+301:
+	addi	r14,r14,24
+	b	300b
+302:
+	li	r11,0
+303:
+.endm
+
 /*
  * An interrupt came in while soft-disabled; We mark paca->irq_happened
  * accordingly and if the interrupt is level sensitive, we hard disable
@@ -909,6 +931,9 @@ kernel_dbg_exc:
  */
 
 .macro masked_interrupt_book3e paca_irq full_mask
+	std	r14,PACA_EXGEN+EX_R14(r13)
+	std	r15,PACA_EXGEN+EX_R15(r13)
+
 	lbz	r10,PACAIRQHAPPENED(r13)
 	.if \full_mask == 1
 	ori	r10,r10,\paca_irq | PACA_IRQ_HARD_DIS
@@ -918,15 +943,23 @@ kernel_dbg_exc:
 	stb	r10,PACAIRQHAPPENED(r13)
 
 	.if \full_mask == 1
-	rldicl	r10,r11,48,1		/* clear MSR_EE */
-	rotldi	r11,r10,16
+	xori	r11,r11,MSR_EE		/* clear MSR_EE */
 	mtspr	SPRN_SRR1,r11
 	.endif
 
+	mfspr	r10,SPRN_SRR0
+	SEARCH_RESTART_TABLE
+	cmpdi	r11,0
+	beq	1f
+	mtspr	SPRN_SRR0,r11		/* return to restart address */
+1:
+
 	lwz	r11,PACA_EXGEN+EX_CR(r13)
 	mtcr	r11
 	ld	r10,PACA_EXGEN+EX_R10(r13)
 	ld	r11,PACA_EXGEN+EX_R11(r13)
+	ld	r14,PACA_EXGEN+EX_R14(r13)
+	ld	r15,PACA_EXGEN+EX_R15(r13)
 	mfspr	r13,SPRN_SPRG_GEN_SCRATCH
 	rfi
 	b	.
arch/powerpc/kernel/exceptions-64s.S +41 -0

59 lines not on this page · raw message

arch/powerpc/kernel/vmlinux.lds.S +10 -0
index 72fa3c00229a..16c5e13e00c4 100644
@@ -9,6 +9,14 @@
 #define EMITS_PT_NOTE
 #define RO_EXCEPTION_TABLE_ALIGN	0
 
+#define RESTART_TABLE(align)						\
+	. = ALIGN(align);						\
+	__restart_table : AT(ADDR(__restart_table) - LOAD_OFFSET) {	\
+		__start___restart_table = .;				\
+		KEEP(*(__restart_table))				\
+		__stop___restart_table = .;				\
+	}
+
 #include <asm/page.h>
 #include <asm-generic/vmlinux.lds.h>
 #include <asm/cache.h>
@@ -124,6 +132,8 @@ SECTIONS
 	RO_DATA(PAGE_SIZE)
 
 #ifdef CONFIG_PPC64
+	RESTART_TABLE(8)
+
 	. = ALIGN(8);
 	__stf_entry_barrier_fixup : AT(ADDR(__stf_entry_barrier_fixup) - LOAD_OFFSET) {
 		__start___stf_entry_barrier_fixup = .;
arch/powerpc/lib/Makefile +1 -1
index cc1a8a0f311e..4c92c80454f3 100644
@@ -42,7 +42,7 @@ obj-$(CONFIG_PPC_BOOK3S_64) += copyuser_power7.o copypage_power7.o \
 			       memcpy_power7.o
 
 obj64-y	+= copypage_64.o copyuser_64.o mem_64.o hweight_64.o \
-	   memcpy_64.o copy_mc_64.o
+	   memcpy_64.o copy_mc_64.o restart_table.o
 
 ifndef CONFIG_PPC_QUEUED_SPINLOCKS
 obj64-$(CONFIG_SMP)	+= locks.o
arch/powerpc/lib/restart_table.c new file +29 -0
new file mode 100644
index 000000000000..3ccb31914036
@@ -0,0 +1,29 @@
+#include <asm/kprobes.h>
+
+struct restart_table_entry {
+	unsigned long start;
+	unsigned long end;
+	unsigned long fixup;
+};
+
+extern struct restart_table_entry __start___restart_table[];
+extern struct restart_table_entry __stop___restart_table[];
+
+/* Given an address, look for it in the kernel exception table */
+unsigned long search_kernel_restart_table(unsigned long addr)
+{
+	struct restart_table_entry *rte = __start___restart_table;
+
+	while (rte < __stop___restart_table) {
+		unsigned long start = rte->start;
+		unsigned long end = rte->end;
+		unsigned long fixup = rte->fixup;
+
+		if (addr >= start && addr < end)
+			return fixup;
+
+		rte++;
+	}
+	return 0;
+}
+NOKPROBE_SYMBOL(search_kernel_restart_table);
Signature · 1 line
2.23.0
author reviewer
#10 lore raw

PATCH v3 9/11 powerpc/64: interrupt soft-enable race fix

In reply to Nicholas Piggin

Prevent interrupt restore from allowing racing hard interrupts going ahead of previous soft-pending ones, by using the soft-masked restart handler to allow a store to clear the soft-mask while knowing nothing is soft-pending.

This probably doesn't matter much in practice, but it's a simple demonstrator / test case to exercise the restart table logic.

---
 arch/powerpc/kernel/irq.c | 95 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
arch/powerpc/kernel/irq.c +95 -0
index 72cb45393ef2..8428caf3194e 100644
@@ -217,6 +217,100 @@ static inline void replay_soft_interrupts_irqrestore(void)
 #define replay_soft_interrupts_irqrestore() replay_soft_interrupts()
 #endif
 
+#ifdef CONFIG_CC_HAS_ASM_GOTO
+notrace void arch_local_irq_restore(unsigned long mask)
+{
+	unsigned char irq_happened;
+
+	/* Write the new soft-enabled value if it is a disable */
+	if (mask) {
+		irq_soft_mask_set(mask);
+		return;
+	}
+
+	/*
+	 * After the stb, interrupts are unmasked and there are no interrupts
+	 * pending replay. The restart sequence makes this atomic with
+	 * respect to soft-masked interrupts. If this was just a simple code
+	 * sequence, a soft-masked interrupt could become pending right after
+	 * the comparison and before the stb.
+	 *
+	 * This allows interrupts to be unmasked without hard disabling, and
+	 * also without new hard interrupts coming in ahead of pending ones.
+	 */
+	asm_volatile_goto(
+"1:					\n"
+"		lbz	9,%0(13)	\n"
+"		cmpwi	9,0		\n"
+"		bne	%l[happened]	\n"
+"		stb	9,%1(13)	\n"
+"2:					\n"
+		RESTART_TABLE(1b, 2b, 1b)
+	: : "i" (offsetof(struct paca_struct, irq_happened)),
+	    "i" (offsetof(struct paca_struct, irq_soft_mask))
+	: "cr0", "r9"
+	: happened);
+
+	if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG))
+		WARN_ON_ONCE(!(mfmsr() & MSR_EE));
+
+	return;
+
+happened:
+	irq_happened = get_irq_happened();
+	if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG))
+		WARN_ON_ONCE(!irq_happened);
+
+	if (irq_happened == PACA_IRQ_HARD_DIS) {
+		if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG))
+			WARN_ON_ONCE(mfmsr() & MSR_EE);
+		irq_soft_mask_set(IRQS_ENABLED);
+		local_paca->irq_happened = 0;
+		__hard_irq_enable();
+		return;
+	}
+
+	/* Have interrupts to replay, need to hard disable first */
+	if (!(irq_happened & PACA_IRQ_HARD_DIS)) {
+		if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG)) {
+			if (!(mfmsr() & MSR_EE)) {
+				/*
+				 * An interrupt could have come in and cleared
+				 * MSR[EE] and set IRQ_HARD_DIS, so check
+				 * IRQ_HARD_DIS again and warn if it is still
+				 * clear.
+				 */
+				irq_happened = get_irq_happened();
+				WARN_ON_ONCE(!(irq_happened & PACA_IRQ_HARD_DIS));
+			}
+		}
+		__hard_irq_disable();
+		local_paca->irq_happened |= PACA_IRQ_HARD_DIS;
+	} else {
+		if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG)) {
+			if (WARN_ON_ONCE(mfmsr() & MSR_EE))
+				__hard_irq_disable();
+		}
+	}
+
+	/*
+	 * Disable preempt here, so that the below preempt_enable will
+	 * perform resched if required (a replayed interrupt may set
+	 * need_resched).
+	 */
+	preempt_disable();
+	irq_soft_mask_set(IRQS_ALL_DISABLED);
+	trace_hardirqs_off();
+
+	replay_soft_interrupts_irqrestore();
+	local_paca->irq_happened = 0;
+
+	trace_hardirqs_on();
+	irq_soft_mask_set(IRQS_ENABLED);
+	__hard_irq_enable();
+	preempt_enable();
+}
+#else
 notrace void arch_local_irq_restore(unsigned long mask)
 {
 	unsigned char irq_happened;
@@ -288,6 +382,7 @@ notrace void arch_local_irq_restore(unsigned long mask)
 	__hard_irq_enable();
 	preempt_enable();
 }
+#endif
 EXPORT_SYMBOL(arch_local_irq_restore);
 
 /*
Signature · 1 line
2.23.0
author reviewer
#11 lore raw

PATCH v3 10/11 powerpc/64: treat low kernel text as irqs soft-masked

In reply to Nicholas Piggin

Treat code below __end_soft_masked as soft-masked for the purpose of alternate return. 64s already mostly does this for scv entry.

This will be used to exit from interrupts without disabling MSR[EE].

---
 arch/powerpc/include/asm/interrupt.h | 11 ++++++++---
 arch/powerpc/kernel/exceptions-64e.S | 12 +++++++++++-
 arch/powerpc/kernel/exceptions-64s.S | 19 +++++++++++--------
 arch/powerpc/kernel/interrupt_64.S   |  6 +++++-
 4 files changed, 35 insertions(+), 13 deletions(-)
arch/powerpc/include/asm/interrupt.h +8 -3
index 49d9a6fd1bb9..88043e46442a 100644
@@ -158,8 +158,13 @@ static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrup
 		 * CT_WARN_ON comes here via program_check_exception,
 		 * so avoid recursion.
 		 */
-		if (TRAP(regs) != INTERRUPT_PROGRAM)
+		if (TRAP(regs) != INTERRUPT_PROGRAM) {
 			CT_WARN_ON(ct_state() != CONTEXT_KERNEL);
+			BUG_ON(regs->nip < (unsigned long)__end_soft_masked);
+		}
+		/* Move this under a debugging check */
+		if (arch_irq_disabled_regs(regs))
+			BUG_ON(search_kernel_restart_table(regs->nip));
 	}
 #endif
 
@@ -254,8 +259,8 @@ static inline void interrupt_nmi_enter_prepare(struct pt_regs *regs, struct inte
 	local_paca->irq_happened |= PACA_IRQ_HARD_DIS;
 
 	if (IS_ENABLED(CONFIG_PPC_BOOK3S_64) && !(regs->msr & MSR_PR) &&
-				regs->nip < (unsigned long)__end_interrupts) {
-		// Kernel code running below __end_interrupts is
+				regs->nip < (unsigned long)__end_soft_masked) {
+		// Kernel code running below __end_soft_masked is
 		// implicitly soft-masked.
 		regs->softe = IRQS_ALL_DISABLED;
 	}
arch/powerpc/kernel/exceptions-64e.S +11 -1
index 1b79f8a75298..22fcd95dd8dc 100644
@@ -342,7 +342,17 @@ ret_from_mc_except:
 #define PROLOG_ADDITION_MASKABLE_GEN(n)					    \
 	lbz	r10,PACAIRQSOFTMASK(r13);	/* are irqs soft-masked? */ \
 	andi.	r10,r10,IRQS_DISABLED;	/* yes -> go out of line */ \
-	bne	masked_interrupt_book3e_##n
+	bne	masked_interrupt_book3e_##n;				    \
+	/* Kernel code below __end_soft_masked is implicitly masked */	    \
+	andi.	r10,r11,MSR_PR;						    \
+	bne	1f;			/* user -> not masked */	    \
+	std	r14,PACA_EXGEN+EX_R14(r13);				    \
+	LOAD_REG_IMMEDIATE_SYM(r14, r10, __end_soft_masked);		    \
+	mfspr	r10,SPRN_SRR0;						    \
+	cmpld	r10,r14;						    \
+	ld	r14,PACA_EXGEN+EX_R14(r13);				    \
+	blt	masked_interrupt_book3e_##n;				    \
+1:
 
 /*
  * Additional regs must be re-loaded from paca before EXCEPTION_COMMON* is
arch/powerpc/kernel/exceptions-64s.S +11 -8
index 17a213f25c92..2d980addc88c 100644
@@ -508,10 +508,13 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real)
 		andi.	r10,r12,MSR_PR
 		bne	2f
 
-		/* Kernel code running below __end_interrupts is implicitly
-		 * soft-masked */
-		LOAD_HANDLER(r10, __end_interrupts)
+		/*
+		 * Kernel code running below __end_soft_masked is implicitly
+		 * soft-masked
+		 */
+		LOAD_HANDLER(r10, __end_soft_masked)
 		cmpld	r11,r10
+
 		li	r10,IMASK
 		blt-	1f
 
@@ -824,17 +827,17 @@ __start_interrupts:
  * scv instructions enter the kernel without changing EE, RI, ME, or HV.
  * In particular, this means we can take a maskable interrupt at any point
  * in the scv handler, which is unlike any other interrupt. This is solved
- * by treating the instruction addresses below __end_interrupts as being
+ * by treating the instruction addresses below __end_soft_masked as being
  * soft-masked.
  *
  * AIL-0 mode scv exceptions go to 0x17000-0x17fff, but we set AIL-3 and
  * ensure scv is never executed with relocation off, which means AIL-0
  * should never happen.
  *
- * Before leaving the below __end_interrupts text, at least of the following
- * must be true:
+ * Before leaving the following inside-__end_soft_masked text, at least of the
+ * following must be true:
  * - MSR[PR]=1 (i.e., return to userspace)
- * - MSR_EE|MSR_RI is set (no reentrant exceptions)
+ * - MSR_EE|MSR_RI is clear (no reentrant exceptions)
  * - Standard kernel environment is set up (stack, paca, etc)
  *
  * Call convention:
@@ -3099,7 +3102,7 @@ kvmppc_skip_Hinterrupt:
 
 USE_FIXED_SECTION(virt_trampolines)
 	/*
-	 * All code below __end_interrupts is treated as soft-masked. If
+	 * All code below __end_soft_masked is treated as soft-masked. If
 	 * any code runs here with MSR[EE]=1, it must then cope with pending
 	 * soft interrupt being raised (i.e., by ensuring it is replayed).
 	 *
arch/powerpc/kernel/interrupt_64.S +5 -1
index 3038c831fc5f..cf53293c8498 100644
@@ -633,4 +633,8 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_STCX_CHECKS_ADDRESS)
 interrupt_return_macro srr
 #ifdef CONFIG_PPC_BOOK3S
 interrupt_return_macro hsrr
-#endif
+#endif /* CONFIG_PPC_BOOK3S */
+
+	.globl __end_soft_masked
+__end_soft_masked:
+DEFINE_FIXED_SYMBOL(__end_soft_masked)
Signature · 1 line
2.23.0
author reviewer
#12 lore raw

PATCH v3 11/11 powerpc/64: use interrupt restart table to speed up return from interrupt

In reply to Nicholas Piggin

Use the restart table facility to return from interrupt or system calls without disabling MSR[EE] or MSR[RI].

Interrupt return asm is put into the low soft-masked region, to prevent interrupts being processed here, although they are still taken as masked interrupts which causes SRRs to be clobbered, and a pending soft-masked interrupt to require replaying.

The return code uses restart table regions to redirct to a fixup handler rather than continue with the exit, if such an interrupt happens. In this case the interrupt return is redirected to a fixup handler which reloads r1 for the interrupt stack and reloads registers and sets state up to replay the soft-masked interrupt and try the exit again.

Some types of security exit fallback flushes and barriers are currently unable to cope with reentrant interrupts, e.g., because they store some state in the scratch SPR which would be clobbered even by masked interrupts. For now the interrupts-enabled exits are disabled when these flushes are used.

---
 arch/powerpc/include/asm/hw_irq.h    |  13 +-
 arch/powerpc/include/asm/interrupt.h |   2 +
 arch/powerpc/include/asm/paca.h      |   3 +
 arch/powerpc/include/asm/ptrace.h    |   1 +
 arch/powerpc/kernel/asm-offsets.c    |   3 +
 arch/powerpc/kernel/interrupt.c      | 385 ++++++++++++++++++---------
 arch/powerpc/kernel/interrupt_64.S   | 131 ++++++++-
 arch/powerpc/lib/feature-fixups.c    |  38 ++-
 8 files changed, 430 insertions(+), 146 deletions(-)
arch/powerpc/include/asm/hw_irq.h +11 -2
index 19bcef666cf6..21cc571ea9c2 100644
@@ -18,8 +18,17 @@
  * PACA flags in paca->irq_happened.
  *
  * This bits are set when interrupts occur while soft-disabled
- * and allow a proper replay. Additionally, PACA_IRQ_HARD_DIS
- * is set whenever we manually hard disable.
+ * and allow a proper replay.
+ *
+ * The PACA_IRQ_HARD_DIS is set whenever we hard disable. It is almost
+ * always in synch with the MSR[EE] state, except:
+ * - A window in interrupt entry, where hardware disables MSR[EE] and that
+ *   must be "reconciled" with the soft mask state.
+ * - NMI interrupts that hit in awkward places, until they fix the state.
+ * - When local irqs are being enabled and state is being fixed up.
+ * - When returning from an interrupt there are some windows where this
+ *   can become out of synch, but gets fixed before the RFI or before
+ *   executing the next user instruction (see arch/powerpc/kernel/interrupt.c).
  */
 #define PACA_IRQ_HARD_DIS	0x01
 #define PACA_IRQ_DBELL		0x02
arch/powerpc/include/asm/interrupt.h +2 -0
index 88043e46442a..c7b4adbe2dfe 100644
@@ -79,6 +79,8 @@ unsigned long search_kernel_restart_table(unsigned long addr);
 #endif
 
 #ifdef CONFIG_PPC_BOOK3S_64
+DECLARE_STATIC_KEY_FALSE(interrupt_exit_not_reentrant);
+
 static inline void srr_regs_clobbered(void)
 {
 	local_paca->srr_valid = 0;
arch/powerpc/include/asm/paca.h +3 -0
index dfc984b0e640..ca0b52e107a8 100644
@@ -167,6 +167,9 @@ struct paca_struct {
 	u64 kstack;			/* Saved Kernel stack addr */
 	u64 saved_r1;			/* r1 save for RTAS calls or PM or EE=0 */
 	u64 saved_msr;			/* MSR saved here by enter_rtas */
+#ifdef CONFIG_PPC64
+	u64 exit_save_r1;		/* Syscall/interrupt R1 save */
+#endif
 #ifdef CONFIG_PPC_BOOK3E
 	u16 trap_save;			/* Used when bad stack is encountered */
 #endif
arch/powerpc/include/asm/ptrace.h +1 -0
index 516117bba4e6..cb2b093dee13 100644
@@ -52,6 +52,7 @@ struct pt_regs
 		struct {
 #ifdef CONFIG_PPC64
 			unsigned long ppr;
+			unsigned long exit_result;
 #endif
 			union {
 #ifdef CONFIG_PPC_KUAP
arch/powerpc/kernel/asm-offsets.c +3 -0
index 1c2b869d9d52..0dbf1c9e3fef 100644
@@ -286,6 +286,9 @@ int main(void)
 	OFFSET(PACAHWCPUID, paca_struct, hw_cpu_id);
 	OFFSET(PACAKEXECSTATE, paca_struct, kexec_state);
 	OFFSET(PACA_DSCR_DEFAULT, paca_struct, dscr_default);
+#ifdef CONFIG_PPC64
+	OFFSET(PACA_EXIT_SAVE_R1, paca_struct, exit_save_r1);
+#endif
 #ifdef CONFIG_PPC_BOOK3E
 	OFFSET(PACA_TRAP_SAVE, paca_struct, trap_save);
 #endif
arch/powerpc/kernel/interrupt.c +253 -132

457 lines not on this page · raw message

arch/powerpc/kernel/interrupt_64.S +120 -11

210 lines not on this page · raw message

arch/powerpc/lib/feature-fixups.c +37 -1
index fe26f2fa0f3f..fbe94e2d5011 100644
@@ -17,6 +17,7 @@
 #include <linux/stop_machine.h>
 #include <asm/cputable.h>
 #include <asm/code-patching.h>
+#include <asm/interrupt.h>
 #include <asm/page.h>
 #include <asm/sections.h>
 #include <asm/setup.h>
@@ -228,10 +229,27 @@ static void do_stf_exit_barrier_fixups(enum stf_barrier_type types)
 		                                           : "unknown");
 }
 
+static bool stf_exit_not_reentrant = false;
+static bool rfi_exit_not_reentrant = false;
+
+static void update_interrupt_exit(void)
+{
+	if (stf_exit_not_reentrant || rfi_exit_not_reentrant)
+		static_branch_enable(&interrupt_exit_not_reentrant);
+	else
+		static_branch_disable(&interrupt_exit_not_reentrant);
+}
+
 static int __do_stf_barrier_fixups(void *data)
 {
 	enum stf_barrier_type *types = data;
 
+	if (*types & STF_BARRIER_FALLBACK || *types & STF_BARRIER_SYNC_ORI)
+		stf_exit_not_reentrant = true;
+	else
+		stf_exit_not_reentrant = false;
+	update_interrupt_exit();
+
 	do_stf_entry_barrier_fixups(*types);
 	do_stf_exit_barrier_fixups(*types);
 
@@ -412,12 +430,19 @@ void do_entry_flush_fixups(enum l1d_flush_type types)
 	stop_machine(__do_entry_flush_fixups, &types, NULL);
 }
 
-void do_rfi_flush_fixups(enum l1d_flush_type types)
+static int __do_rfi_flush_fixups(void *data)
 {
+	enum l1d_flush_type types = *(enum l1d_flush_type *)data;
 	unsigned int instrs[3], *dest;
 	long *start, *end;
 	int i;
 
+	if (types & L1D_FLUSH_FALLBACK)
+		rfi_exit_not_reentrant = true;
+	else
+		rfi_exit_not_reentrant = false;
+	update_interrupt_exit();
+
 	start = PTRRELOC(&__start___rfi_flush_fixup);
 	end = PTRRELOC(&__stop___rfi_flush_fixup);
 
@@ -456,6 +481,17 @@ void do_rfi_flush_fixups(enum l1d_flush_type types)
 							: "ori type" :
 		(types &  L1D_FLUSH_MTTRIG)     ? "mttrig type"
 						: "unknown");
+
+	return 0;
+}
+
+void do_rfi_flush_fixups(enum l1d_flush_type types)
+{
+	/*
+	 * The interrupt_exit_not_reentrant static branch together with the
+	 * patching is not safe to perform while other CPUs are executing.
+	 */
+	stop_machine(__do_rfi_flush_fixups, &types, NULL);
 }
 
 void do_barrier_nospec_fixups_range(bool enable, void *fixup_start, void *fixup_end)
Signature · 1 line
2.23.0
#4 lore raw

Re: [PATCH v3 03/11] powerpc/64s: avoid reloading (H)SRR registers if they are still valid

In reply to Nicholas Piggin

arch/powerpc/kernel/fpu.S END_FTR_SECTION_IFSET(CPU_FTR_VSX) · line 110

Quoted 13 lines · Nicholas Piggin
arch/powerpc/kernel/fpu.S +2 -0
index 2c57ece6671c..44526c157bff 100644
@@ -103,6 +103,8 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX)
 	ori	r12,r12,MSR_FP
 	or	r12,r12,r4
 	std	r12,_MSR(r1)
+	li	r4,0
+	stb	r4,PACASRR_VALID(r13)
 #endif
 	li	r4,1
 	stb	r4,THREAD_LOAD_FP(r5)

This didn't build for 64e. I changed it to:

#ifdef CONFIG_PPC_BOOK3S_64
	li	r4,0
	stb	r4,PACASRR_VALID(r13)

#endif

And similarly for vector.S.

cheers

#13 lore raw

Re: [PATCH v3 11/11] powerpc/64: use interrupt restart table to speed up return from interrupt

In reply to Nicholas Piggin
Quoted 11 lines · Nicholas Piggin
arch/powerpc/lib/feature-fixups.c +1 -1
index fe26f2fa0f3f..fbe94e2d5011 100644
@@ -412,12 +430,19 @@ void do_entry_flush_fixups(enum l1d_flush_type types)
 	stop_machine(__do_entry_flush_fixups, &types, NULL);
 }
 
-void do_rfi_flush_fixups(enum l1d_flush_type types)
+static int __do_rfi_flush_fixups(void *data)
 {

arch/powerpc/lib/feature-fixups.c void do_entry_flush_fixups(enum l1d_flush_type types) · line 444

Quoted 10 lines
arch/powerpc/lib/feature-fixups.c +6 -0
+	enum l1d_flush_type types = *(enum l1d_flush_type *)data;
 	unsigned int instrs[3], *dest;
 	long *start, *end;
 	int i;
 
+	if (types & L1D_FLUSH_FALLBACK)
+		rfi_exit_not_reentrant = true;
+	else
+		rfi_exit_not_reentrant = false;
+	update_interrupt_exit();

This is not happy:

[    0.000000][    T0] ============================================
[    0.000000][    T0] WARNING: possible recursive locking detected
[    0.000000][    T0] 5.13.0-rc2-00118-gca433a3a44e3 #1 Not tainted
[    0.000000][    T0] --------------------------------------------
[    0.000000][    T0] swapper/0 is trying to acquire lock:
[    0.000000][    T0] c00000000252fa10 (cpu_hotplug_lock){....}-{0:0}, at: static_key_enable+0x24/0x50
[    0.000000][    T0]
[    0.000000][    T0] but task is already holding lock:
[    0.000000][    T0] c00000000252fa10 (cpu_hotplug_lock){....}-{0:0}, at: stop_machine+0x2c/0x60
[    0.000000][    T0]
[    0.000000][    T0] other info that might help us debug this:
[    0.000000][    T0]  Possible unsafe locking scenario:
[    0.000000][    T0]
[    0.000000][    T0]        CPU0
[    0.000000][    T0]        ----
[    0.000000][    T0]   lock(cpu_hotplug_lock);
[    0.000000][    T0]   lock(cpu_hotplug_lock);
[    0.000000][    T0]
[    0.000000][    T0]  *** DEADLOCK ***
[    0.000000][    T0]
[    0.000000][    T0]  May be due to missing lock nesting notation
[    0.000000][    T0]
[    0.000000][    T0] 1 lock held by swapper/0:
[    0.000000][    T0]  #0: c00000000252fa10 (cpu_hotplug_lock){....}-{0:0}, at: stop_machine+0x2c/0x60
[    0.000000][    T0]
[    0.000000][    T0] stack backtrace:
[    0.000000][    T0] CPU: 0 PID: 0 Comm: swapper Not tainted 5.13.0-rc2-00118-gca433a3a44e3 #1
[    0.000000][    T0] Call Trace:
[    0.000000][    T0] [c0000000027db8f0] [c00000000093dd28] dump_stack+0xec/0x144 (unreliable)
[    0.000000][    T0] [c0000000027db940] [c0000000001ed5b4] __lock_acquire+0x1744/0x28b0
[    0.000000][    T0] [c0000000027dba70] [c0000000001ef338] lock_acquire+0x128/0x600
[    0.000000][    T0] [c0000000027dbb70] [c00000000015035c] cpus_read_lock+0x4c/0x170
[    0.000000][    T0] [c0000000027dbba0] [c0000000003c2594] static_key_enable+0x24/0x50
[    0.000000][    T0] [c0000000027dbbd0] [c0000000000ae87c] __do_rfi_flush_fixups+0x7c/0x300
[    0.000000][    T0] [c0000000027dbc80] [c0000000002ab7e4] stop_machine_cpuslocked+0xe4/0x200
[    0.000000][    T0] [c0000000027dbcf0] [c0000000002ab940] stop_machine+0x40/0x60
[    0.000000][    T0] [c0000000027dbd30] [c0000000000aef30] do_rfi_flush_fixups+0x30/0x50
[    0.000000][    T0] [c0000000027dbd60] [c000000000040890] setup_rfi_flush+0xa0/0x140
[    0.000000][    T0] [c0000000027dbdd0] [c00000000201c6c4] pnv_setup_arch+0x304/0x4ac
[    0.000000][    T0] [c0000000027dbe60] [c00000000200a31c] setup_arch+0x374/0x3c4
[    0.000000][    T0] [c0000000027dbee0] [c000000002003d08] start_kernel+0xb0/0x790
[    0.000000][    T0] [c0000000027dbf90] [c00000000000d79c] start_here_common+0x1c/0x600
[    0.000000][    T0] rfi-flush: patched 12 locations (fallback displacement flush)

cheers

#15 lore raw
In reply to Nicholas Piggin
Quoted 2 lines · Nicholas Piggin

This series attempts to improve the speed of interrupts and system calls in three major ways.

With the full series applied I'm seeing various warnings.

I feel like I must be missing some lead up patch?

cheers

mambo, p8, hash, be:

[    0.001038][    T0] ------------[ cut here ]------------
[    0.001287][    T0] Interrupts were enabled early
[    0.001510][    T0] WARNING: CPU: 0 PID: 0 at init/main.c:1004 .start_kernel+0x53c/0x720
[    0.001902][    T0] Modules linked in:
[    0.002084][    T0] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.13.0-rc2-00118-gca433a3a44e3-dirty #1
[    0.002518][    T0] NIP:  c000000002004920 LR: c00000000200491c CTR: c0000000000e3cd0
[    0.002887][    T0] REGS: c000000002acfc30 TRAP: 0700   Not tainted  (5.13.0-rc2-00118-gca433a3a44e3-dirty)
[    0.003342][    T0] MSR:  9000000002029032 <SF,HV,VEC,EE,ME,IR,DR,RI>  CR: 28002222  XER: 22000000
[    0.003818][    T0] CFAR: 000000003002afb0 IRQMASK: 0
[    0.003818][    T0] GPR00: c00000000200491c c000000002acfed0 c000000002ad1a00 000000000000001d
[    0.003818][    T0] GPR04: 0000000000000001 c0000000027645b0 0000000000000001 c0000000027645e0
[    0.003818][    T0] GPR08: 00000000fd5c0000 c000000002764560 c000000002764560 9000000032001032
[    0.003818][    T0] GPR12: 0000000028002222 c000000002cd0000 0000000000000000 0000000000000000
[    0.003818][    T0] GPR16: 0000000031c10000 0000000030003180 0000000030000000 0000000000000000
[    0.003818][    T0] GPR20: 0000000000000000 0000000000000000 000000003012e4eb 0000000030110ece
[    0.003818][    T0] GPR24: 00000000301105f5 c000000000000000 0000000020010000 c000000002090008
[    0.003818][    T0] GPR28: c000000002090008 c0000000011ae620 c0000000011ae620 c000000002090008
[    0.007455][    T0] NIP [c000000002004920] .start_kernel+0x53c/0x720
[    0.007765][    T0] LR [c00000000200491c] .start_kernel+0x538/0x720
[    0.008072][    T0] Call Trace:
[    0.008223][    T0] [c000000002acfed0] [c00000000200491c] .start_kernel+0x538/0x720 (unreliable)
[    0.008655][    T0] [c000000002acff90] [c00000000000dd2c] start_here_common+0x1c/0x670
[    0.009041][    T0] Instruction dump:
[    0.009219][    T0] 60000000 4a23dadd 60000000 48035821 60000000 892d0932 71290001 40820018
[    0.009659][    T0] 3c62fe6d 3863f810 4a171751 60000000 <0fe00000> 39200000 3d420003 38600000
[    0.010111][    T0] ---[ end trace e2dce2fbc72d04fa ]---

mambo, p9, hash, be:

[   17.668403][  T267] ------------[ cut here ]------------
[   17.668433][  T267] WARNING: CPU: 1 PID: 267 at arch/powerpc/include/asm/book3s/64/kup.h:304 .arch_local_irq_restore+0x190/0x1a0
[   17.668503][  T267] Modules linked in:
[   17.668532][  T267] CPU: 1 PID: 267 Comm: halt Tainted: G        W         5.13.0-rc2-00118-gca433a3a44e3-dirty #1
[   17.668588][  T267] NIP:  c000000000016680 LR: c00000000002f804 CTR: c0000000005901c0
[   17.668632][  T267] REGS: c000000009f339b0 TRAP: 0700   Tainted: G        W          (5.13.0-rc2-00118-gca433a3a44e3-dirty)
[   17.668686][  T267] MSR:  9000000000021032 <SF,HV,ME,IR,DR,RI>  CR: 48244222  XER: 04000000
[   17.668777][  T267] CFAR: c00000000002fdcc IRQMASK: 3
[   17.668777][  T267] GPR00: c00000000002f804 c000000009f33c50 c000000002ad1a00 0000000000000000
[   17.668777][  T267] GPR04: 0000000000000000 0000000000000020 00000000000000b7 00000000000c9495
[   17.668777][  T267] GPR08: 0000000000000008 fcffffffffffffff 3cffffffffffffff 0000000000000003
[   17.668777][  T267] GPR12: 0000000028244228 c0000000fffff300 0000000000000000 0000000000000000
[   17.668777][  T267] GPR16: 0000000000000000 0000000000000000 0000000000000000 00000000100a0a18
[   17.668777][  T267] GPR20: 00007fffdf62ffac 00000000100a09d8 00000100011405f0 0000000002002000
[   17.668777][  T267] GPR24: 0000000000000002 0000000000000001 0000000002802000 0000000000000000
[   17.668777][  T267] GPR28: c00000000cc56080 0000000000000000 0000000000000000 3cffffffffffffff
[   17.669291][  T267] NIP [c000000000016680] .arch_local_irq_restore+0x190/0x1a0
[   17.669343][  T267] LR [c00000000002f804] .interrupt_exit_user_prepare_main+0x144/0x2b0
[   17.669394][  T267] Call Trace:
[   17.669418][  T267] [c000000009f33c50] [c000000009f33ce0] 0xc000000009f33ce0 (unreliable)
[   17.669475][  T267] [c000000009f33cd0] [c00000000002f804] .interrupt_exit_user_prepare_main+0x144/0x2b0
[   17.669538][  T267] [c000000009f33d90] [c00000000002fe08] .syscall_exit_restart+0x68/0x80
[   17.669596][  T267] [c000000009f33e10] [c00000000000d168] system_call_common+0x278/0x2dc
[   17.669652][  T267] --- interrupt: c00 at 0x7fffad5b867c
[   17.669688][  T267] NIP:  00007fffad5b867c LR: 00000000100992a0 CTR: 0000000000000000
[   17.669731][  T267] REGS: c000000009f33e80 TRAP: 0c00   Tainted: G        W          (5.13.0-rc2-00118-gca433a3a44e3-dirty)
[   17.669785][  T267] MSR:  900000000280f032 <SF,HV,VEC,VSX,EE,PR,FP,ME,IR,DR,RI>  CR: 28244222  XER: 00000000
[   17.669910][  T267] IRQMASK: 0
[   17.669910][  T267] GPR00: 0000000000000006 00007fffd8758060 00007fffad687300 0000000000000000
[   17.669910][  T267] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   17.669910][  T267] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   17.669910][  T267] GPR12: 0000000000000000 00007fffad70a3a0 0000000000000000 0000000000000000
[   17.669910][  T267] GPR16: 0000000000000000 0000000000000000 0000000000000000 00000000100a0a18
[   17.669910][  T267] GPR20: 00007fffdf62ffac 00000000100a09d8 00000100011405f0 00000000100e24d0
[   17.669910][  T267] GPR24: 00000000100b91e5 00000000100c1e14 0000010019e002a0 00000000100c1e1e
[   17.669910][  T267] GPR28: 00007fffd875821b 0000000000000004 00007fffd8758210 0000000000000000
[   17.670410][  T267] NIP [00007fffad5b867c] 0x7fffad5b867c
[   17.670446][  T267] LR [00000000100992a0] 0x100992a0
[   17.670480][  T267] --- interrupt: c00
[   17.670507][  T267] Instruction dump:
[   17.670535][  T267] 4e800020 60000000 60000000 60000000 39200000 992d0932 39400000 992d0933
[   17.670627][  T267] 614a8002 7d410164 4e800020 60000000 <0fe00000> 4bfffefc 60000000 60000000
[   17.670719][  T267] ---[ end trace 5ba1b854b2957ce3 ]---

qemu powernv:

[    3.446262][   T51] init[51]: segfault (11) at 0 nip 0 lr 1000ca78 code 1 in busybox[10000000+d0000]
[    3.448351][   T51] init[51]: code: XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
[    3.449598][   T51] init[51]: code: XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
[    3.460559][   T52] init[52]: segfault (11) at 0 nip 0 lr 1000ca78 code 1 in busybox[10000000+d0000]
[    3.461409][   T52] init[52]: code: XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
[    3.462496][   T52] init[52]: code: XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX

... to infinity.

qemu 64e gets to running /init but gets stuck there with no further output.

qemu pseries, p8, be:

[    3.836653][   T75] S20urandom (75) used greatest stack depth: 9792 bytes left
[    4.718806][    C0] ------------[ cut here ]------------
[    4.718971][    C0] irq 17 handler .vp_interrupt+0x0/0xc0 enabled interrupts
[    4.720956][    C0] WARNING: CPU: 0 PID: 99 at kernel/irq/handle.c:159 .__handle_irq_event_percpu+0x230/0x270
[    4.721113][    C0] Modules linked in:
[    4.721502][    C0] CPU: 0 PID: 99 Comm: ip Not tainted 5.13.0-rc2-00118-gca433a3a44e3-dirty #1
[    4.721861][    C0] NIP:  c0000000002108b0 LR: c0000000002108ac CTR: c0000000009bb220
[    4.721943][    C0] REGS: c0000000fffeba00 TRAP: 0700   Not tainted  (5.13.0-rc2-00118-gca433a3a44e3-dirty)
[    4.722081][    C0] MSR:  8000000000029032 <SF,EE,ME,IR,DR,RI>  CR: 28002247  XER: 20000000
[    4.722404][    C0] CFAR: c0000000001760d0 IRQMASK: 0
[    4.722404][    C0] GPR00: c0000000002108ac c0000000fffebca0 c000000002ad1a00 0000000000000038
[    4.722404][    C0] GPR04: 0000000000000001 c0000000027645b0 0000000000000027 c0000000ff906e10
[    4.722404][    C0] GPR08: 0000000000000023 ffffffffffffffd8 0000000000000027 c0000000011d7b40
[    4.722404][    C0] GPR12: 0000000028002247 c000000002cd0000 0000000000000000 0000000000000000
[    4.722404][    C0] GPR16: 0000000000000000 0000000000000000 0000000000000000 00000000100b7866
[    4.722404][    C0] GPR20: 00000000100b7933 00000000100b5562 0000000000000000 0000000000000001
[    4.722404][    C0] GPR24: 0000000000000002 0000000000000001 0000000000000011 c000000008891800
[    4.722404][    C0] GPR28: 0000000000000011 c0000000fffebdd4 c00000000814a880 0000000000000000
[    4.723412][    C0] NIP [c0000000002108b0] .__handle_irq_event_percpu+0x230/0x270
[    4.723497][    C0] LR [c0000000002108ac] .__handle_irq_event_percpu+0x22c/0x270
[    4.723719][    C0] Call Trace:
[    4.723907][    C0] [c0000000fffebca0] [c0000000002108ac] .__handle_irq_event_percpu+0x22c/0x270 (unreliable)
[    4.724205][    C0] [c0000000fffebd60] [c000000000210ae0] .handle_irq_event+0x130/0x190
[    4.724309][    C0] [c0000000fffebe00] [c0000000002182bc] .handle_fasteoi_irq+0xbc/0x310
[    4.724397][    C0] [c0000000fffebea0] [c00000000020ea74] .generic_handle_irq+0x54/0x80
[    4.724575][    C0] [c0000000fffebf10] [c000000000015770] .__do_irq+0x70/0x1c0
[    4.724667][    C0] [c0000000fffebf90] [c0000000000159d4] .do_IRQ+0x114/0x250
[    4.724764][    C0] [c00000000a0730c0] [c00000000001596c] .do_IRQ+0xac/0x250
[    4.724848][    C0] [c00000000a073160] [c000000000009230] hardware_interrupt_common_virt+0x1b0/0x1c0
[    4.724952][    C0] --- interrupt: 500 at .virtnet_send_command+0x138/0x1b0
[    4.725031][    C0] NIP:  c000000000b4db68 LR: c000000000b4db58 CTR: c000000000984910
[    4.725090][    C0] REGS: c00000000a0731d0 TRAP: 0500   Not tainted  (5.13.0-rc2-00118-gca433a3a44e3-dirty)
[    4.725156][    C0] MSR:  8000000000009032 <SF,EE,ME,IR,DR,RI>  CR: 24002444  XER: 20000000
[    4.725299][    C0] CFAR: c00000000097e110 IRQMASK: 0
[    4.725299][    C0] GPR00: c000000000b4db58 c00000000a073470 c000000002ad1a00 0000000000000000
[    4.725299][    C0] GPR04: c00000000a0734e4 0000000000000000 0000000000000000 0000000000000000
[    4.725299][    C0] GPR08: 0000000000000000 0000000000000000 0000000000000000 c00c00000002bb48
[    4.725299][    C0] GPR12: 0000000024002442 c000000002cd0000 0000000000000000 0000000000000000
[    4.725299][    C0] GPR16: 0000000000000000 0000000000000000 0000000000000000 00000000100b7866
[    4.725299][    C0] GPR20: 00000000100b7933 00000000100b5562 00000000100b78c8 c000000008d89000
[    4.725299][    C0] GPR24: 0000000000000003 0000000000001002 c00000000a0734e8 0000000000000002
[    4.725299][    C0] GPR28: 0000000000000000 0000000000000000 c00000000a073508 c00000000ac41900
[    4.726021][    C0] NIP [c000000000b4db68] .virtnet_send_command+0x138/0x1b0
[    4.726091][    C0] LR [c000000000b4db58] .virtnet_send_command+0x128/0x1b0
[    4.726166][    C0] --- interrupt: 500
[    4.726226][    C0] [c00000000a073580] [c000000000b4ee2c] .virtnet_set_rx_mode+0xdc/0x3d0
[    4.726312][    C0] [c00000000a073670] [c000000000d98bac] .__dev_set_rx_mode+0x8c/0x120
[    4.726397][    C0] [c00000000a073700] [c000000000d98e50] .__dev_open+0x1a0/0x220
[    4.726477][    C0] [c00000000a0737b0] [c000000000d994ac] .__dev_change_flags+0x23c/0x2e0
[    4.726557][    C0] [c00000000a073880] [c000000000d99598] .dev_change_flags+0x48/0xa0
[    4.726639][    C0] [c00000000a073910] [c000000000ec2b18] .devinet_ioctl+0x358/0xa00
[    4.726729][    C0] [c00000000a0739f0] [c000000000ec5b1c] .inet_ioctl+0x23c/0x340
[    4.726808][    C0] [c00000000a073b00] [c000000000d52c98] .sock_do_ioctl+0x68/0x1c0
[    4.726891][    C0] [c00000000a073bd0] [c000000000d55ef4] .sock_ioctl+0x394/0x670
[    4.726971][    C0] [c00000000a073cc0] [c0000000004d3e2c] .__se_sys_ioctl+0xdc/0x140
[    4.727081][    C0] [c00000000a073d60] [c00000000002faec] .system_call_exception+0x17c/0x2e0
[    4.727169][    C0] [c00000000a073e10] [c00000000000cfe4] system_call_common+0xf4/0x2dc
[    4.727260][    C0] --- interrupt: c00 at 0x7fff7f9bf2f0
[    4.727630][    C0] NIP:  00007fff7f9bf2f0 LR: 000000001000c6b4 CTR: 0000000000000000
[    4.727692][    C0] REGS: c00000000a073e80 TRAP: 0c00   Not tainted  (5.13.0-rc2-00118-gca433a3a44e3-dirty)
[    4.727758][    C0] MSR:  800000000280f032 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI>  CR: 22000248  XER: 00000000
[    4.727956][    C0] IRQMASK: 0
[    4.727956][    C0] GPR00: 0000000000000036 00007ffffba7db00 00007fff7fa87300 0000000000000003
[    4.727956][    C0] GPR04: 0000000000008914 00007ffffba7dc60 00000000100b554e 0000000000000000
[    4.727956][    C0] GPR08: 0000000000000003 0000000000000000 0000000000000000 0000000000000000
[    4.727956][    C0] GPR12: 0000000000000000 00007fff7fb0a3a0 0000000000000000 0000000000000000
[    4.727956][    C0] GPR16: 0000000000000000 0000000000000000 0000000000000000 00000000100b7866
[    4.727956][    C0] GPR20: 00000000100b7933 00000000100b5562 00000000100b78c8 0000000000000000
[    4.727956][    C0] GPR24: 0000000000000000 ffffffffffffffff ffffffffffffffff 0000000000000000
[    4.727956][    C0] GPR28: 00007ffffba7ff69 0000000000000001 0000000000000003 00000000100b554e
[    4.728666][    C0] NIP [00007fff7f9bf2f0] 0x7fff7f9bf2f0
[    4.728719][    C0] LR [000000001000c6b4] 0x1000c6b4
[    4.728770][    C0] --- interrupt: c00
[    4.728909][    C0] Instruction dump:
[    4.729071][    C0] 38636c58 4bffbab9 60000000 4bfffed4 e8be0000 3c62fe70 3d22ffe7 7f44d378
[    4.729251][    C0] 38636c18 9ae9f062 4bf657c1 60000000 <0fe00000> 9b2d0932 4bfffe70 60000000
[    4.729494][    C0] ---[ end trace 0d7d391060ba35b4 ]---

qemu-system-ppc64: terminating on signal 1 from pid 462523 (

author reviewer
#14 lore raw

Re: [PATCH v3 11/11] powerpc/64: use interrupt restart table to speed up return from interrupt

In reply to Michael Ellerman

Excerpts from Michael Ellerman's message of June 15, 2021 11:44 pm:

Quoted 17 lines
Quoted 16 lines · Nicholas Piggin
arch/powerpc/lib/feature-fixups.c +2 -1
index fe26f2fa0f3f..fbe94e2d5011 100644
@@ -412,12 +430,19 @@ void do_entry_flush_fixups(enum l1d_flush_type types)
 	stop_machine(__do_entry_flush_fixups, &types, NULL);
 }
 
-void do_rfi_flush_fixups(enum l1d_flush_type types)
+static int __do_rfi_flush_fixups(void *data)
 {
+	enum l1d_flush_type types = *(enum l1d_flush_type *)data;
 	unsigned int instrs[3], *dest;
 	long *start, *end;
 	int i;
 
Quoted 7 lines

arch/powerpc/lib/feature-fixups.c void do_entry_flush_fixups(enum l1d_flush_type types) · line 444

Quoted 5 lines
arch/powerpc/lib/feature-fixups.c +5 -0
+	if (types & L1D_FLUSH_FALLBACK)
+		rfi_exit_not_reentrant = true;
+	else
+		rfi_exit_not_reentrant = false;
+	update_interrupt_exit();

This is not happy:

Ah, needs a static_key_enable_cpuslocked. I'll fix.

Thanks,
Nick

Quoted 48 lines
[    0.000000][    T0] ============================================
[    0.000000][    T0] WARNING: possible recursive locking detected
[    0.000000][    T0] 5.13.0-rc2-00118-gca433a3a44e3 #1 Not tainted
[    0.000000][    T0] --------------------------------------------
[    0.000000][    T0] swapper/0 is trying to acquire lock:
[    0.000000][    T0] c00000000252fa10 (cpu_hotplug_lock){....}-{0:0}, at: static_key_enable+0x24/0x50
[    0.000000][    T0]
[    0.000000][    T0] but task is already holding lock:
[    0.000000][    T0] c00000000252fa10 (cpu_hotplug_lock){....}-{0:0}, at: stop_machine+0x2c/0x60
[    0.000000][    T0]
[    0.000000][    T0] other info that might help us debug this:
[    0.000000][    T0]  Possible unsafe locking scenario:
[    0.000000][    T0]
[    0.000000][    T0]        CPU0
[    0.000000][    T0]        ----
[    0.000000][    T0]   lock(cpu_hotplug_lock);
[    0.000000][    T0]   lock(cpu_hotplug_lock);
[    0.000000][    T0]
[    0.000000][    T0]  *** DEADLOCK ***
[    0.000000][    T0]
[    0.000000][    T0]  May be due to missing lock nesting notation
[    0.000000][    T0]
[    0.000000][    T0] 1 lock held by swapper/0:
[    0.000000][    T0]  #0: c00000000252fa10 (cpu_hotplug_lock){....}-{0:0}, at: stop_machine+0x2c/0x60
[    0.000000][    T0]
[    0.000000][    T0] stack backtrace:
[    0.000000][    T0] CPU: 0 PID: 0 Comm: swapper Not tainted 5.13.0-rc2-00118-gca433a3a44e3 #1
[    0.000000][    T0] Call Trace:
[    0.000000][    T0] [c0000000027db8f0] [c00000000093dd28] dump_stack+0xec/0x144 (unreliable)
[    0.000000][    T0] [c0000000027db940] [c0000000001ed5b4] __lock_acquire+0x1744/0x28b0
[    0.000000][    T0] [c0000000027dba70] [c0000000001ef338] lock_acquire+0x128/0x600
[    0.000000][    T0] [c0000000027dbb70] [c00000000015035c] cpus_read_lock+0x4c/0x170
[    0.000000][    T0] [c0000000027dbba0] [c0000000003c2594] static_key_enable+0x24/0x50
[    0.000000][    T0] [c0000000027dbbd0] [c0000000000ae87c] __do_rfi_flush_fixups+0x7c/0x300
[    0.000000][    T0] [c0000000027dbc80] [c0000000002ab7e4] stop_machine_cpuslocked+0xe4/0x200
[    0.000000][    T0] [c0000000027dbcf0] [c0000000002ab940] stop_machine+0x40/0x60
[    0.000000][    T0] [c0000000027dbd30] [c0000000000aef30] do_rfi_flush_fixups+0x30/0x50
[    0.000000][    T0] [c0000000027dbd60] [c000000000040890] setup_rfi_flush+0xa0/0x140
[    0.000000][    T0] [c0000000027dbdd0] [c00000000201c6c4] pnv_setup_arch+0x304/0x4ac
[    0.000000][    T0] [c0000000027dbe60] [c00000000200a31c] setup_arch+0x374/0x3c4
[    0.000000][    T0] [c0000000027dbee0] [c000000002003d08] start_kernel+0xb0/0x790
[    0.000000][    T0] [c0000000027dbf90] [c00000000000d79c] start_here_common+0x1c/0x600
[    0.000000][    T0] rfi-flush: patched 12 locations (fallback displacement flush)

cheers

author reviewer
#16 lore raw
In reply to Michael Ellerman

Excerpts from Michael Ellerman's message of June 16, 2021 12:07 am:

Quoted 5 lines
Quoted 2 lines · Nicholas Piggin

This series attempts to improve the speed of interrupts and system calls in three major ways.

With the full series applied I'm seeing various warnings.

Gah, sorry :( I've tested hash/radix guest host mambo qemu real hw... Are they all hash, or qemu powernv radix?

Obviously not 64e since the interrupt rewrite, but it did seem to be working (I'll fix and re test).

Quoted 1 line

I feel like I must be missing some lead up patch?

Shouldn't be, I would guess a config option maybe not marking srrs clobbered somewhere? I'm generally using the defconfigs plus some random smattering of debug options at any time.

I'll grab some .configs from you...

Thanks,
Nick

Quoted 177 lines

cheers

mambo, p8, hash, be:

[    0.001038][    T0] ------------[ cut here ]------------
[    0.001287][    T0] Interrupts were enabled early
[    0.001510][    T0] WARNING: CPU: 0 PID: 0 at init/main.c:1004 .start_kernel+0x53c/0x720
[    0.001902][    T0] Modules linked in:
[    0.002084][    T0] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.13.0-rc2-00118-gca433a3a44e3-dirty #1
[    0.002518][    T0] NIP:  c000000002004920 LR: c00000000200491c CTR: c0000000000e3cd0
[    0.002887][    T0] REGS: c000000002acfc30 TRAP: 0700   Not tainted  (5.13.0-rc2-00118-gca433a3a44e3-dirty)
[    0.003342][    T0] MSR:  9000000002029032 <SF,HV,VEC,EE,ME,IR,DR,RI>  CR: 28002222  XER: 22000000
[    0.003818][    T0] CFAR: 000000003002afb0 IRQMASK: 0
[    0.003818][    T0] GPR00: c00000000200491c c000000002acfed0 c000000002ad1a00 000000000000001d
[    0.003818][    T0] GPR04: 0000000000000001 c0000000027645b0 0000000000000001 c0000000027645e0
[    0.003818][    T0] GPR08: 00000000fd5c0000 c000000002764560 c000000002764560 9000000032001032
[    0.003818][    T0] GPR12: 0000000028002222 c000000002cd0000 0000000000000000 0000000000000000
[    0.003818][    T0] GPR16: 0000000031c10000 0000000030003180 0000000030000000 0000000000000000
[    0.003818][    T0] GPR20: 0000000000000000 0000000000000000 000000003012e4eb 0000000030110ece
[    0.003818][    T0] GPR24: 00000000301105f5 c000000000000000 0000000020010000 c000000002090008
[    0.003818][    T0] GPR28: c000000002090008 c0000000011ae620 c0000000011ae620 c000000002090008
[    0.007455][    T0] NIP [c000000002004920] .start_kernel+0x53c/0x720
[    0.007765][    T0] LR [c00000000200491c] .start_kernel+0x538/0x720
[    0.008072][    T0] Call Trace:
[    0.008223][    T0] [c000000002acfed0] [c00000000200491c] .start_kernel+0x538/0x720 (unreliable)
[    0.008655][    T0] [c000000002acff90] [c00000000000dd2c] start_here_common+0x1c/0x670
[    0.009041][    T0] Instruction dump:
[    0.009219][    T0] 60000000 4a23dadd 60000000 48035821 60000000 892d0932 71290001 40820018
[    0.009659][    T0] 3c62fe6d 3863f810 4a171751 60000000 <0fe00000> 39200000 3d420003 38600000
[    0.010111][    T0] ---[ end trace e2dce2fbc72d04fa ]---

mambo, p9, hash, be:

[   17.668403][  T267] ------------[ cut here ]------------
[   17.668433][  T267] WARNING: CPU: 1 PID: 267 at arch/powerpc/include/asm/book3s/64/kup.h:304 .arch_local_irq_restore+0x190/0x1a0
[   17.668503][  T267] Modules linked in:
[   17.668532][  T267] CPU: 1 PID: 267 Comm: halt Tainted: G        W         5.13.0-rc2-00118-gca433a3a44e3-dirty #1
[   17.668588][  T267] NIP:  c000000000016680 LR: c00000000002f804 CTR: c0000000005901c0
[   17.668632][  T267] REGS: c000000009f339b0 TRAP: 0700   Tainted: G        W          (5.13.0-rc2-00118-gca433a3a44e3-dirty)
[   17.668686][  T267] MSR:  9000000000021032 <SF,HV,ME,IR,DR,RI>  CR: 48244222  XER: 04000000
[   17.668777][  T267] CFAR: c00000000002fdcc IRQMASK: 3
[   17.668777][  T267] GPR00: c00000000002f804 c000000009f33c50 c000000002ad1a00 0000000000000000
[   17.668777][  T267] GPR04: 0000000000000000 0000000000000020 00000000000000b7 00000000000c9495
[   17.668777][  T267] GPR08: 0000000000000008 fcffffffffffffff 3cffffffffffffff 0000000000000003
[   17.668777][  T267] GPR12: 0000000028244228 c0000000fffff300 0000000000000000 0000000000000000
[   17.668777][  T267] GPR16: 0000000000000000 0000000000000000 0000000000000000 00000000100a0a18
[   17.668777][  T267] GPR20: 00007fffdf62ffac 00000000100a09d8 00000100011405f0 0000000002002000
[   17.668777][  T267] GPR24: 0000000000000002 0000000000000001 0000000002802000 0000000000000000
[   17.668777][  T267] GPR28: c00000000cc56080 0000000000000000 0000000000000000 3cffffffffffffff
[   17.669291][  T267] NIP [c000000000016680] .arch_local_irq_restore+0x190/0x1a0
[   17.669343][  T267] LR [c00000000002f804] .interrupt_exit_user_prepare_main+0x144/0x2b0
[   17.669394][  T267] Call Trace:
[   17.669418][  T267] [c000000009f33c50] [c000000009f33ce0] 0xc000000009f33ce0 (unreliable)
[   17.669475][  T267] [c000000009f33cd0] [c00000000002f804] .interrupt_exit_user_prepare_main+0x144/0x2b0
[   17.669538][  T267] [c000000009f33d90] [c00000000002fe08] .syscall_exit_restart+0x68/0x80
[   17.669596][  T267] [c000000009f33e10] [c00000000000d168] system_call_common+0x278/0x2dc
[   17.669652][  T267] --- interrupt: c00 at 0x7fffad5b867c
[   17.669688][  T267] NIP:  00007fffad5b867c LR: 00000000100992a0 CTR: 0000000000000000
[   17.669731][  T267] REGS: c000000009f33e80 TRAP: 0c00   Tainted: G        W          (5.13.0-rc2-00118-gca433a3a44e3-dirty)
[   17.669785][  T267] MSR:  900000000280f032 <SF,HV,VEC,VSX,EE,PR,FP,ME,IR,DR,RI>  CR: 28244222  XER: 00000000
[   17.669910][  T267] IRQMASK: 0
[   17.669910][  T267] GPR00: 0000000000000006 00007fffd8758060 00007fffad687300 0000000000000000
[   17.669910][  T267] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   17.669910][  T267] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   17.669910][  T267] GPR12: 0000000000000000 00007fffad70a3a0 0000000000000000 0000000000000000
[   17.669910][  T267] GPR16: 0000000000000000 0000000000000000 0000000000000000 00000000100a0a18
[   17.669910][  T267] GPR20: 00007fffdf62ffac 00000000100a09d8 00000100011405f0 00000000100e24d0
[   17.669910][  T267] GPR24: 00000000100b91e5 00000000100c1e14 0000010019e002a0 00000000100c1e1e
[   17.669910][  T267] GPR28: 00007fffd875821b 0000000000000004 00007fffd8758210 0000000000000000
[   17.670410][  T267] NIP [00007fffad5b867c] 0x7fffad5b867c
[   17.670446][  T267] LR [00000000100992a0] 0x100992a0
[   17.670480][  T267] --- interrupt: c00
[   17.670507][  T267] Instruction dump:
[   17.670535][  T267] 4e800020 60000000 60000000 60000000 39200000 992d0932 39400000 992d0933
[   17.670627][  T267] 614a8002 7d410164 4e800020 60000000 <0fe00000> 4bfffefc 60000000 60000000
[   17.670719][  T267] ---[ end trace 5ba1b854b2957ce3 ]---

qemu powernv:

[    3.446262][   T51] init[51]: segfault (11) at 0 nip 0 lr 1000ca78 code 1 in busybox[10000000+d0000]
[    3.448351][   T51] init[51]: code: XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
[    3.449598][   T51] init[51]: code: XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
[    3.460559][   T52] init[52]: segfault (11) at 0 nip 0 lr 1000ca78 code 1 in busybox[10000000+d0000]
[    3.461409][   T52] init[52]: code: XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
[    3.462496][   T52] init[52]: code: XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX

... to infinity.

qemu 64e gets to running /init but gets stuck there with no further output.

qemu pseries, p8, be:

[    3.836653][   T75] S20urandom (75) used greatest stack depth: 9792 bytes left
[    4.718806][    C0] ------------[ cut here ]------------
[    4.718971][    C0] irq 17 handler .vp_interrupt+0x0/0xc0 enabled interrupts
[    4.720956][    C0] WARNING: CPU: 0 PID: 99 at kernel/irq/handle.c:159 .__handle_irq_event_percpu+0x230/0x270
[    4.721113][    C0] Modules linked in:
[    4.721502][    C0] CPU: 0 PID: 99 Comm: ip Not tainted 5.13.0-rc2-00118-gca433a3a44e3-dirty #1
[    4.721861][    C0] NIP:  c0000000002108b0 LR: c0000000002108ac CTR: c0000000009bb220
[    4.721943][    C0] REGS: c0000000fffeba00 TRAP: 0700   Not tainted  (5.13.0-rc2-00118-gca433a3a44e3-dirty)
[    4.722081][    C0] MSR:  8000000000029032 <SF,EE,ME,IR,DR,RI>  CR: 28002247  XER: 20000000
[    4.722404][    C0] CFAR: c0000000001760d0 IRQMASK: 0
[    4.722404][    C0] GPR00: c0000000002108ac c0000000fffebca0 c000000002ad1a00 0000000000000038
[    4.722404][    C0] GPR04: 0000000000000001 c0000000027645b0 0000000000000027 c0000000ff906e10
[    4.722404][    C0] GPR08: 0000000000000023 ffffffffffffffd8 0000000000000027 c0000000011d7b40
[    4.722404][    C0] GPR12: 0000000028002247 c000000002cd0000 0000000000000000 0000000000000000
[    4.722404][    C0] GPR16: 0000000000000000 0000000000000000 0000000000000000 00000000100b7866
[    4.722404][    C0] GPR20: 00000000100b7933 00000000100b5562 0000000000000000 0000000000000001
[    4.722404][    C0] GPR24: 0000000000000002 0000000000000001 0000000000000011 c000000008891800
[    4.722404][    C0] GPR28: 0000000000000011 c0000000fffebdd4 c00000000814a880 0000000000000000
[    4.723412][    C0] NIP [c0000000002108b0] .__handle_irq_event_percpu+0x230/0x270
[    4.723497][    C0] LR [c0000000002108ac] .__handle_irq_event_percpu+0x22c/0x270
[    4.723719][    C0] Call Trace:
[    4.723907][    C0] [c0000000fffebca0] [c0000000002108ac] .__handle_irq_event_percpu+0x22c/0x270 (unreliable)
[    4.724205][    C0] [c0000000fffebd60] [c000000000210ae0] .handle_irq_event+0x130/0x190
[    4.724309][    C0] [c0000000fffebe00] [c0000000002182bc] .handle_fasteoi_irq+0xbc/0x310
[    4.724397][    C0] [c0000000fffebea0] [c00000000020ea74] .generic_handle_irq+0x54/0x80
[    4.724575][    C0] [c0000000fffebf10] [c000000000015770] .__do_irq+0x70/0x1c0
[    4.724667][    C0] [c0000000fffebf90] [c0000000000159d4] .do_IRQ+0x114/0x250
[    4.724764][    C0] [c00000000a0730c0] [c00000000001596c] .do_IRQ+0xac/0x250
[    4.724848][    C0] [c00000000a073160] [c000000000009230] hardware_interrupt_common_virt+0x1b0/0x1c0
[    4.724952][    C0] --- interrupt: 500 at .virtnet_send_command+0x138/0x1b0
[    4.725031][    C0] NIP:  c000000000b4db68 LR: c000000000b4db58 CTR: c000000000984910
[    4.725090][    C0] REGS: c00000000a0731d0 TRAP: 0500   Not tainted  (5.13.0-rc2-00118-gca433a3a44e3-dirty)
[    4.725156][    C0] MSR:  8000000000009032 <SF,EE,ME,IR,DR,RI>  CR: 24002444  XER: 20000000
[    4.725299][    C0] CFAR: c00000000097e110 IRQMASK: 0
[    4.725299][    C0] GPR00: c000000000b4db58 c00000000a073470 c000000002ad1a00 0000000000000000
[    4.725299][    C0] GPR04: c00000000a0734e4 0000000000000000 0000000000000000 0000000000000000
[    4.725299][    C0] GPR08: 0000000000000000 0000000000000000 0000000000000000 c00c00000002bb48
[    4.725299][    C0] GPR12: 0000000024002442 c000000002cd0000 0000000000000000 0000000000000000
[    4.725299][    C0] GPR16: 0000000000000000 0000000000000000 0000000000000000 00000000100b7866
[    4.725299][    C0] GPR20: 00000000100b7933 00000000100b5562 00000000100b78c8 c000000008d89000
[    4.725299][    C0] GPR24: 0000000000000003 0000000000001002 c00000000a0734e8 0000000000000002
[    4.725299][    C0] GPR28: 0000000000000000 0000000000000000 c00000000a073508 c00000000ac41900
[    4.726021][    C0] NIP [c000000000b4db68] .virtnet_send_command+0x138/0x1b0
[    4.726091][    C0] LR [c000000000b4db58] .virtnet_send_command+0x128/0x1b0
[    4.726166][    C0] --- interrupt: 500
[    4.726226][    C0] [c00000000a073580] [c000000000b4ee2c] .virtnet_set_rx_mode+0xdc/0x3d0
[    4.726312][    C0] [c00000000a073670] [c000000000d98bac] .__dev_set_rx_mode+0x8c/0x120
[    4.726397][    C0] [c00000000a073700] [c000000000d98e50] .__dev_open+0x1a0/0x220
[    4.726477][    C0] [c00000000a0737b0] [c000000000d994ac] .__dev_change_flags+0x23c/0x2e0
[    4.726557][    C0] [c00000000a073880] [c000000000d99598] .dev_change_flags+0x48/0xa0
[    4.726639][    C0] [c00000000a073910] [c000000000ec2b18] .devinet_ioctl+0x358/0xa00
[    4.726729][    C0] [c00000000a0739f0] [c000000000ec5b1c] .inet_ioctl+0x23c/0x340
[    4.726808][    C0] [c00000000a073b00] [c000000000d52c98] .sock_do_ioctl+0x68/0x1c0
[    4.726891][    C0] [c00000000a073bd0] [c000000000d55ef4] .sock_ioctl+0x394/0x670
[    4.726971][    C0] [c00000000a073cc0] [c0000000004d3e2c] .__se_sys_ioctl+0xdc/0x140
[    4.727081][    C0] [c00000000a073d60] [c00000000002faec] .system_call_exception+0x17c/0x2e0
[    4.727169][    C0] [c00000000a073e10] [c00000000000cfe4] system_call_common+0xf4/0x2dc
[    4.727260][    C0] --- interrupt: c00 at 0x7fff7f9bf2f0
[    4.727630][    C0] NIP:  00007fff7f9bf2f0 LR: 000000001000c6b4 CTR: 0000000000000000
[    4.727692][    C0] REGS: c00000000a073e80 TRAP: 0c00   Not tainted  (5.13.0-rc2-00118-gca433a3a44e3-dirty)
[    4.727758][    C0] MSR:  800000000280f032 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI>  CR: 22000248  XER: 00000000
[    4.727956][    C0] IRQMASK: 0
[    4.727956][    C0] GPR00: 0000000000000036 00007ffffba7db00 00007fff7fa87300 0000000000000003
[    4.727956][    C0] GPR04: 0000000000008914 00007ffffba7dc60 00000000100b554e 0000000000000000
[    4.727956][    C0] GPR08: 0000000000000003 0000000000000000 0000000000000000 0000000000000000
[    4.727956][    C0] GPR12: 0000000000000000 00007fff7fb0a3a0 0000000000000000 0000000000000000
[    4.727956][    C0] GPR16: 0000000000000000 0000000000000000 0000000000000000 00000000100b7866
[    4.727956][    C0] GPR20: 00000000100b7933 00000000100b5562 00000000100b78c8 0000000000000000
[    4.727956][    C0] GPR24: 0000000000000000 ffffffffffffffff ffffffffffffffff 0000000000000000
[    4.727956][    C0] GPR28: 00007ffffba7ff69 0000000000000001 0000000000000003 00000000100b554e
[    4.728666][    C0] NIP [00007fff7f9bf2f0] 0x7fff7f9bf2f0
[    4.728719][    C0] LR [000000001000c6b4] 0x1000c6b4
[    4.728770][    C0] --- interrupt: c00
[    4.728909][    C0] Instruction dump:
[    4.729071][    C0] 38636c58 4bffbab9 60000000 4bfffed4 e8be0000 3c62fe70 3d22ffe7 7f44d378
[    4.729251][    C0] 38636c18 9ae9f062 4bf657c1 60000000 <0fe00000> 9b2d0932 4bfffe70 60000000
[    4.729494][    C0] ---[ end trace 0d7d391060ba35b4 ]---

qemu-system-ppc64: terminating on signal 1 from pid 462523 (