arm64: Move overflow sp into SP_EL1 and kernel sp into SP_EL0

Hi folks,

This is version two of the kernel stack juggling patches I previously posted here:

https://lore.kernel.org/r/[email protected]

Changes since v1 include:
* Fixed suspend/resume paths to handle the stack pointers properly
* Fixed restoration of ptrauth keys on resume
* Added tags

Cheers,

Will

--->8

Mostafa Saleh (1):
KVM: arm64: Protect TPIDRRO_EL0 across guest entry/exit

Will Deacon (20):
arm64: entry: Defer setting of TPIDRRO_EL0 until exit to userspace
arm64: entry: Only check for stack overflow on exceptions from EL1
arm64: stackprotector: Temporarily disable per-task stackprotector
arm64: bpf: Add support for generating reads of TPIDRRO_EL0
arm64: Store 'current' in TPIDRRO_EL0 instead of SP_EL0 selftests/bpf: arm64: Use TPIDRRO_EL0 instead of SP_EL0 for 'current' scripts/gdb: arm64: Use TPIDRRO_EL0 instead of SP_EL0 for 'current'
arm64: stackprotector: Re-enable per-task stackprotector
arm64: percpu: Specialise set_my_cpu_offset() for the primary CPU
arm64: percpu: Annotate __kern_my_cpu_offset() as '__always_inline'
KVM: arm64: Preserve handler/thread bit of EL1 mode in

    __finalise_el2()

arm64: sdei: Guard most of asm/sdei.h with CONFIG_ARM_SDE_INTERFACE
arm64: sdei: Support SDEI events from kernel handler and thread modes
arm64: entry: Point SP_EL0 at the overflow stack
arm64: entry: Implement EL1t exception handlers for overflow stack
arm64: entry: Use SPSel to switch to overflow stack
arm64: entry: Split up kernel_ventry macro into separate helper macros
arm64: entry: The great stack switcheroo
arm64: tracing: Advertise a mode of EL1t in synthetic kernel regs
arm64: Rename 'overflow_stack' and OVERFLOW_STACK_SIZE

 arch/arm64/Kconfig                            |   2 +-
 arch/arm64/Makefile                           |   2 +-
 arch/arm64/include/asm/assembler.h            |   2 +-
 arch/arm64/include/asm/current.h              |   6 +-
 arch/arm64/include/asm/ftrace.h               |   4 +-
 arch/arm64/include/asm/insn.h                 |   1 +
 arch/arm64/include/asm/kvm_asm.h              |  26 +++++
 arch/arm64/include/asm/memory.h               |  10 +-
 arch/arm64/include/asm/percpu.h               |  10 +-
 arch/arm64/include/asm/perf_event.h           |   2 +-
 arch/arm64/include/asm/sdei.h                 |   2 +
 arch/arm64/include/asm/stacktrace.h           |   9 +-
 arch/arm64/include/asm/stacktrace/nvhe.h      |   3 +-
 arch/arm64/include/uapi/asm/ptrace.h          |   1 +
 arch/arm64/kernel/asm-offsets.c               |   2 +
 arch/arm64/kernel/entry-common.c              |  51 +++++++--
 arch/arm64/kernel/entry.S                     | 108 +++++++++---------
 arch/arm64/kernel/head.S                      |   8 +-
 arch/arm64/kernel/hyp-stub.S                  |   7 +-
 arch/arm64/kernel/process.c                   |  21 +---
 arch/arm64/kernel/sdei.c                      |   6 +-
 arch/arm64/kernel/sleep.S                     |   3 +-
 arch/arm64/kernel/smp.c                       |  19 ++-
 arch/arm64/kernel/stacktrace.c                |   2 +-
 arch/arm64/kernel/sys_compat.c                |   7 --
 arch/arm64/kernel/traps.c                     |  14 +--
 arch/arm64/kvm/hyp/entry.S                    |  15 ++-
 arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h    |   2 -
 arch/arm64/kvm/hyp/nvhe/host.S                |   2 +-
 arch/arm64/kvm/hyp/nvhe/stacktrace.c          |   4 +-
 arch/arm64/kvm/stacktrace.c                   |   4 +-
 arch/arm64/mm/proc.S                          |  16 ++-
 arch/arm64/net/bpf_jit.h                      |   2 +
 arch/arm64/net/bpf_jit_comp.c                 |   4 +-
 scripts/gdb/linux/cpus.py                     |   2 +-
 .../selftests/bpf/progs/verifier_jit_inline.c |   2 +-
 36 files changed, 230 insertions(+), 151 deletions(-)

2.55.0.1082.g2b9226bbc0-goog

In preparation for using TPIDRRO_EL0 to point at 'current' while running inside the kernel, defer its userspace initialisation from the context-switch patch to exception return. This has the added benefit of not having to worry about keeping the in-memory value and the register value in-sync during preemptible sections.

---
 arch/arm64/kernel/asm-offsets.c |  1 +
 arch/arm64/kernel/entry.S       |  4 +++-
 arch/arm64/kernel/process.c     | 16 +---------------
 arch/arm64/kernel/sys_compat.c  |  7 -------
 4 files changed, 5 insertions(+), 23 deletions(-)

index 9c853ed3ceab..6038ab3beb25 100644
@@ -42,6 +42,7 @@ int main(void)
 #endif
   BLANK();
   DEFINE(THREAD_CPU_CONTEXT,	offsetof(struct task_struct, thread.cpu_context));
+  DEFINE(THREAD_TP_VALUE,	offsetof(struct task_struct, thread.uw.tp_value));
   DEFINE(THREAD_SCTLR_USER,	offsetof(struct task_struct, thread.sctlr_user));
 #ifdef CONFIG_ARM64_PTR_AUTH
   DEFINE(THREAD_KEYS_USER,	offsetof(struct task_struct, thread.keys_user));

index f63049ac32dc..59f045e496ec 100644
@@ -47,7 +47,6 @@
 	b	.Lskip_tramp_vectors_cleanup\@
 	.if	\regsize == 64
 	mrs	x30, tpidrro_el0
-	msr	tpidrro_el0, xzr
 	.else
 	mov	x30, xzr
 	.endif

@@ -360,9 +359,12 @@ alternative_else_nop_endif
 	.if	\el == 0
 	ldr	x23, [sp, #S_SP]		// load return stack pointer
 	msr	sp_el0, x23
+	msr	tpidrro_el0, xzr
 	tst	x22, #PSR_MODE32_BIT		// native task?
 	b.eq	3f
 
+	ldr	x0, [tsk, #THREAD_TP_VALUE]
+	msr	tpidrro_el0, x0
 #ifdef CONFIG_ARM64_ERRATUM_845719
 alternative_if ARM64_WORKAROUND_845719
 #ifdef CONFIG_PID_IN_CONTEXTIDR

index 581f80e9b9b7..bfdc12166895 100644
@@ -256,17 +256,8 @@ static void tls_thread_flush(void)
 	if (system_supports_tpidr2())
 		write_sysreg_s(0, SYS_TPIDR2_EL0);
 
-	if (is_compat_task()) {
+	if (is_compat_task())
 		current->thread.uw.tp_value = 0;
-
-		/*
-		 * We need to ensure ordering between the shadow state and the
-		 * hardware state, so that we don't corrupt the hardware state
-		 * with a stale shadow state during context switch.
-		 */
-		barrier();
-		write_sysreg(0, tpidrro_el0);
-	}
 }
 
 static void flush_tagged_addr_state(void)

@@ -531,11 +522,6 @@ static void tls_thread_switch(struct task_struct *next)
 {
 	tls_preserve_current_state();
 
-	if (is_compat_thread(task_thread_info(next)))
-		write_sysreg(next->thread.uw.tp_value, tpidrro_el0);
-	else
-		write_sysreg(0, tpidrro_el0);
-
 	write_sysreg(*task_user_tls(next), tpidr_el0);
 	if (system_supports_tpidr2())
 		write_sysreg_s(next->thread.tpidr2_el0, SYS_TPIDR2_EL0);

index 0451f96c2c3f..59eee301c6f4 100644
@@ -89,13 +89,6 @@ long compat_arm_syscall(struct pt_regs *regs, int scno)
 
 	case __ARM_NR_compat_set_tls:
 		current->thread.uw.tp_value = regs->regs[0];
-
-		/*
-		 * Protect against register corruption from context switch.
-		 * See comment in tls_thread_flush.
-		 */
-		barrier();
-		write_sysreg(regs->regs[0], tpidrro_el0);
 		return 0;
 
 	default:

2.55.0.1082.g2b9226bbc0-goog

Hi Will,

long time before I had a chance to look at this.

I wrote a similar patch at one time so I like the concept.

I would add some elaboration to the commit message as to what is going on in the patch, and correct me if I got it wrong!

- Userspace sees 0 in the TPIDRRO_EL0 when running native tasks so we just zero it when exiting to userspace for those.

- For compat mode, TPIDRRO_EL0 should contain current->thread.uw.tp_value, so in that case we just write that before exiting to userspace.

- Userspace can't modify the register so we don't care about saving it on entry.

These three seems to be what the patch is exploiting.

If this is correct, with the above description added:

Yours,
Linus Walleij

Checking for kernel stack overflow on entry from EL0 is largely pointless and gets in the way of using SPSel to switch to the overflow stack. If the kernel stack overflows on entry from userspace, that implies that it was unbalanced on the last exception return which means we can't sensibly reason about the state of the system given that we could've returned to an arbitrary virtual address in an arbitrary mode.

Remove the stack overflow check from the EL0 exception entry path.

---
 arch/arm64/kernel/entry.S | 6 ++++++
 1 file changed, 6 insertions(+)

index 59f045e496ec..b5d8277f608a 100644
@@ -54,6 +54,8 @@
 	.endif
 
 	sub	sp, sp, #PT_REGS_SIZE
+
+	.if	\el == 1
 	/*
 	 * Test whether the SP has overflowed, without corrupting a GPR.
 	 * Task and IRQ stacks are aligned so that SP & (1 << THREAD_SHIFT)

@@ -64,8 +66,11 @@
 	tbnz	x0, #THREAD_SHIFT, 0f
 	sub	x0, sp, x0			// x0'' = sp' - x0' = (sp + x0) - sp = x0
 	sub	sp, sp, x0			// sp'' = sp' - x0 = (sp + x0) - x0 = sp
+	.endif
+
 	b	el\el\ht\()_\regsize\()_\label
 
+	.if	\el == 1
 0:
 	/*
 	 * Either we've just detected an overflow, or we've taken an exception

@@ -96,6 +101,7 @@
 	sub	sp, sp, x0
 	mrs	x0, tpidrro_el0
 	b	el\el\ht\()_\regsize\()_\label
+	.endif
 .org .Lventry_start\@ + 128	// Did we overflow the ventry slot?
 	.endm
 

2.55.0.1082.g2b9226bbc0-goog

I agree with this reasoning, and it confirms my own understanding of the futility of checking this at entry from EL0.

Yours,
Linus Walleij

Per-task stackprotector relies on telling the compiler about the system register used to hold a pointer to the current task. As we're about to change that around, temporarily disable per-task stackprotector for now.

---
 arch/arm64/Kconfig | 1 +
 1 file changed, 1 insertion(+)

index b5a51b0ef944..b8270b301809 100644
@@ -2455,6 +2455,7 @@ config CC_HAVE_STACKPROTECTOR_SYSREG
 config STACKPROTECTOR_PER_TASK
 	def_bool y
 	depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_SYSREG
+	depends on BROKEN
 
 config UNWIND_PATCH_PAC_INTO_SCS
 	bool "Enable shadow call stack dynamically using code patching"

2.55.0.1082.g2b9226bbc0-goog

That's not the only thing that breaks in the patches between here and re-enabling it, but what is the alternative other than to just squash all the patches into one :/

Yours,
Linus Walleij

In preparation for storing 'current' in TPIDRRO_EL0, add support for the instruction encoding of "MRS Rt, TPIDRRO_EL0" to the BPF JIT.

---
 arch/arm64/include/asm/insn.h | 1 +
 arch/arm64/net/bpf_jit.h      | 2 ++
 2 files changed, 3 insertions(+)

index 1ce75a442638..0e739740513d 100644
@@ -136,6 +136,7 @@ enum aarch64_insn_special_register {
 };
 
 enum aarch64_insn_system_register {
+	AARCH64_INSN_SYSREG_TPIDRRO_EL0	= 0x5e83,
 	AARCH64_INSN_SYSREG_TPIDR_EL1	= 0x4684,
 	AARCH64_INSN_SYSREG_TPIDR_EL2	= 0x6682,
 	AARCH64_INSN_SYSREG_SP_EL0	= 0x4208,

index b2fe6e6dcf44..df469a005373 100644
@@ -331,6 +331,8 @@
 	aarch64_insn_gen_adr(0, offset, Rd, AARCH64_INSN_ADR_TYPE_ADR)
 
 /* MRS */
+#define A64_MRS_TPIDRRO_EL0(Rt) \
+	aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_TPIDRRO_EL0)
 #define A64_MRS_TPIDR_EL1(Rt) \
 	aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_TPIDR_EL1)
 #define A64_MRS_TPIDR_EL2(Rt) \

2.55.0.1082.g2b9226bbc0-goog

Add to the commit message something like, "so that BPF programs can get a pointer to current."?

Yours,
Linus Walleij

From: Mostafa Saleh <[email protected]>

To prepare for using TPIDRRO_EL0 to hold 'current' and repurposing SP_EL0 as an overflow stack pointer, eagerly switch both registers when entering and exiting the guest.

---
 arch/arm64/include/asm/kvm_asm.h           | 10 ++++++++++
 arch/arm64/kernel/asm-offsets.c            |  1 +
 arch/arm64/kvm/hyp/entry.S                 | 12 ++++++++----
 arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h |  2 --
 4 files changed, 19 insertions(+), 6 deletions(-)

index e5b92ac09e69..1cfe9216a2ff 100644
@@ -408,6 +408,16 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt,
 	msr	sp_el0, \tmp
 .endm
 
+.macro save_tpidrro_el0 ctxt, tmp
+	mrs	\tmp,	tpidrro_el0
+	str	\tmp,	[\ctxt, #CPU_TPIDRRO_EL0_OFFSET]
+.endm
+
+.macro restore_tpidrro_el0 ctxt, tmp
+	ldr	\tmp,	  [\ctxt, #CPU_TPIDRRO_EL0_OFFSET]
+	msr	tpidrro_el0, \tmp
+.endm
+
 #endif
 
 #endif /* __ARM_KVM_ASM_H__ */

index 6038ab3beb25..cb8448cc5f51 100644
@@ -108,6 +108,7 @@ int main(void)
   DEFINE(VCPU_HCR_EL2,		offsetof(struct kvm_vcpu, arch.hcr_el2));
   DEFINE(CPU_USER_PT_REGS,	offsetof(struct kvm_cpu_context, regs));
   DEFINE(CPU_ELR_EL2,		offsetof(struct kvm_cpu_context, sys_regs[ELR_EL2]));
+  DEFINE(CPU_TPIDRRO_EL0_OFFSET, offsetof(struct kvm_cpu_context, sys_regs[TPIDRRO_EL0]));
   DEFINE(CPU_RGSR_EL1,		offsetof(struct kvm_cpu_context, sys_regs[RGSR_EL1]));
   DEFINE(CPU_GCR_EL1,		offsetof(struct kvm_cpu_context, sys_regs[GCR_EL1]));
   DEFINE(CPU_APIAKEYLO_EL1,	offsetof(struct kvm_cpu_context, sys_regs[APIAKEYLO_EL1]));

index 308100ed25de..4c89931a6a92 100644
@@ -30,8 +30,9 @@ SYM_FUNC_START(__guest_enter)
 	// Store the hyp regs
 	save_callee_saved_regs x1
 
-	// Save hyp's sp_el0
+	// Save hyp's sp_el0 and tpidrro_el0
 	save_sp_el0	x1, x2
+	save_tpidrro_el0 x1, x2
 
 	// Now the hyp state is stored if we have a pending RAS SError it must
 	// affect the host or hyp. If any asynchronous exception is pending we

@@ -66,8 +67,9 @@ alternative_else_nop_endif
 	// when this feature is enabled for kernel code.
 	ptrauth_switch_to_guest x29, x0, x1, x2
 
-	// Restore the guest's sp_el0
+	// Restore the guest's sp_el0 and tpidrro_el0
 	restore_sp_el0 x29, x0
+	restore_tpidrro_el0 x29, x0
 
 	// Restore guest regs x0-x17
 	ldp	x0, x1,   [x29, #CPU_XREG_OFFSET(0)]

@@ -146,8 +148,9 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL)
 	// Store the guest regs x18-x29, lr
 	save_callee_saved_regs x1
 
-	// Store the guest's sp_el0
+	// Store the guest's sp_el0 and tpidrro_el0
 	save_sp_el0	x1, x2
+	save_tpidrro_el0 x1, x2
 
 	adr_this_cpu x2, kvm_hyp_ctxt, x3
 

@@ -161,8 +164,9 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL)
 	// mte_switch_to_hyp(g_ctxt, h_ctxt, reg1)
 	mte_switch_to_hyp x1, x2, x3
 
-	// Restore hyp's sp_el0
+	// Restore hyp's sp_el0 and tpidrro_el0
 	restore_sp_el0 x2, x3
+	restore_tpidrro_el0 x2, x3
 
 	// Now restore the hyp regs
 	restore_callee_saved_regs x2

index c382848d3194..b5f7ef8e6371 100644
@@ -66,7 +66,6 @@ static inline void __sysreg_save_common_state(struct kvm_cpu_context *ctxt)
 static inline void __sysreg_save_user_state(struct kvm_cpu_context *ctxt)
 {
 	ctxt_sys_reg(ctxt, TPIDR_EL0)	= read_sysreg(tpidr_el0);
-	ctxt_sys_reg(ctxt, TPIDRRO_EL0)	= read_sysreg(tpidrro_el0);
 }
 
 static inline bool ctxt_has_mte(struct kvm_cpu_context *ctxt)

@@ -209,7 +208,6 @@ static inline void __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
 static inline void __sysreg_restore_user_state(struct kvm_cpu_context *ctxt)
 {
 	write_sysreg(ctxt_sys_reg(ctxt, TPIDR_EL0),	tpidr_el0);
-	write_sysreg(ctxt_sys_reg(ctxt, TPIDRRO_EL0),	tpidrro_el0);
 }
 
 static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt,

2.55.0.1082.g2b9226bbc0-goog

Yours,
Linus Walleij

To make SP_EL0 available to point at the per-cpu kernel overflow stack, we must relocate 'current' elsewhere. With TPIDRRO_EL0 now restored late in the return-to-user path, we can otherwise use it to hold 'current' and update the various accessors accordingly.

Move 'current' from SP_EL0 to TPIDRRO_EL0.

---
 arch/arm64/include/asm/assembler.h |  2 +-
 arch/arm64/include/asm/current.h   |  6 +++---
 arch/arm64/kernel/entry.S          | 20 ++++++++++----------
 arch/arm64/kernel/head.S           |  2 +-
 arch/arm64/kernel/process.c        |  5 +++--
 arch/arm64/mm/proc.S               |  6 +++++-
 arch/arm64/net/bpf_jit_comp.c      |  4 ++--
 7 files changed, 25 insertions(+), 20 deletions(-)

index 0b58b550e8dc..fd1ae5337935 100644
@@ -587,7 +587,7 @@ alternative_else_nop_endif
  * Return the current task_struct.
  */
 	.macro	get_current_task, rd
-	mrs	\rd, sp_el0
+	mrs	\rd, tpidrro_el0
 	.endm
 
 /*

index c92912eaf186..8c7efac05394 100644
@@ -14,11 +14,11 @@ struct task_struct;
  */
 static __always_inline struct task_struct *get_current(void)
 {
-	unsigned long sp_el0;
+	unsigned long tpidrro_el0;
 
-	asm ("mrs %0, sp_el0" : "=r" (sp_el0));
+	asm ("mrs %0, tpidrro_el0" : "=r" (tpidrro_el0));
 
-	return (struct task_struct *)sp_el0;
+	return (struct task_struct *)tpidrro_el0;
 }
 
 #define current get_current()

index b5d8277f608a..a45be0a837c8 100644
@@ -81,9 +81,9 @@
 	/* Stash the original SP (minus PT_REGS_SIZE) in tpidr_el0. */
 	msr	tpidr_el0, x0
 
-	/* Recover the original x0 value and stash it in tpidrro_el0 */
+	/* Recover the original x0 value and stash it in sp_el0 */
 	sub	x0, sp, x0
-	msr	tpidrro_el0, x0
+	msr	sp_el0, x0
 
 	/* Switch to the overflow stack */
 	adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0

@@ -99,7 +99,7 @@
 
 	/* We were already on the overflow stack. Restore sp/x0 and carry on. */
 	sub	sp, sp, x0
-	mrs	x0, tpidrro_el0
+	mrs	x0, sp_el0
 	b	el\el\ht\()_\regsize\()_\label
 	.endif
 .org .Lventry_start\@ + 128	// Did we overflow the ventry slot?

@@ -226,7 +226,7 @@ alternative_cb_end
 	clear_gp_regs
 	mrs	x21, sp_el0
 	ldr_this_cpu	tsk, __entry_task, x20
-	msr	sp_el0, tsk
+	msr	tpidrro_el0, tsk
 
 	/*
 	 * Ensure MDSCR_EL1.SS is clear, since we can unmask debug exceptions

@@ -548,7 +548,7 @@ SYM_CODE_START_LOCAL(__bad_stack)
 	 */
 
 	/* Restore the original x0 value */
-	mrs	x0, tpidrro_el0
+	mrs	x0, sp_el0
 
 	/*
 	 * Store the original GPRs to the new stack. The orginal SP (minus

@@ -843,7 +843,7 @@ SYM_FUNC_START(cpu_switch_to)
 	ldp	x29, x9, [x8], #16
 	ldr	lr, [x8]
 	mov	sp, x9
-	msr	sp_el0, x1
+	msr	tpidrro_el0, x1
 	ptrauth_keys_install_kernel x1, x8, x9, x10
 	scs_save x0
 	scs_load_current

@@ -1031,11 +1031,11 @@ SYM_CODE_START(__sdei_asm_handler)
 
 	/*
 	 * We may have interrupted userspace, or a guest, or exit-from or
-	 * return-to either of these. We can't trust sp_el0, restore it.
+	 * return-to either of these. We can't trust tpidrro_el0, restore it.
 	 */
-	mrs	x28, sp_el0
+	mrs	x28, tpidrro_el0
 	ldr_this_cpu	dst=x0, sym=__entry_task, tmp=x1
-	msr	sp_el0, x0
+	msr	tpidrro_el0, x0
 
 	/* If we interrupted the kernel point to the previous stack/frame. */
 	and     x0, x3, #0xc

@@ -1051,7 +1051,7 @@ SYM_CODE_START(__sdei_asm_handler)
 	mov	x1, x19
 	bl	__sdei_handler
 
-	msr	sp_el0, x28
+	msr	tpidrro_el0, x28
 	/* restore regs >x17 that we clobbered */
 	mov	x4, x19         // keep x4 for __sdei_asm_exit_trampoline
 	ldp	x28, x29, [x4, #SDEI_EVENT_INTREGS + 16 * 14]

index 87a822e5c4ca..c33551ee6572 100644
@@ -193,7 +193,7 @@ SYM_CODE_END(preserve_boot_args)
 	 * for consistency with user tasks and kthreads.
 	 */
 	.macro	init_cpu_task tsk, tmp1, tmp2
-	msr	sp_el0, \tsk
+	msr	tpidrro_el0, \tsk
 
 	ldr	\tmp1, [\tsk, #TSK_STACK]
 	add	sp, \tmp1, #THREAD_SIZE

index bfdc12166895..79a19be88ced 100644
@@ -551,8 +551,9 @@ static void ssbs_thread_switch(struct task_struct *next)
 }
 
 /*
- * We store our current task in sp_el0, which is clobbered by userspace. Keep a
- * shadow copy so that we can restore this upon entry from userspace.
+ * We store our current task in tpidrro_el0, which is clobbered when returning
+ * to userspace. Keep a shadow copy so that we can restore this upon entry from
+ * userspace.
  *
  * This is *only* for exception entry from EL0, and is not valid until we
  * __switch_to() a user task.

index 12aacc74f764..ad4965d34978 100644
@@ -129,6 +129,10 @@ SYM_FUNC_START(cpu_do_resume)
 	ldp	x9, x10, [x0, #48]
 	ldp	x11, x12, [x0, #64]
 	ldp	x13, x14, [x0, #80]
+
+	/* Move 'current' somewhere safe */
+	mov	x15, x3
+
 	/*
 	 * Restore x18, as it may be used as a platform register, and clear
 	 * the buffer to minimize the risk of exposure when used for shadow

@@ -176,7 +180,7 @@ alternative_if ARM64_HAS_GIC_PRIO_MASKING
 alternative_else_nop_endif
 #endif
 
-	ptrauth_keys_install_kernel_nosync x14, x1, x2, x3
+	ptrauth_keys_install_kernel_nosync x15, x1, x2, x3
 	isb
 	ret
 SYM_FUNC_END(cpu_do_resume)

index c18e005a41db..51d853b92443 100644
@@ -1694,7 +1694,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 		if (insn->src_reg == 0 && insn->imm == BPF_FUNC_get_smp_processor_id) {
 			cpu_offset = offsetof(struct thread_info, cpu);
 
-			emit(A64_MRS_SP_EL0(tmp), ctx);
+			emit(A64_MRS_TPIDRRO_EL0(tmp), ctx);
 			if (is_lsi_offset(cpu_offset, 2)) {
 				emit(A64_LDR32I(r0, tmp, cpu_offset), ctx);
 			} else {

@@ -1707,7 +1707,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 		/* Implement helper call to bpf_get_current_task/_btf() inline */
 		if (insn->src_reg == 0 && (insn->imm == BPF_FUNC_get_current_task ||
 					   insn->imm == BPF_FUNC_get_current_task_btf)) {
-			emit(A64_MRS_SP_EL0(r0), ctx);
+			emit(A64_MRS_TPIDRRO_EL0(r0), ctx);
 			break;
 		}
 

2.55.0.1082.g2b9226bbc0-goog

I just intuitively like this change, if for nothing else then for the fact that SP_EL0 "stack pointer" was a bad register to use for this and something named "thread/process ID register" (something) is a more natural name for something that gets to contain current.

Yours,
Linus Walleij

The current task pointer now lives in TPIDRRO_EL0 instead of SP_EL0, so update the bpf selftests with the new register allocation.

---
 tools/testing/selftests/bpf/progs/verifier_jit_inline.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 02e562f56f9d..a3d6c095acc3 100644
@@ -9,7 +9,7 @@ __success __retval(0)
 __arch_x86_64
 __jited("	addq	%gs:{{.*}}, %rax")
 __arch_arm64
-__jited("	mrs	x8, SP_EL0")
+__jited("	mrs	x8, TPIDRRO_EL0")
 __arch_riscv64
 __jited("	mv	a5, tp")
 __arch_loongarch

2.55.0.1082.g2b9226bbc0-goog

Yours,
Linus Walleij

The current task pointer now lives in TPIDRRO_EL0 instead of SP_EL0, so update the gdb 'cpus.py' helper script with the new register allocation.

---
 scripts/gdb/linux/cpus.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 6edf4ef61636..547a623280ae 100644
@@ -185,7 +185,7 @@ def get_current_task(cpu):
             var_ptr = gdb.parse_and_eval("&current_task")
             return per_cpu(var_ptr, cpu).dereference()
     elif utils.is_target_arch("aarch64"):
-        current_task_addr = gdb.parse_and_eval("(unsigned long)$SP_EL0")
+        current_task_addr = gdb.parse_and_eval("(unsigned long)$TPIDRRO_EL0")
         if (current_task_addr >> 63) != 0:
             current_task = current_task_addr.cast(task_ptr_type)
             return current_task.dereference()

2.55.0.1082.g2b9226bbc0-goog

Yours,
Linus Walleij

Now that 'current' resides in TPIDRRO_EL0, update the per-task stackprotector compiler invocation to use the new register.

Note that this requires a very recent build of clang [1], as (unlike GCC), it historically rejected all registers other than SP_EL0.

---
 arch/arm64/Kconfig  | 3 +--
 arch/arm64/Makefile | 2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)

index b8270b301809..020fcaf36b1b 100644
@@ -2450,12 +2450,11 @@ config RANDOMIZE_MODULE_REGION_FULL
 	  exhaustion, modules might be able to fall back to a larger 2GB area.
 
 config CC_HAVE_STACKPROTECTOR_SYSREG
-	def_bool $(cc-option,-mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=0)
+	def_bool $(cc-option,-mstack-protector-guard=sysreg -mstack-protector-guard-reg=tpidrro_el0 -mstack-protector-guard-offset=0)
 
 config STACKPROTECTOR_PER_TASK
 	def_bool y
 	depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_SYSREG
-	depends on BROKEN
 
 config UNWIND_PATCH_PAC_INTO_SCS
 	bool "Enable shadow call stack dynamically using code patching"

index 6b005c8fef70..a293ab67818f 100644
@@ -72,7 +72,7 @@ ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y)
 prepare: stack_protector_prepare
 stack_protector_prepare: prepare0
 	$(eval KBUILD_CFLAGS += -mstack-protector-guard=sysreg		  \
-				-mstack-protector-guard-reg=sp_el0	  \
+				-mstack-protector-guard-reg=tpidrro_el0	  \
 				-mstack-protector-guard-offset=$(shell	  \
 			awk '{if ($$2 == "TSK_STACK_CANARY") print $$3;}' \
 				$(objtree)/include/generated/asm-offsets.h))

2.55.0.1082.g2b9226bbc0-goog

I get why you need to do this now.

Yours,
Linus Walleij

set_my_cpu_offset() is only called from smp_prepare_boot_cpu() which runs once on the primary CPU and before alternatives have been applied.

Rework the function so that it operates only on the boot CPU offset, drop the unused alternative code sequence and internalise it into smp.c alongside its only caller.

---
 arch/arm64/include/asm/percpu.h | 8 --------
 arch/arm64/kernel/smp.c         | 8 +++++++-
 2 files changed, 7 insertions(+), 9 deletions(-)

index b57b2bb00967..3dc7291b16e9 100644
@@ -12,14 +12,6 @@
 #include <asm/stack_pointer.h>
 #include <asm/sysreg.h>
 
-static inline void set_my_cpu_offset(unsigned long off)
-{
-	asm volatile(ALTERNATIVE("msr tpidr_el1, %0",
-				 "msr tpidr_el2, %0",
-				 ARM64_HAS_VIRT_HOST_EXTN)
-			:: "r" (off) : "memory");
-}
-
 static inline unsigned long __hyp_my_cpu_offset(void)
 {
 	/*

index a61dc3016a11..702dee8d0db5 100644
@@ -446,6 +446,12 @@ void __init smp_cpus_done(unsigned int max_cpus)
 	mark_linear_text_alias_ro();
 }
 
+static void __init set_boot_cpu_offset(void)
+{
+	asm volatile("msr tpidr_el1, %0"
+			:: "r" (per_cpu_offset(0)) : "memory");
+}
+
 void __init smp_prepare_boot_cpu(void)
 {
 	/*

@@ -453,7 +459,7 @@ void __init smp_prepare_boot_cpu(void)
 	 * setup_per_cpu_areas(), and CPU0's boot time per-cpu area will be
 	 * freed shortly, so we must move over to the runtime per-cpu area.
 	 */
-	set_my_cpu_offset(per_cpu_offset(smp_processor_id()));
+	set_boot_cpu_offset();
 
 	cpuinfo_store_boot_cpu();
 	setup_boot_cpu_features();

2.55.0.1082.g2b9226bbc0-goog

Yours,
Linus Walleij

Allow the use of per-cpu variables from noinstr functions by marking __kern_my_cpu_offset() as '__always_inline'. This will later be relied upon to detect stack overflow of the overflow stack from noinstr C code.

---
 arch/arm64/include/asm/percpu.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 3dc7291b16e9..cb9ae8a1ae87 100644
@@ -21,7 +21,7 @@ static inline unsigned long __hyp_my_cpu_offset(void)
 	return read_sysreg(tpidr_el2);
 }
 
-static inline unsigned long __kern_my_cpu_offset(void)
+static __always_inline unsigned long __kern_my_cpu_offset(void)
 {
 	unsigned long off;
 

2.55.0.1082.g2b9226bbc0-goog

Yours,
Linus Walleij

__finalise_el2() unconditionally returns to EL2h when VHE is being used. In preparation for handling callers from both EL1h and EL1t, preserve the handler/thread bit of the calling mode when rewriting the exception level in the SPSR.

---
 arch/arm64/include/uapi/asm/ptrace.h | 1 +
 arch/arm64/kernel/hyp-stub.S         | 7 ++++---
 2 files changed, 5 insertions(+), 3 deletions(-)

index 15649a253a57..25d9d9a0a6f7 100644
@@ -37,6 +37,7 @@
 #define PSR_MODE_EL3t	0x0000000c
 #define PSR_MODE_EL3h	0x0000000d
 #define PSR_MODE_MASK	0x0000000f
+#define PSR_MODE_h_BIT	0x00000001
 
 /* AArch32 CPSR bits */
 #define PSR_MODE32_BIT		0x00000010

index 37c6976e44a4..f61778cccd3c 100644
@@ -149,10 +149,11 @@ SYM_CODE_START_LOCAL(__finalise_el2)
 
 	isb
 
-	// Hack the exception return to stay at EL2
+	// Hack the exception return to stay at EL2 ...
 	mrs	x0, spsr_el1
-	and	x0, x0, #~PSR_MODE_MASK
-	mov	x1, #PSR_MODE_EL2h
+	// ... preserving the 'h' bit so we don't swizzle SP.
+	and	x0, x0, #(~PSR_MODE_MASK | PSR_MODE_h_BIT)
+	mov	x1, #PSR_MODE_EL2t
 	orr	x0, x0, x1
 	msr	spsr_el1, x0
 

2.55.0.1082.g2b9226bbc0-goog

The SDEI header includes are a bit of a mess: linux/arm_sdei.h only includes asm/sdei.h if CONFIG_ARM_SDE_INTERFACE=y and then goes on to provide a definition of the sdei_arch_get_entry_point() macro if it hasn't already been defined. asm/sdei.h, however, defines the same macro unconditionally and so this breaks the build if CONFIG_ARM_SDE_INTERFACE is not set and asm/sdei.h is included after linux/arm_sdei.h.

The kbuild robot ran into this situation with randconfig testing, thanks to the transitive inclusion of asm/sdei.h via asm/stacktrace.h.

Guard most of asm/sdei.h with CONFIG_ARM_SDE_INTERFACE to avoid redefining macros provided by linux/arm_sdei.h.

---
 arch/arm64/include/asm/sdei.h | 2 ++
 1 file changed, 2 insertions(+)

index b2248bd3cb58..b96eb9e22d4b 100644
@@ -9,6 +9,7 @@
 
 #define SDEI_STACK_SIZE		IRQ_STACK_SIZE
 
+#ifdef CONFIG_ARM_SDE_INTERFACE
 #ifndef __ASSEMBLER__
 
 #include <linux/linkage.h>

@@ -50,4 +51,5 @@ unsigned long sdei_arch_get_entry_point(int conduit);
 #define sdei_arch_get_entry_point(x)	sdei_arch_get_entry_point(x)
 
 #endif /* __ASSEMBLER__ */
+#endif /* CONFIG_ARM_SDE_INTERFACE */
 #endif	/* __ASM_SDEI_H */

2.55.0.1082.g2b9226bbc0-goog

Section 5.2.2 ("Event resume context") of the SDEI specification states that, as a result of an SDEI_EVENT_COMPLETE_AND_RESUME call:

| SPSR_ELc is set to the PSTATE when the event was taken

In preparation for making use of both thread and handler modes in the kernel, adjust the SDEI event handler so that we return to the correct kernel vector for the interrupted mode.

---
 arch/arm64/kernel/sdei.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

index 778f2a1faac8..6337ce905593 100644
@@ -208,7 +208,7 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 	int i, err = 0;
 	int clobbered_registers = 4;
 	u64 elr = read_sysreg(elr_el1);
-	u32 kernel_mode = read_sysreg(CurrentEL) | 1;	/* +SPSel */
+	u32 kernel_mode = read_sysreg(CurrentEL);
 	unsigned long vbar = read_sysreg(vbar_el1);
 
 	if (arm64_kernel_unmapped_at_el0())

@@ -233,7 +233,7 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 		pr_warn("unsafe: exception during handler\n");
 	}
 
-	mode = regs->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK);
+	mode = regs->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK) & ~PSR_MODE_h_BIT;
 
 	/*
 	 * If we interrupted the kernel with interrupts masked, we always go

@@ -251,7 +251,7 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 	 * address'.
 	 */
 	if (mode == kernel_mode)
-		return vbar + 0x280;
+		return vbar + (regs->pstate & PSR_MODE_h_BIT ? 0x280 : 0x80);
 	else if (mode & PSR_MODE32_BIT)
 		return vbar + 0x680;
 

2.55.0.1082.g2b9226bbc0-goog

With SP_EL0 now available for use in the kernel, point it at the overflow stack so that exception entry from EL1h will later be able to switch to the overflow stack by writing to SPSel.

---
 arch/arm64/kernel/entry.S |  2 ++
 arch/arm64/kernel/head.S  |  3 +++
 arch/arm64/kernel/smp.c   | 12 ++++++++++--
 3 files changed, 15 insertions(+), 2 deletions(-)

index a45be0a837c8..6958ee238649 100644
@@ -227,6 +227,8 @@ alternative_cb_end
 	mrs	x21, sp_el0
 	ldr_this_cpu	tsk, __entry_task, x20
 	msr	tpidrro_el0, tsk
+	adr_this_cpu	x19, overflow_stack + OVERFLOW_STACK_SIZE, x20
+	msr	sp_el0, x19
 
 	/*
 	 * Ensure MDSCR_EL1.SS is clear, since we can unmask debug exceptions

index c33551ee6572..27a19e1a0ee6 100644
@@ -210,6 +210,9 @@ SYM_CODE_END(preserve_boot_args)
 	ldr	w\tmp2, [\tsk, #TSK_TI_CPU]
 	ldr	\tmp1, [\tmp1, \tmp2, lsl #3]
 	set_this_cpu_offset \tmp1
+
+	adr_this_cpu	\tmp1, overflow_stack + OVERFLOW_STACK_SIZE, \tmp2
+	msr	sp_el0, \tmp1
 	.endm
 
 /*

index 702dee8d0db5..4c122d6598de 100644
@@ -51,6 +51,7 @@
 #include <asm/processor.h>
 #include <asm/smp_plat.h>
 #include <asm/sections.h>
+#include <asm/stacktrace.h>
 #include <asm/tlbflush.h>
 #include <asm/ptrace.h>
 #include <asm/virt.h>

@@ -448,8 +449,15 @@ void __init smp_cpus_done(unsigned int max_cpus)
 
 static void __init set_boot_cpu_offset(void)
 {
-	asm volatile("msr tpidr_el1, %0"
-			:: "r" (per_cpu_offset(0)) : "memory");
+	u64 ovf_sp = (u64)raw_cpu_ptr(overflow_stack) + OVERFLOW_STACK_SIZE;
+
+	asm volatile(
+	"	msr	tpidr_el1, %1\n"
+	"	add	%0, %0, %1\n"
+	"	msr	sp_el0, %0" /* Update the overflow stack pointer */
+	: "+r" (ovf_sp)
+	: "r" (per_cpu_offset(0))
+	: "memory");
 }
 
 void __init smp_prepare_boot_cpu(void)

2.55.0.1082.g2b9226bbc0-goog

In preparation for using EL1t to handle exceptions taken whilst running on the overflow stack, implement some simple wrappers around the EL1h handlers which perform a best-effort stack overflow check before proceeding.

---
 arch/arm64/kernel/entry-common.c | 43 +++++++++++++++++++++++++++++---
 1 file changed, 39 insertions(+), 4 deletions(-)

index 72c03ccea59f..9738142780df 100644
@@ -327,10 +327,45 @@ static void debug_exception_exit(struct pt_regs *regs)
 }
 NOKPROBE_SYMBOL(debug_exception_exit);
 
-UNHANDLED(el1t, 64, sync)
-UNHANDLED(el1t, 64, irq)
-UNHANDLED(el1t, 64, fiq)
-UNHANDLED(el1t, 64, error)
+static void noinstr el1t_64_check_overflow_stack(struct pt_regs *regs)
+{
+	unsigned long sp = kernel_stack_pointer(regs) - sizeof(*regs);
+	unsigned long ovf_stack = (unsigned long)this_cpu_ptr(overflow_stack);
+
+	/*
+	 * We're in big trouble if we've overflowed the overflow stack
+	 * so perform a best-effort check before we proceed. If our SP
+	 * is outside of the overflow stack for this CPU then presumably
+	 * we're already corrupting memory, so park ourselves here in an
+	 * attempt to contain the damage.
+	 */
+	if (sp < ovf_stack || sp > ovf_stack + OVERFLOW_STACK_SIZE)
+		cpu_park_loop();
+}
+
+asmlinkage void noinstr el1t_64_sync_handler(struct pt_regs *regs)
+{
+	el1t_64_check_overflow_stack(regs);
+	el1h_64_sync_handler(regs);
+}
+
+asmlinkage void noinstr el1t_64_irq_handler(struct pt_regs *regs)
+{
+	el1t_64_check_overflow_stack(regs);
+	el1h_64_irq_handler(regs);
+}
+
+asmlinkage void noinstr el1t_64_fiq_handler(struct pt_regs *regs)
+{
+	el1t_64_check_overflow_stack(regs);
+	el1h_64_fiq_handler(regs);
+}
+
+asmlinkage void noinstr el1t_64_error_handler(struct pt_regs *regs)
+{
+	el1t_64_check_overflow_stack(regs);
+	el1h_64_error_handler(regs);
+}
 
 static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
 {

2.55.0.1082.g2b9226bbc0-goog

In the commit message, say something about the fact that EL1t handlers architecturally use the stack pointer from SP_EL0 which we have now started to repurpose for the kernel-internal overflow stack and not EL0 (as the name says) at all, and handlers here will by definition be running on the kernel overflow stack, so people understand what is going on here.

If I understood it right:

Yours,
Linus Walleij

When detecting a stack overflow on exception entry from EL1, use SPSel to switch to the overflow stack without corrupting any GPRs. Not only is this simpler than the previous logic, but it also opens the door to more complicated overflow checks (for example, based on per-task stack sizes or stacks which are not aligned to a specific page order) as well as the possibility of returning from a stack fault if we were able to resolve it.

---
 arch/arm64/kernel/entry.S | 69 ++++++++++++---------------------------
 1 file changed, 21 insertions(+), 48 deletions(-)

index 6958ee238649..b4df2f23ebe7 100644
@@ -51,11 +51,11 @@
 	mov	x30, xzr
 	.endif
 .Lskip_tramp_vectors_cleanup\@:
-	.endif
-
+	sub	sp, sp, #PT_REGS_SIZE
+	.else	/* \el == 1 */
+	.ifc	\ht, h
 	sub	sp, sp, #PT_REGS_SIZE
 
-	.if	\el == 1
 	/*
 	 * Test whether the SP has overflowed, without corrupting a GPR.
 	 * Task and IRQ stacks are aligned so that SP & (1 << THREAD_SHIFT)

@@ -63,45 +63,16 @@
 	 */
 	add	sp, sp, x0			// sp' = sp + x0
 	sub	x0, sp, x0			// x0' = sp' - x0 = (sp + x0) - x0 = sp
-	tbnz	x0, #THREAD_SHIFT, 0f
+	tbnz	x0, #THREAD_SHIFT, __bad_stack
 	sub	x0, sp, x0			// x0'' = sp' - x0' = (sp + x0) - sp = x0
 	sub	sp, sp, x0			// sp'' = sp' - x0 = (sp + x0) - x0 = sp
+	.else	/* EL1t */
+	msr	spsel, #0			// Stay on the overflow stack
+	sub	sp, sp, #PT_REGS_SIZE
+	.endif
 	.endif
 
 	b	el\el\ht\()_\regsize\()_\label
-
-	.if	\el == 1
-0:
-	/*
-	 * Either we've just detected an overflow, or we've taken an exception
-	 * while on the overflow stack. Either way, we won't return to
-	 * userspace, and can clobber EL0 registers to free up GPRs.
-	 */
-
-	/* Stash the original SP (minus PT_REGS_SIZE) in tpidr_el0. */
-	msr	tpidr_el0, x0
-
-	/* Recover the original x0 value and stash it in sp_el0 */
-	sub	x0, sp, x0
-	msr	sp_el0, x0
-
-	/* Switch to the overflow stack */
-	adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0
-
-	/*
-	 * Check whether we were already on the overflow stack. This may happen
-	 * after panic() re-enables interrupts.
-	 */
-	mrs	x0, tpidr_el0			// sp of interrupted context
-	sub	x0, sp, x0			// delta with top of overflow stack
-	tst	x0, #~(OVERFLOW_STACK_SIZE - 1)	// within range?
-	b.ne	__bad_stack			// no? -> bad stack pointer
-
-	/* We were already on the overflow stack. Restore sp/x0 and carry on. */
-	sub	sp, sp, x0
-	mrs	x0, sp_el0
-	b	el\el\ht\()_\regsize\()_\label
-	.endif
 .org .Lventry_start\@ + 128	// Did we overflow the ventry slot?
 	.endm
 

@@ -544,22 +515,24 @@ SYM_CODE_END(vectors)
 
 SYM_CODE_START_LOCAL(__bad_stack)
 	/*
-	 * We detected an overflow in kernel_ventry, which switched to the
-	 * overflow stack. Stash the exception regs, and head to our overflow
-	 * handler.
+	 * We detected an overflow in kernel_ventry.
+	 * Restore SP and X0.
 	 */
+	sub	x0, sp, x0
+	sub	sp, sp, x0
+	add	sp, sp, #PT_REGS_SIZE
 
-	/* Restore the original x0 value */
-	mrs	x0, sp_el0
+	/* Switch to the overflow stack */
+	msr	spsel, #0
 
-	/*
-	 * Store the original GPRs to the new stack. The orginal SP (minus
-	 * PT_REGS_SIZE) was stashed in tpidr_el0 by kernel_ventry.
-	 */
+	/* Stash the exception regs */
 	sub	sp, sp, #PT_REGS_SIZE
 	kernel_entry 1
-	mrs	x0, tpidr_el0
-	add	x0, x0, #PT_REGS_SIZE
+
+	/* Fix-up the saved SP */
+	msr	spsel, #1
+	mov	x0, sp
+	msr	spsel, #0
 	str	x0, [sp, #S_SP]
 
 	/* Stash the regs for handle_bad_stack */

2.55.0.1082.g2b9226bbc0-goog

I really like what I see here. The change up to this point stands alone as a serious stability improvement because you are not deleting code that made me feel very icky when I was reading it. Before this patch it was pretty much an
"overflow and crash gracefully" stack, not an overflow stack.

Yours,
Linus Walleij

Handling entry from EL0, EL1h and EL1t in the 'kernel_ventry' assembly macro is getting a bit unwieldy as the control flow becomes increasingly dependent on the macro arguments.

Split 'kernel_ventry' into three leaf helper macros for EL0, EL1h and EL1t, which are then dispatched by simple wrappers.

---
 arch/arm64/kernel/entry.S | 29 +++++++++++++++++++----------
 1 file changed, 19 insertions(+), 10 deletions(-)

index b4df2f23ebe7..38f9327e6a0a 100644
@@ -36,10 +36,7 @@
 	.endr
 	.endm
 
-	.macro kernel_ventry, el:req, ht:req, regsize:req, label:req
-	.align 7
-.Lventry_start\@:
-	.if	\el == 0
+	.macro kernel_ventry_el0, ht:req, regsize:req, label:req
 	/*
 	 * This must be the first instruction of the EL0 vector entries. It is
 	 * skipped by the trampoline vectors, to trigger the cleanup.

@@ -52,8 +49,10 @@
 	.endif
 .Lskip_tramp_vectors_cleanup\@:
 	sub	sp, sp, #PT_REGS_SIZE
-	.else	/* \el == 1 */
-	.ifc	\ht, h
+	b	el0\ht\()_\regsize\()_\label
+	.endm
+
+	.macro kernel_ventry_el1h, regsize:req, label:req
 	sub	sp, sp, #PT_REGS_SIZE
 
 	/*

@@ -66,13 +65,23 @@
 	tbnz	x0, #THREAD_SHIFT, __bad_stack
 	sub	x0, sp, x0			// x0'' = sp' - x0' = (sp + x0) - sp = x0
 	sub	sp, sp, x0			// sp'' = sp' - x0 = (sp + x0) - x0 = sp
-	.else	/* EL1t */
+	b	el1h_\regsize\()_\label
+	.endm
+
+	.macro kernel_ventry_el1t, regsize:req, label:req
 	msr	spsel, #0			// Stay on the overflow stack
 	sub	sp, sp, #PT_REGS_SIZE
-	.endif
-	.endif
+	b	el1t_\regsize\()_\label
+	.endm
 
-	b	el\el\ht\()_\regsize\()_\label
+	.macro kernel_ventry, el:req, ht:req, regsize:req, label:req
+	.align 7
+.Lventry_start\@:
+	.if	\el == 0
+	kernel_ventry_el0	\ht, \regsize, \label
+	.else
+	kernel_ventry_el1\ht	\regsize, \label
+	.endif
 .org .Lventry_start\@ + 128	// Did we overflow the ventry slot?
 	.endm
 

2.55.0.1082.g2b9226bbc0-goog

I agree with this change, this is easier to read and understand.

Yours,
Linus Walleij

With the kernel stack pointer in SP_EL1 and the overflow stack pointer in SP_EL0, it is now straightforward to switch between the two on exception entry from EL1 by writing to SPSel. However, since exception entry sets PSTATE.SP to 1 (selecting SP_EL1 as the stack pointer), repurposing the overflow stack as a more general kernel exception stack would require writing to SPSel on every exception entry from the kernel.

Switch things around so that the overflow stack resides in SP_EL1, with the kernel stack residing in SP_EL0.

---
 arch/arm64/include/asm/kvm_asm.h | 16 +++++++++++++++
 arch/arm64/kernel/entry-common.c | 34 ++++++++++++++++----------------
 arch/arm64/kernel/entry.S        | 26 +++++++++++++-----------
 arch/arm64/kernel/head.S         |  5 ++++-
 arch/arm64/kernel/sleep.S        |  3 ++-
 arch/arm64/kernel/smp.c          |  8 +++++---
 arch/arm64/kvm/hyp/entry.S       |  3 +++
 arch/arm64/mm/proc.S             | 10 ++++++++--
 8 files changed, 70 insertions(+), 35 deletions(-)

index 1cfe9216a2ff..bde046c3ca79 100644
@@ -398,6 +398,22 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt,
 	ldp	x29, lr,  [\ctxt, #CPU_XREG_OFFSET(29)]
 .endm
 
+#ifdef __KVM_VHE_HYPERVISOR__
+.macro activate_exception_stack
+msr	spsel, #1
+.endm
+
+.macro deactivate_exception_stack
+msr	spsel, #0
+.endm
+#else
+.macro activate_exception_stack
+.endm
+
+.macro deactivate_exception_stack
+.endm
+#endif
+
 .macro save_sp_el0 ctxt, tmp
 	mrs	\tmp,	sp_el0
 	str	\tmp,	[\ctxt, #CPU_SP_EL0_OFFSET]

index 9738142780df..9d734cd09f62 100644
@@ -327,7 +327,7 @@ static void debug_exception_exit(struct pt_regs *regs)
 }
 NOKPROBE_SYMBOL(debug_exception_exit);
 
-static void noinstr el1t_64_check_overflow_stack(struct pt_regs *regs)
+static void noinstr el1h_64_check_overflow_stack(struct pt_regs *regs)
 {
 	unsigned long sp = kernel_stack_pointer(regs) - sizeof(*regs);
 	unsigned long ovf_stack = (unsigned long)this_cpu_ptr(overflow_stack);

@@ -343,28 +343,28 @@ static void noinstr el1t_64_check_overflow_stack(struct pt_regs *regs)
 		cpu_park_loop();
 }
 
-asmlinkage void noinstr el1t_64_sync_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
 {
-	el1t_64_check_overflow_stack(regs);
-	el1h_64_sync_handler(regs);
+	el1h_64_check_overflow_stack(regs);
+	el1t_64_sync_handler(regs);
 }
 
-asmlinkage void noinstr el1t_64_irq_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
 {
-	el1t_64_check_overflow_stack(regs);
-	el1h_64_irq_handler(regs);
+	el1h_64_check_overflow_stack(regs);
+	el1t_64_irq_handler(regs);
 }
 
-asmlinkage void noinstr el1t_64_fiq_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
 {
-	el1t_64_check_overflow_stack(regs);
-	el1h_64_fiq_handler(regs);
+	el1h_64_check_overflow_stack(regs);
+	el1t_64_fiq_handler(regs);
 }
 
-asmlinkage void noinstr el1t_64_error_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
 {
-	el1t_64_check_overflow_stack(regs);
-	el1h_64_error_handler(regs);
+	el1h_64_check_overflow_stack(regs);
+	el1t_64_error_handler(regs);
 }
 
 static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)

@@ -494,7 +494,7 @@ static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr)
 	arm64_exit_to_kernel_mode(regs, state);
 }
 
-asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1t_64_sync_handler(struct pt_regs *regs)
 {
 	unsigned long esr = read_sysreg(esr_el1);
 

@@ -578,17 +578,17 @@ static void noinstr el1_interrupt(struct pt_regs *regs,
 		__el1_irq(regs, handler);
 }
 
-asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1t_64_irq_handler(struct pt_regs *regs)
 {
 	el1_interrupt(regs, handle_arch_irq);
 }
 
-asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1t_64_fiq_handler(struct pt_regs *regs)
 {
 	el1_interrupt(regs, handle_arch_fiq);
 }
 
-asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1t_64_error_handler(struct pt_regs *regs)
 {
 	unsigned long esr = read_sysreg(esr_el1);
 	irqentry_state_t state;

index 38f9327e6a0a..afcd84510daf 100644
@@ -54,6 +54,12 @@
 
 	.macro kernel_ventry_el1h, regsize:req, label:req
 	sub	sp, sp, #PT_REGS_SIZE
+	b	el1h_\regsize\()_\label
+	.endm
+
+	.macro kernel_ventry_el1t, regsize:req, label:req
+	msr	spsel, #0			// Stay on the kernel stack
+	sub	sp, sp, #PT_REGS_SIZE
 
 	/*
 	 * Test whether the SP has overflowed, without corrupting a GPR.

@@ -65,12 +71,6 @@
 	tbnz	x0, #THREAD_SHIFT, __bad_stack
 	sub	x0, sp, x0			// x0'' = sp' - x0' = (sp + x0) - sp = x0
 	sub	sp, sp, x0			// sp'' = sp' - x0 = (sp + x0) - x0 = sp
-	b	el1h_\regsize\()_\label
-	.endm
-
-	.macro kernel_ventry_el1t, regsize:req, label:req
-	msr	spsel, #0			// Stay on the overflow stack
-	sub	sp, sp, #PT_REGS_SIZE
 	b	el1t_\regsize\()_\label
 	.endm
 

@@ -208,7 +208,10 @@ alternative_cb_end
 	ldr_this_cpu	tsk, __entry_task, x20
 	msr	tpidrro_el0, tsk
 	adr_this_cpu	x19, overflow_stack + OVERFLOW_STACK_SIZE, x20
-	msr	sp_el0, x19
+	mov	x20, sp
+	msr	sp_el0, x20
+	mov	sp, x19
+	msr	spsel, #0
 
 	/*
 	 * Ensure MDSCR_EL1.SS is clear, since we can unmask debug exceptions

@@ -346,6 +349,9 @@ alternative_else_nop_endif
 
 	.if	\el == 0
 	ldr	x23, [sp, #S_SP]		// load return stack pointer
+	mov	x0, sp
+	msr	spsel, #1
+	mov	sp, x0
 	msr	sp_el0, x23
 	msr	tpidrro_el0, xzr
 	tst	x22, #PSR_MODE32_BIT		// native task?

@@ -532,16 +538,14 @@ SYM_CODE_START_LOCAL(__bad_stack)
 	add	sp, sp, #PT_REGS_SIZE
 
 	/* Switch to the overflow stack */
-	msr	spsel, #0
+	msr	spsel, #1
 
 	/* Stash the exception regs */
 	sub	sp, sp, #PT_REGS_SIZE
 	kernel_entry 1
 
 	/* Fix-up the saved SP */
-	msr	spsel, #1
-	mov	x0, sp
-	msr	spsel, #0
+	mrs	x0, sp_el0
 	str	x0, [sp, #S_SP]
 
 	/* Stash the regs for handle_bad_stack */

index 27a19e1a0ee6..7419e1d8b7ec 100644
@@ -211,8 +211,11 @@ SYM_CODE_END(preserve_boot_args)
 	ldr	\tmp1, [\tmp1, \tmp2, lsl #3]
 	set_this_cpu_offset \tmp1
 
-	adr_this_cpu	\tmp1, overflow_stack + OVERFLOW_STACK_SIZE, \tmp2
+	mov	\tmp1, sp
 	msr	sp_el0, \tmp1
+	adr_this_cpu	\tmp1, overflow_stack + OVERFLOW_STACK_SIZE, \tmp2
+	mov	sp, \tmp1
+	msr	spsel, #0
 	.endm
 
 /*

index f093cdf71be1..facf3f1cc3b1 100644
@@ -133,7 +133,8 @@ SYM_FUNC_START(_cpu_resume)
 	add	x0, x0, #SLEEP_STACK_DATA_SYSTEM_REGS
 	/* load sp from context */
 	ldr	x2, [x0, #CPU_CTX_SP]
-	mov	sp, x2
+	msr	sp_el0, x2
+
 	/*
 	 * cpu_do_resume expects x0 to contain context address pointer
 	 */

index 4c122d6598de..b2e778aae881 100644
@@ -447,14 +447,16 @@ void __init smp_cpus_done(unsigned int max_cpus)
 	mark_linear_text_alias_ro();
 }
 
-static void __init set_boot_cpu_offset(void)
+static void __init update_boot_cpu_offset_and_overflow_stack(void)
 {
 	u64 ovf_sp = (u64)raw_cpu_ptr(overflow_stack) + OVERFLOW_STACK_SIZE;
 
 	asm volatile(
 	"	msr	tpidr_el1, %1\n"
 	"	add	%0, %0, %1\n"
-	"	msr	sp_el0, %0" /* Update the overflow stack pointer */
+	"	msr	spsel, #1\n"
+	"	mov	sp, %0\n" /* Update the overflow stack pointer */
+	"	msr	spsel, #0"
 	: "+r" (ovf_sp)
 	: "r" (per_cpu_offset(0))
 	: "memory");

@@ -467,7 +469,7 @@ void __init smp_prepare_boot_cpu(void)
 	 * setup_per_cpu_areas(), and CPU0's boot time per-cpu area will be
 	 * freed shortly, so we must move over to the runtime per-cpu area.
 	 */
-	set_boot_cpu_offset();
+	update_boot_cpu_offset_and_overflow_stack();
 
 	cpuinfo_store_boot_cpu();
 	setup_boot_cpu_features();

index 4c89931a6a92..4d1205d04383 100644
@@ -31,6 +31,7 @@ SYM_FUNC_START(__guest_enter)
 	save_callee_saved_regs x1
 
 	// Save hyp's sp_el0 and tpidrro_el0
+	activate_exception_stack
 	save_sp_el0	x1, x2
 	save_tpidrro_el0 x1, x2
 

@@ -50,6 +51,7 @@ alternative_else_nop_endif
 	// that would usually be synchonized by the ERET.
 	isb
 	mov	x0, #ARM_EXCEPTION_IRQ
+	deactivate_exception_stack
 	ret
 
 1:

@@ -167,6 +169,7 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL)
 	// Restore hyp's sp_el0 and tpidrro_el0
 	restore_sp_el0 x2, x3
 	restore_tpidrro_el0 x2, x3
+	deactivate_exception_stack
 
 	// Now restore the hyp regs
 	restore_callee_saved_regs x2

index ad4965d34978..bec7f858fae9 100644
@@ -87,6 +87,7 @@
  * This must be kept in sync with struct cpu_suspend_ctx in <asm/suspend.h>.
  */
 SYM_FUNC_START(cpu_do_suspend)
+	msr	spsel, #1
 	mrs	x2, tpidr_el0
 	mrs	x3, tpidrro_el0
 	mrs	x4, contextidr_el1

@@ -98,7 +99,7 @@ SYM_FUNC_START(cpu_do_suspend)
 	mrs	x10, oslsr_el1
 	mrs	x11, sctlr_el1
 	get_this_cpu_offset x12
-	mrs	x13, sp_el0
+	mov	x13, sp				// SP_EL1
 	stp	x2, x3, [x0]
 	stp	x4, x5, [x0, #16]
 	stp	x6, x7, [x0, #32]

@@ -114,6 +115,7 @@ alternative_if ARM64_HAS_TCR2
 	mrs	x2, REG_TCR2_EL1
 	str	x2, [x0, #104]
 alternative_else_nop_endif
+	msr	spsel, #0
 	ret
 SYM_FUNC_END(cpu_do_suspend)
 

@@ -121,6 +123,8 @@ SYM_FUNC_END(cpu_do_suspend)
  * cpu_do_resume - restore CPU register context
  *
  * x0: Address of context pointer
+ *
+ * Entered with SPSel == 1, returns with SPSel == 0.
  */
 SYM_FUNC_START(cpu_do_resume)
 	ldp	x2, x3, [x0]

@@ -159,7 +163,8 @@ alternative_else_nop_endif
 
 	msr	sctlr_el1, x12
 	set_this_cpu_offset x13
-	msr	sp_el0, x14
+	mov	sp, x14				// SP_EL1
+
 	/*
 	 * Restore oslsr_el1 by writing oslar_el1
 	 */

@@ -182,6 +187,7 @@ alternative_else_nop_endif
 
 	ptrauth_keys_install_kernel_nosync x15, x1, x2, x3
 	isb
+	msr	spsel, #0
 	ret
 SYM_FUNC_END(cpu_do_resume)
 #endif

2.55.0.1082.g2b9226bbc0-goog

Hi will, sigh.

https://lore.kernel.org/all/[email protected]/

Please be careful this won't conflict, I'll resubmit this.

This patch gives me panic, and as can be seen from the review comments need some love...

I think maybe we can merge the initial part of the patch series first (up until the overflow stack gets worthy of its name at least) for one kernel release and continue from there after we hashed out that part?

That said I see no direct problem with this approach, other than it may have side effects on some expected behaviours we cannot foresee.

Yours,
Linus Walleij

Please can you share the panic message? I'll definitely be reworking this based on the review comments, but if there's a bug it would be great to see the details now!

Thanks,

Will

Haha. I was not referring to the kernel function but the emotion ;D

Yours,
Linus Walleij

Oh! That kind of panic is easier to fix :) Are you going to LPC?

Will

Sadly no. This year looks great but I'm just not in travelling mode for the moment. But later!

Yours,
Linus Walleij

When synthesising a kernel regs structure for ftrace or perf, advertise a mode of EL1t to reflect the mode in which the kernel runs for the vast majority of the time.

---
 arch/arm64/include/asm/ftrace.h     | 4 ++--
 arch/arm64/include/asm/perf_event.h | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

index 1621c84f44b3..d55a019cdf0a 100644
@@ -153,7 +153,7 @@ ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs)
 	regs->pc = afregs->pc;
 	regs->regs[29] = afregs->fp;
 	regs->regs[30] = afregs->lr;
-	regs->pstate = PSR_MODE_EL1h;
+	regs->pstate = PSR_MODE_EL1t;
 	return regs;
 }
 

@@ -161,7 +161,7 @@ ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs)
 		(_regs)->pc = arch_ftrace_regs(fregs)->pc;			\
 		(_regs)->regs[29] = arch_ftrace_regs(fregs)->fp;		\
 		(_regs)->sp = arch_ftrace_regs(fregs)->sp;			\
-		(_regs)->pstate = PSR_MODE_EL1h;		\
+		(_regs)->pstate = PSR_MODE_EL1t;		\
 	} while (0)
 
 int ftrace_regs_query_register_offset(const char *name);

index ee45b4e77347..b3e7c24a5aab 100644
@@ -17,7 +17,7 @@
 	(regs)->pc = (__ip);    \
 	(regs)->regs[29] = (unsigned long) __builtin_frame_address(0); \
 	(regs)->sp = current_stack_pointer; \
-	(regs)->pstate = PSR_MODE_EL1h;	\
+	(regs)->pstate = PSR_MODE_EL1t;	\
 }
 
 #endif

2.55.0.1082.g2b9226bbc0-goog

For consistency, not that it makes any difference in practice, should we also do:

index 79a19be88ced..c7f954f1ddbf 100644
@@ -489,7 +489,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 		 * be initialized by start_thread() or start_compat_thread().
 		 */
 		memset(childregs, 0, sizeof(struct pt_regs));
-		childregs->pstate = PSR_MODE_EL1h | PSR_IL_BIT;
+		childregs->pstate = PSR_MODE_EL1t | PSR_IL_BIT;
 		childregs->stackframe.type = FRAME_META_TYPE_FINAL;
 
 		p->thread.cpu_context.x19 = (unsigned long)args->fn;

Catalin

Now that the kernel "overflow" stack resides in SP_EL1, it is the active stack for any exception taken from the kernel. Even though we currently switch immediately to the kernel stack in the EL1t entry code, rename the "overflow_stack" to "kernel_exception_stack" to better reflect its usage. Introduce NVHE_OVERFLOW_STACK_SIZE to keep the EL2 overflow stack size distinct from the EL1 kernel exception stack size.

---
 arch/arm64/include/asm/memory.h          | 10 ++++++----
 arch/arm64/include/asm/stacktrace.h      |  9 +++++----
 arch/arm64/include/asm/stacktrace/nvhe.h |  3 +--
 arch/arm64/kernel/entry-common.c         | 18 +++++++++---------
 arch/arm64/kernel/entry.S                |  2 +-
 arch/arm64/kernel/head.S                 |  2 +-
 arch/arm64/kernel/smp.c                  |  9 +++++----
 arch/arm64/kernel/stacktrace.c           |  2 +-
 arch/arm64/kernel/traps.c                | 14 +++++++-------
 arch/arm64/kvm/hyp/nvhe/host.S           |  2 +-
 arch/arm64/kvm/hyp/nvhe/stacktrace.c     |  4 ++--
 arch/arm64/kvm/stacktrace.c              |  4 ++--
 12 files changed, 41 insertions(+), 38 deletions(-)

index a2b7a33966ff..93ce6ef65573 100644
@@ -139,10 +139,11 @@
 
 #define IRQ_STACK_SIZE		THREAD_SIZE
 
-#define OVERFLOW_STACK_SIZE	SZ_4K
+#define KERNEL_EXC_STACK_SIZE	SZ_4K
 
-#define NVHE_STACK_SHIFT       PAGE_SHIFT
-#define NVHE_STACK_SIZE        (UL(1) << NVHE_STACK_SHIFT)
+#define NVHE_OVERFLOW_STACK_SIZE	SZ_4K
+#define NVHE_STACK_SHIFT		PAGE_SHIFT
+#define NVHE_STACK_SIZE			(UL(1) << NVHE_STACK_SHIFT)
 
 /*
  * With the minimum frame size of [x29, x30], exactly half the combined

@@ -150,7 +151,8 @@
  * save the unwinded stacktrace; plus an additional entry to delimit the
  * end.
  */
-#define NVHE_STACKTRACE_SIZE	((OVERFLOW_STACK_SIZE + NVHE_STACK_SIZE) / 2 + sizeof(long))
+#define NVHE_STACKTRACE_SIZE	((NVHE_OVERFLOW_STACK_SIZE + NVHE_STACK_SIZE) \
+				 / 2 + sizeof(long))
 
 /*
  * Alignment of kernel segments (e.g. .text, .data).

index 6d3280932bf5..eb18d7cee62f 100644
@@ -59,12 +59,13 @@ static inline bool on_task_stack(const struct task_struct *tsk,
 
 #define on_thread_stack()	(on_task_stack(current, current_stack_pointer, 1))
 
-DECLARE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack);
+DECLARE_PER_CPU(unsigned long [KERNEL_EXC_STACK_SIZE/sizeof(long)],
+		kernel_exception_stack);
 
-static inline struct stack_info stackinfo_get_overflow(void)
+static inline struct stack_info stackinfo_get_kernel_exception(void)
 {
-	unsigned long low = (unsigned long)raw_cpu_ptr(overflow_stack);
-	unsigned long high = low + OVERFLOW_STACK_SIZE;
+	unsigned long low = (unsigned long)raw_cpu_ptr(kernel_exception_stack);
+	unsigned long high = low + KERNEL_EXC_STACK_SIZE;
 
 	return (struct stack_info) {
 		.low = low,

index 171f9edef49f..629dd2d2caa3 100644
@@ -44,8 +44,7 @@ static inline void kvm_nvhe_unwind_init(struct unwind_state *state,
  * In non-protected mode, the unwinding is done from kernel proper context
  * (by the host in EL1).
  */
-
-DECLARE_KVM_NVHE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack);
+DECLARE_KVM_NVHE_PER_CPU(unsigned long [NVHE_OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack);
 DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_stacktrace_info, kvm_stacktrace_info);
 DECLARE_PER_CPU(unsigned long, kvm_arm_hyp_stack_base);
 

index 9d734cd09f62..7e9b26cf1bdc 100644
@@ -327,43 +327,43 @@ static void debug_exception_exit(struct pt_regs *regs)
 }
 NOKPROBE_SYMBOL(debug_exception_exit);
 
-static void noinstr el1h_64_check_overflow_stack(struct pt_regs *regs)
+static void noinstr el1h_64_check_exception_stack(struct pt_regs *regs)
 {
 	unsigned long sp = kernel_stack_pointer(regs) - sizeof(*regs);
-	unsigned long ovf_stack = (unsigned long)this_cpu_ptr(overflow_stack);
+	unsigned long exc_stack = (unsigned long)this_cpu_ptr(kernel_exception_stack);
 
 	/*
-	 * We're in big trouble if we've overflowed the overflow stack
+	 * We're in big trouble if we've overflowed the exception stack
 	 * so perform a best-effort check before we proceed. If our SP
-	 * is outside of the overflow stack for this CPU then presumably
+	 * is outside of the exceptoin stack for this CPU then presumably
 	 * we're already corrupting memory, so park ourselves here in an
 	 * attempt to contain the damage.
 	 */
-	if (sp < ovf_stack || sp > ovf_stack + OVERFLOW_STACK_SIZE)
+	if (sp < exc_stack || sp > exc_stack + KERNEL_EXC_STACK_SIZE)
 		cpu_park_loop();
 }
 
 asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
 {
-	el1h_64_check_overflow_stack(regs);
+	el1h_64_check_exception_stack(regs);
 	el1t_64_sync_handler(regs);
 }
 
 asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
 {
-	el1h_64_check_overflow_stack(regs);
+	el1h_64_check_exception_stack(regs);
 	el1t_64_irq_handler(regs);
 }
 
 asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
 {
-	el1h_64_check_overflow_stack(regs);
+	el1h_64_check_exception_stack(regs);
 	el1t_64_fiq_handler(regs);
 }
 
 asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
 {
-	el1h_64_check_overflow_stack(regs);
+	el1h_64_check_exception_stack(regs);
 	el1t_64_error_handler(regs);
 }
 

index afcd84510daf..a31ef890a2ee 100644
@@ -207,7 +207,7 @@ alternative_cb_end
 	mrs	x21, sp_el0
 	ldr_this_cpu	tsk, __entry_task, x20
 	msr	tpidrro_el0, tsk
-	adr_this_cpu	x19, overflow_stack + OVERFLOW_STACK_SIZE, x20
+	adr_this_cpu	x19, kernel_exception_stack + KERNEL_EXC_STACK_SIZE, x20
 	mov	x20, sp
 	msr	sp_el0, x20
 	mov	sp, x19

index 7419e1d8b7ec..794cefaf8482 100644
@@ -213,7 +213,7 @@ SYM_CODE_END(preserve_boot_args)
 
 	mov	\tmp1, sp
 	msr	sp_el0, \tmp1
-	adr_this_cpu	\tmp1, overflow_stack + OVERFLOW_STACK_SIZE, \tmp2
+	adr_this_cpu	\tmp1, kernel_exception_stack + KERNEL_EXC_STACK_SIZE, \tmp2
 	mov	sp, \tmp1
 	msr	spsel, #0
 	.endm

index b2e778aae881..5aaeda5362f7 100644
@@ -447,9 +447,10 @@ void __init smp_cpus_done(unsigned int max_cpus)
 	mark_linear_text_alias_ro();
 }
 
-static void __init update_boot_cpu_offset_and_overflow_stack(void)
+static void __init update_boot_cpu_offset_and_exception_stack(void)
 {
-	u64 ovf_sp = (u64)raw_cpu_ptr(overflow_stack) + OVERFLOW_STACK_SIZE;
+	u64 exc_sp = (u64)raw_cpu_ptr(kernel_exception_stack) +
+		     KERNEL_EXC_STACK_SIZE;
 
 	asm volatile(
 	"	msr	tpidr_el1, %1\n"

@@ -457,7 +458,7 @@ static void __init update_boot_cpu_offset_and_overflow_stack(void)
 	"	msr	spsel, #1\n"
 	"	mov	sp, %0\n" /* Update the overflow stack pointer */
 	"	msr	spsel, #0"
-	: "+r" (ovf_sp)
+	: "+r" (exc_sp)
 	: "r" (per_cpu_offset(0))
 	: "memory");
 }

@@ -469,7 +470,7 @@ void __init smp_prepare_boot_cpu(void)
 	 * setup_per_cpu_areas(), and CPU0's boot time per-cpu area will be
 	 * freed shortly, so we must move over to the runtime per-cpu area.
 	 */
-	update_boot_cpu_offset_and_overflow_stack();
+	update_boot_cpu_offset_and_exception_stack();
 
 	cpuinfo_store_boot_cpu();
 	setup_boot_cpu_features();

index 3ebcf8c53fb0..8e65b81c7502 100644
@@ -337,7 +337,7 @@ kunwind_stack_walk(kunwind_consume_fn consume_state,
 	struct stack_info stacks[] = {
 		stackinfo_get_task(task),
 		STACKINFO_CPU(irq),
-		STACKINFO_CPU(overflow),
+		STACKINFO_CPU(kernel_exception),
 #if defined(CONFIG_ARM_SDE_INTERFACE)
 		STACKINFO_SDEI(normal),
 		STACKINFO_SDEI(critical),

index 914282016069..ce0e213d772d 100644
@@ -897,14 +897,14 @@ void bad_el0_sync(struct pt_regs *regs, int reason, unsigned long esr)
 			      "Bad EL0 synchronous exception");
 }
 
-DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)
-	__aligned(16);
+DEFINE_PER_CPU(unsigned long [KERNEL_EXC_STACK_SIZE/sizeof(long)],
+	       kernel_exception_stack) __aligned(16);
 
 void __noreturn panic_bad_stack(struct pt_regs *regs, unsigned long esr, unsigned long far)
 {
 	unsigned long tsk_stk = (unsigned long)current->stack;
 	unsigned long irq_stk = (unsigned long)this_cpu_read(irq_stack_ptr);
-	unsigned long ovf_stk = (unsigned long)this_cpu_ptr(overflow_stack);
+	unsigned long exc_stk = (unsigned long)this_cpu_ptr(kernel_exception_stack);
 
 	console_verbose();
 	pr_emerg("Insufficient stack space to handle exception!");

@@ -912,12 +912,12 @@ void __noreturn panic_bad_stack(struct pt_regs *regs, unsigned long esr, unsigne
 	pr_emerg("ESR: 0x%016lx -- %s\n", esr, esr_get_class_string(esr));
 	pr_emerg("FAR: 0x%016lx\n", far);
 
-	pr_emerg("Task stack:     [0x%016lx..0x%016lx]\n",
+	pr_emerg("Task stack:      [0x%016lx..0x%016lx]\n",
 		 tsk_stk, tsk_stk + THREAD_SIZE);
-	pr_emerg("IRQ stack:      [0x%016lx..0x%016lx]\n",
+	pr_emerg("IRQ stack:       [0x%016lx..0x%016lx]\n",
 		 irq_stk, irq_stk + IRQ_STACK_SIZE);
-	pr_emerg("Overflow stack: [0x%016lx..0x%016lx]\n",
-		 ovf_stk, ovf_stk + OVERFLOW_STACK_SIZE);
+	pr_emerg("Exception stack: [0x%016lx..0x%016lx]\n",
+		 exc_stk, exc_stk + KERNEL_EXC_STACK_SIZE);
 
 	__show_regs(regs);
 

index 9393fe3ea6a1..b05211a55074 100644
@@ -205,7 +205,7 @@ SYM_FUNC_END(__host_hvc)
 
 .L__hyp_sp_overflow\@:
 	/* Switch to the overflow stack */
-	adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0
+	adr_this_cpu sp, overflow_stack + NVHE_OVERFLOW_STACK_SIZE, x0
 
 	b	hyp_panic_bad_stack
 	ASM_BUG()

index 7c832d60d22b..92fe8b6d4de0 100644
@@ -9,7 +9,7 @@
 #include <asm/memory.h>
 #include <asm/percpu.h>
 
-DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)
+DEFINE_PER_CPU(unsigned long [NVHE_OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)
 	__aligned(16);
 
 DEFINE_PER_CPU(struct kvm_nvhe_stacktrace_info, kvm_stacktrace_info);

@@ -42,7 +42,7 @@ DEFINE_PER_CPU(unsigned long [NVHE_STACKTRACE_SIZE/sizeof(long)], pkvm_stacktrac
 static struct stack_info stackinfo_get_overflow(void)
 {
 	unsigned long low = (unsigned long)this_cpu_ptr(overflow_stack);
-	unsigned long high = low + OVERFLOW_STACK_SIZE;
+	unsigned long high = low + NVHE_OVERFLOW_STACK_SIZE;
 
 	return (struct stack_info) {
 		.low = low,

index 9724c320126b..e5b878f959fc 100644
@@ -27,7 +27,7 @@ static struct stack_info stackinfo_get_overflow(void)
 	struct kvm_nvhe_stacktrace_info *stacktrace_info
 				= this_cpu_ptr_nvhe_sym(kvm_stacktrace_info);
 	unsigned long low = (unsigned long)stacktrace_info->overflow_stack_base;
-	unsigned long high = low + OVERFLOW_STACK_SIZE;
+	unsigned long high = low + NVHE_OVERFLOW_STACK_SIZE;
 
 	return (struct stack_info) {
 		.low = low,

@@ -38,7 +38,7 @@ static struct stack_info stackinfo_get_overflow(void)
 static struct stack_info stackinfo_get_overflow_kern_va(void)
 {
 	unsigned long low = (unsigned long)this_cpu_ptr_nvhe_sym(overflow_stack);
-	unsigned long high = low + OVERFLOW_STACK_SIZE;
+	unsigned long high = low + NVHE_OVERFLOW_STACK_SIZE;
 
 	return (struct stack_info) {
 		.low = low,

2.55.0.1082.g2b9226bbc0-goog

I tested this series with Fastpath on two different server-class machines:
AWS m7g.metal and AmpereOne. Fastpath marks statistically significant improvements as (I) and regressions as (R). Please see the attached file for detailed results.

The statistically significant R and I were counted:

- AWS m7g.metal:
- perf: 5 R, 1 I
- micromm: 0 R, 1 I
- mmtests: 84 R, 0 I

- AmpereOne:
- perf: 3 R, 0 I
- micromm: 0 R, 0 I

For reference, I've also put the detailed stats for the perf at the bottom of the file to prove that the variance during the testing was low.

Thanks,
Usama

Fastpath runs repeatable Linux kernel benchmarks on defined systems under test (SUTs). Read each row as one benchmark metric: the base column is the before mean, the final column is the after result as a percentage change from that baseline; (I) marks a statistically significant improvement, (R) a regression, and an unmarked value no significant change.

Results for SUT Class aws-m7g.metal:

+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| Benchmark                          | Result Class                                             | will-stack-v2-v7.3-rc1-before-m7g-perf (base) | will-stack-v2-v7.3-rc1-after-m7g-perf |
+====================================+==========================================================+===============================================+=======================================+
| perf/futex                         | hash (ops/sec)                                           |                                    2633400.13 |                            (R) -8.21% |
|                                    | lock-pi (ops/sec)                                        |                                        280.40 |                                -0.14% |
|                                    | requeue (ms)                                             |                                         0.028 |                                -3.67% |
|                                    | wake (ms)                                                |                                         0.075 |                                -1.29% |
|                                    | wake-parallel (ms)                                       |                                        0.0041 |                                -6.44% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| perf/mem                           | memset -k 1GB -f default -s 16GB (bytes/sec)             |                                65813285254.01 |                                -0.50% |
|                                    | mmap -p 1GB -f demand -s 32GB -l 5 (bytes/sec)           |                                64096741623.58 |                                -0.84% |
|                                    | mmap -p 4KB -f demand -s 32GB -l 5 (bytes/sec)           |                                49942401198.50 |                                -0.55% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| perf/sched                         | messaging -l 10000 (sec)                                 |                                          1.32 |                            (R) -3.44% |
|                                    | pipe (ops/sec)                                           |                                     588652.50 |                            (I) 18.50% |
|                                    | seccomp-notify (ops/sec)                                 |                                     243222.80 |                            (R) -7.07% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| perf/syscall                       | basic (ops/sec)                                          |                                    5172519.93 |                           (R) -18.35% |
|                                    | execve (ops/sec)                                         |                                       2360.57 |                                -0.59% |
|                                    | fork (ops/sec)                                           |                                       4148.93 |                                -0.34% |
|                                    | getpgid (ops/sec)                                        |                                    5188203.73 |                           (R) -18.99% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| micromm/fork                       | fork: p:1, d:10 (seconds)                                |                                        0.0078 |                                 0.49% |
|                                    | fork: p:512, d:10 (seconds)                              |                                        0.0056 |                                -0.36% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| micromm/munmap                     | munmap: p:1, d:10 (seconds)                              |                                         0.061 |                                -0.93% |
|                                    | munmap: p:512, d:10 (seconds)                            |                                        0.0046 |                                 0.12% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| micromm/vmalloc                    | fix_align_alloc_test: p:1, h:0, l:500000 (usec)          |                                     636383.00 |                                 0.69% |
|                                    | fix_size_alloc_test: p:1, h:0, l:500000 (usec)           |                                     373708.00 |                                 0.65% |
|                                    | fix_size_alloc_test: p:4, h:0, l:500000 (usec)           |                                     468532.83 |                             (I) 2.10% |
|                                    | fix_size_alloc_test: p:16, h:0, l:500000 (usec)          |                                     869582.17 |                                -1.21% |
|                                    | fix_size_alloc_test: p:16, h:1, l:500000 (usec)          |                                     813165.50 |                                -1.12% |
|                                    | fix_size_alloc_test: p:64, h:0, l:100000 (usec)          |                                     499635.33 |                                -0.50% |
|                                    | fix_size_alloc_test: p:64, h:1, l:100000 (usec)          |                                     465521.67 |                                -0.16% |
|                                    | fix_size_alloc_test: p:256, h:0, l:100000 (usec)         |                                    1815456.17 |                                -0.35% |
|                                    | fix_size_alloc_test: p:256, h:1, l:100000 (usec)         |                                    1701639.17 |                                -0.28% |
|                                    | fix_size_alloc_test: p:512, h:0, l:100000 (usec)         |                                    3557058.33 |                                -0.22% |
|                                    | fix_size_alloc_test: p:512, h:1, l:100000 (usec)         |                                    2624327.00 |                                -0.15% |
|                                    | full_fit_alloc_test: p:1, h:0, l:500000 (usec)           |                                     565714.33 |                                 0.27% |
|                                    | kvfree_rcu_1_arg_vmalloc_test: p:1, h:0, l:500000 (usec) |                                     369038.17 |                                 0.27% |
|                                    | kvfree_rcu_2_arg_vmalloc_test: p:1, h:0, l:500000 (usec) |                                     369115.50 |                                 0.73% |
|                                    | long_busy_list_alloc_test: p:1, h:0, l:500000 (usec)     |                                    4383806.67 |                                -0.72% |
|                                    | pcpu_alloc_test: p:1, h:0, l:500000 (usec)               |                                     198536.00 |                                -0.45% |
|                                    | random_size_align_alloc_test: p:1, h:0, l:500000 (usec)  |                                     744627.00 |                                -3.57% |
|                                    | random_size_alloc_test: p:1, h:0, l:500000 (usec)        |                                    2294946.00 |                                -0.83% |
|                                    | vm_map_ram_test: p:1, h:0, l:500000 (usec)               |                                      81581.33 |                                 0.90% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/hackbench                  | hackbench-process-pipes-1 (seconds)                      |                                          0.15 |                            (R) -3.36% |
|                                    | hackbench-process-pipes-4 (seconds)                      |                                          0.38 |                            (R) -6.12% |
|                                    | hackbench-process-pipes-7 (seconds)                      |                                          0.66 |                            (R) -5.12% |
|                                    | hackbench-process-pipes-12 (seconds)                     |                                          1.10 |                            (R) -5.53% |
|                                    | hackbench-process-pipes-21 (seconds)                     |                                          1.48 |                            (R) -6.69% |
|                                    | hackbench-process-pipes-30 (seconds)                     |                                          1.87 |                            (R) -7.30% |
|                                    | hackbench-process-pipes-48 (seconds)                     |                                          2.57 |                            (R) -8.22% |
|                                    | hackbench-process-pipes-79 (seconds)                     |                                          3.61 |                            (R) -9.50% |
|                                    | hackbench-process-pipes-110 (seconds)                    |                                          4.68 |                            (R) -9.85% |
|                                    | hackbench-process-pipes-141 (seconds)                    |                                          5.78 |                           (R) -10.04% |
|                                    | hackbench-process-pipes-172 (seconds)                    |                                          6.95 |                            (R) -9.99% |
|                                    | hackbench-process-pipes-203 (seconds)                    |                                          8.08 |                           (R) -10.16% |
|                                    | hackbench-process-pipes-234 (seconds)                    |                                          9.29 |                           (R) -10.12% |
|                                    | hackbench-process-pipes-256 (seconds)                    |                                         10.08 |                           (R) -10.39% |
|                                    | hackbench-process-sockets-1 (seconds)                    |                                          0.28 |                            (R) -2.13% |
|                                    | hackbench-process-sockets-4 (seconds)                    |                                          0.63 |                            (R) -2.77% |
|                                    | hackbench-process-sockets-7 (seconds)                    |                                          0.99 |                            (R) -3.21% |
|                                    | hackbench-process-sockets-12 (seconds)                   |                                          1.63 |                            (R) -2.56% |
|                                    | hackbench-process-sockets-21 (seconds)                   |                                          2.82 |                                -1.97% |
|                                    | hackbench-process-sockets-30 (seconds)                   |                                          4.03 |                            (R) -2.05% |
|                                    | hackbench-process-sockets-48 (seconds)                   |                                          6.42 |                            (R) -2.46% |
|                                    | hackbench-process-sockets-79 (seconds)                   |                                          9.89 |                            (R) -3.43% |
|                                    | hackbench-process-sockets-110 (seconds)                  |                                         13.76 |                            (R) -3.44% |
|                                    | hackbench-process-sockets-141 (seconds)                  |                                         17.67 |                            (R) -3.96% |
|                                    | hackbench-process-sockets-172 (seconds)                  |                                         21.88 |                            (R) -3.75% |
|                                    | hackbench-process-sockets-203 (seconds)                  |                                         26.17 |                            (R) -3.86% |
|                                    | hackbench-process-sockets-234 (seconds)                  |                                         30.53 |                            (R) -3.90% |
|                                    | hackbench-process-sockets-256 (seconds)                  |                                         33.63 |                            (R) -3.95% |
|                                    | hackbench-thread-pipes-1 (seconds)                       |                                          0.18 |                            (R) -4.43% |
|                                    | hackbench-thread-pipes-4 (seconds)                       |                                          0.40 |                                -3.85% |
|                                    | hackbench-thread-pipes-7 (seconds)                       |                                          0.69 |                            (R) -4.23% |
|                                    | hackbench-thread-pipes-12 (seconds)                      |                                          1.15 |                            (R) -4.93% |
|                                    | hackbench-thread-pipes-21 (seconds)                      |                                          1.53 |                            (R) -5.90% |
|                                    | hackbench-thread-pipes-30 (seconds)                      |                                          1.95 |                            (R) -7.12% |
|                                    | hackbench-thread-pipes-48 (seconds)                      |                                          2.68 |                            (R) -8.46% |
|                                    | hackbench-thread-pipes-79 (seconds)                      |                                          3.66 |                           (R) -10.08% |
|                                    | hackbench-thread-pipes-110 (seconds)                     |                                          4.81 |                           (R) -10.04% |
|                                    | hackbench-thread-pipes-141 (seconds)                     |                                          5.92 |                           (R) -10.60% |
|                                    | hackbench-thread-pipes-172 (seconds)                     |                                          7.16 |                           (R) -10.37% |
|                                    | hackbench-thread-pipes-203 (seconds)                     |                                          8.38 |                           (R) -10.33% |
|                                    | hackbench-thread-pipes-234 (seconds)                     |                                          9.62 |                           (R) -10.56% |
|                                    | hackbench-thread-pipes-256 (seconds)                     |                                         10.49 |                           (R) -10.44% |
|                                    | hackbench-thread-sockets-1 (seconds)                     |                                          0.30 |                                -1.74% |
|                                    | hackbench-thread-sockets-4 (seconds)                     |                                          0.64 |                            (R) -3.70% |
|                                    | hackbench-thread-sockets-7 (seconds)                     |                                          0.99 |                            (R) -3.81% |
|                                    | hackbench-thread-sockets-12 (seconds)                    |                                          1.59 |                            (R) -3.91% |
|                                    | hackbench-thread-sockets-21 (seconds)                    |                                          2.71 |                            (R) -4.14% |
|                                    | hackbench-thread-sockets-30 (seconds)                    |                                          3.85 |                            (R) -3.93% |
|                                    | hackbench-thread-sockets-48 (seconds)                    |                                          6.12 |                            (R) -3.65% |
|                                    | hackbench-thread-sockets-79 (seconds)                    |                                          9.50 |                            (R) -3.97% |
|                                    | hackbench-thread-sockets-110 (seconds)                   |                                         13.19 |                            (R) -3.83% |
|                                    | hackbench-thread-sockets-141 (seconds)                   |                                         16.99 |                            (R) -3.56% |
|                                    | hackbench-thread-sockets-172 (seconds)                   |                                         21.04 |                            (R) -3.70% |
|                                    | hackbench-thread-sockets-203 (seconds)                   |                                         25.24 |                            (R) -3.55% |
|                                    | hackbench-thread-sockets-234 (seconds)                   |                                         29.48 |                            (R) -3.35% |
|                                    | hackbench-thread-sockets-256 (seconds)                   |                                         32.52 |                            (R) -3.25% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/kernbench                  | kernbench-elsp-64 (seconds)                              |                                        360.03 |                                -0.25% |
|                                    | kernbench-syst-64 (seconds)                              |                                       1129.20 |                                -1.07% |
|                                    | kernbench-user-64 (seconds)                              |                                      19755.83 |                                -0.05% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/stressng-bad-altstack      | stressng-bad-altstack-1 (ops/second)                     |                                          9.68 |                                 0.07% |
|                                    | stressng-bad-altstack-2 (ops/second)                     |                                         19.02 |                                 0.21% |
|                                    | stressng-bad-altstack-4 (ops/second)                     |                                         37.40 |                                -0.01% |
|                                    | stressng-bad-altstack-8 (ops/second)                     |                                         55.42 |                                -0.50% |
|                                    | stressng-bad-altstack-16 (ops/second)                    |                                      17466.33 |                                 1.13% |
|                                    | stressng-bad-altstack-32 (ops/second)                    |                                      54185.30 |                                -0.48% |
|                                    | stressng-bad-altstack-64 (ops/second)                    |                                      98314.32 |                                -0.00% |
|                                    | stressng-bad-altstack-128 (ops/second)                   |                                     104605.90 |                                -1.52% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/stressng-class-io-parallel | stressng-aio-64 (ops/second)                             |                                       3463.65 |                                -2.28% |
|                                    | stressng-hdd-64 (ops/second)                             |                                       4212.36 |                                 0.14% |
|                                    | stressng-io-uring-64 (ops/second)                        |                                        776.51 |                                -6.25% |
|                                    | stressng-readahead-64 (ops/second)                       |                                   18119014.75 |                                -3.88% |
|                                    | stressng-revio-64 (ops/second)                           |                                    2789119.01 |                                -4.00% |
|                                    | stressng-seek-64 (ops/second)                            |                                    2753470.04 |                               -12.39% |
|                                    | stressng-sync-file-64 (ops/second)                       |                                         27.63 |                                -7.59% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/stressng-context           | stressng-context-1 (ops/second)                          |                                       2514.25 |                           (R) -15.70% |
|                                    | stressng-context-2 (ops/second)                          |                                       5025.68 |                           (R) -15.80% |
|                                    | stressng-context-4 (ops/second)                          |                                      10055.60 |                           (R) -15.92% |
|                                    | stressng-context-8 (ops/second)                          |                                      20093.56 |                           (R) -15.86% |
|                                    | stressng-context-16 (ops/second)                         |                                      40100.87 |                           (R) -15.77% |
|                                    | stressng-context-32 (ops/second)                         |                                      80155.74 |                           (R) -15.68% |
|                                    | stressng-context-64 (ops/second)                         |                                     160064.84 |                           (R) -15.61% |
|                                    | stressng-context-128 (ops/second)                        |                                     159299.81 |                           (R) -15.34% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/stressng-fork              | stressng-fork-1 (ops/second)                             |                                       3781.34 |                                -0.34% |
|                                    | stressng-fork-2 (ops/second)                             |                                       7268.31 |                                 0.24% |
|                                    | stressng-fork-4 (ops/second)                             |                                      14267.94 |                                -1.75% |
|                                    | stressng-fork-8 (ops/second)                             |                                      27227.12 |                                -1.69% |
|                                    | stressng-fork-16 (ops/second)                            |                                      49947.85 |                                -0.92% |
|                                    | stressng-fork-32 (ops/second)                            |                                      81903.86 |                                -2.31% |
|                                    | stressng-fork-64 (ops/second)                            |                                      94812.06 |                                 0.03% |
|                                    | stressng-fork-128 (ops/second)                           |                                      93409.14 |                                -1.74% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/stressng-get               | stressng-get-1 (ops/second)                              |                                       7771.42 |                            (R) -3.77% |
|                                    | stressng-get-2 (ops/second)                              |                                      14923.01 |                            (R) -4.15% |
|                                    | stressng-get-4 (ops/second)                              |                                      28916.93 |                            (R) -3.08% |
|                                    | stressng-get-8 (ops/second)                              |                                      29471.61 |                                -0.42% |
|                                    | stressng-get-16 (ops/second)                             |                                      29430.88 |                                -0.32% |
|                                    | stressng-get-32 (ops/second)                             |                                      29390.56 |                                -0.36% |
|                                    | stressng-get-64 (ops/second)                             |                                      29403.05 |                                -0.28% |
|                                    | stressng-get-128 (ops/second)                            |                                      29397.55 |                                -0.33% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/stressng-getdent           | stressng-getdent-1 (ops/second)                          |                                     143257.70 |                            (R) -3.15% |
|                                    | stressng-getdent-2 (ops/second)                          |                                     256672.76 |                            (R) -3.08% |
|                                    | stressng-getdent-4 (ops/second)                          |                                     323023.87 |                                -2.79% |
|                                    | stressng-getdent-8 (ops/second)                          |                                     336126.91 |                                 1.09% |
|                                    | stressng-getdent-16 (ops/second)                         |                                     344968.94 |                                -1.74% |
|                                    | stressng-getdent-32 (ops/second)                         |                                     388775.02 |                                -0.64% |
|                                    | stressng-getdent-64 (ops/second)                         |                                     374204.16 |                                 0.04% |
|                                    | stressng-getdent-128 (ops/second)                        |                                     196093.77 |                                 2.77% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/stressng-madvise           | stressng-madvise-1 (ops/second)                          |                                         37.05 |                                       |
|                                    | stressng-madvise-2 (ops/second)                          |                                         74.97 |                           (R) -12.91% |
|                                    | stressng-madvise-4 (ops/second)                          |                                        143.90 |                           (R) -11.65% |
|                                    | stressng-madvise-8 (ops/second)                          |                                        265.46 |                            (R) -8.69% |
|                                    | stressng-madvise-16 (ops/second)                         |                                        319.68 |                                -1.94% |
|                                    | stressng-madvise-32 (ops/second)                         |                                        431.97 |                                -0.09% |
|                                    | stressng-madvise-64 (ops/second)                         |                                        457.43 |                                 1.49% |
|                                    | stressng-madvise-128 (ops/second)                        |                                        519.15 |                                 3.63% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/stressng-mmap              | stressng-mmap-1 (ops/second)                             |                                          0.73 |                                -0.91% |
|                                    | stressng-mmap-2 (ops/second)                             |                                          2.83 |                            (R) -2.00% |
|                                    | stressng-mmap-4 (ops/second)                             |                                         10.80 |                                -1.68% |
|                                    | stressng-mmap-8 (ops/second)                             |                                         39.26 |                                -0.90% |
|                                    | stressng-mmap-16 (ops/second)                            |                                        115.64 |                                -0.79% |
|                                    | stressng-mmap-32 (ops/second)                            |                                        224.88 |                                -0.31% |
|                                    | stressng-mmap-64 (ops/second)                            |                                        458.60 |                                -0.81% |
|                                    | stressng-mmap-128 (ops/second)                           |                                        928.63 |                                -0.90% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/stressng-vm-splice         | stressng-vm-splice-1 (ops/second)                        |                                     391522.81 |                            (R) -6.46% |
|                                    | stressng-vm-splice-2 (ops/second)                        |                                    1006787.55 |                            (R) -7.53% |
|                                    | stressng-vm-splice-4 (ops/second)                        |                                    2395582.21 |                            (R) -9.11% |
|                                    | stressng-vm-splice-8 (ops/second)                        |                                    5260118.81 |                            (R) -9.96% |
|                                    | stressng-vm-splice-16 (ops/second)                       |                                   11130004.33 |                           (R) -10.78% |
|                                    | stressng-vm-splice-32 (ops/second)                       |                                   22230580.82 |                           (R) -10.54% |
|                                    | stressng-vm-splice-64 (ops/second)                       |                                   44458735.50 |                           (R) -10.69% |
|                                    | stressng-vm-splice-128 (ops/second)                      |                                   43738226.27 |                           (R) -10.98% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/stressng-wait              | stressng-wait-1 (ops/second)                             |                                       2998.90 |                                -2.73% |
|                                    | stressng-wait-2 (ops/second)                             |                                       5994.53 |                            (R) -8.36% |
|                                    | stressng-wait-4 (ops/second)                             |                                      11834.43 |                            (R) -8.48% |
|                                    | stressng-wait-8 (ops/second)                             |                                      23711.92 |                            (R) -7.53% |
|                                    | stressng-wait-16 (ops/second)                            |                                      49787.93 |                                 0.21% |
|                                    | stressng-wait-32 (ops/second)                            |                                      92168.93 |                            (R) -3.67% |
|                                    | stressng-wait-64 (ops/second)                            |                                     140977.84 |                                -0.52% |
|                                    | stressng-wait-128 (ops/second)                           |                                     221533.93 |                                 0.49% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/stressng-zombie            | stressng-zombie-1 (ops/second)                           |                                      11283.86 |                                -0.78% |
|                                    | stressng-zombie-2 (ops/second)                           |                                      21403.19 |                                -0.74% |
|                                    | stressng-zombie-4 (ops/second)                           |                                      40659.77 |                                -1.27% |
|                                    | stressng-zombie-8 (ops/second)                           |                                      72426.55 |                                -0.56% |
|                                    | stressng-zombie-16 (ops/second)                          |                                      98712.29 |                                 0.38% |
|                                    | stressng-zombie-32 (ops/second)                          |                                      96167.89 |                                -2.76% |
|                                    | stressng-zombie-64 (ops/second)                          |                                      77641.77 |                                -0.31% |
|                                    | stressng-zombie-128 (ops/second)                         |                                      66592.01 |                                -1.73% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/sysbench-cpu               | sysbenchcpu-1 (ops/second)                               |                                        835.75 |                                -0.03% |
|                                    | sysbenchcpu-4 (ops/second)                               |                                       3342.56 |                                -0.02% |
|                                    | sysbenchcpu-7 (ops/second)                               |                                       5849.17 |                                -0.02% |
|                                    | sysbenchcpu-12 (ops/second)                              |                                      10026.47 |                                -0.02% |
|                                    | sysbenchcpu-21 (ops/second)                              |                                      17544.40 |                                -0.02% |
|                                    | sysbenchcpu-30 (ops/second)                              |                                      25061.15 |                                -0.01% |
|                                    | sysbenchcpu-48 (ops/second)                              |                                      40092.05 |                                -0.01% |
|                                    | sysbenchcpu-79 (ops/second)                              |                                      53442.39 |                                 0.00% |
|                                    | sysbenchcpu-110 (ops/second)                             |                                      53441.13 |                                 0.00% |
|                                    | sysbenchcpu-128 (ops/second)                             |                                      53438.59 |                                 0.01% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/sysbench-mutex             | sysbenchmutex-1 (usec)                                   |                                        194.31 |                                -0.02% |
|                                    | sysbenchmutex-4 (usec)                                   |                                        200.57 |                                 0.06% |
|                                    | sysbenchmutex-7 (usec)                                   |                                        201.56 |                                 0.23% |
|                                    | sysbenchmutex-12 (usec)                                  |                                        202.18 |                                 0.20% |
|                                    | sysbenchmutex-21 (usec)                                  |                                        202.18 |                                -0.22% |
|                                    | sysbenchmutex-30 (usec)                                  |                                        203.09 |                                -0.31% |
|                                    | sysbenchmutex-48 (usec)                                  |                                        234.78 |                                -2.87% |
|                                    | sysbenchmutex-79 (usec)                                  |                                        359.85 |                                 0.57% |
|                                    | sysbenchmutex-110 (usec)                                 |                                        512.77 |                                 0.06% |
|                                    | sysbenchmutex-128 (usec)                                 |                                        593.54 |                                 3.77% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+
| mmtests/sysbench-thread            | sysbenchthread-1 (usec)                                  |                                          0.35 |                           (R) -16.67% |
|                                    | sysbenchthread-4 (usec)                                  |                                          0.52 |                           (R) -10.92% |
|                                    | sysbenchthread-7 (usec)                                  |                                          2.61 |                               -10.42% |
|                                    | sysbenchthread-12 (usec)                                 |                                          5.92 |                                13.47% |
|                                    | sysbenchthread-21 (usec)                                 |                                         14.75 |                                16.49% |
|                                    | sysbenchthread-30 (usec)                                 |                                         18.19 |                                23.85% |
|                                    | sysbenchthread-48 (usec)                                 |                                         33.75 |                                 6.74% |
|                                    | sysbenchthread-79 (usec)                                 |                                         60.46 |                               -11.26% |
|                                    | sysbenchthread-110 (usec)                                |                                         88.05 |                               -11.91% |
|                                    | sysbenchthread-128 (usec)                                |                                        109.27 |                                -2.42% |
+------------------------------------+----------------------------------------------------------+-----------------------------------------------+---------------------------------------+

Results for SUT Class cesw-aarch64-ampereone-1s-a192-32x:

+-----------------+----------------------------------------------------------+-----------------------------------------------------+---------------------------------------------+
| Benchmark       | Result Class                                             | will-stack-v2-v7.3-rc1-before-ampereone-perf (base) | will-stack-v2-v7.3-rc1-after-ampereone-perf |
+=================+==========================================================+=====================================================+=============================================+
| perf/futex      | hash (ops/sec)                                           |                                          2631145.87 |                                      -8.65% |
|                 | lock-pi (ops/sec)                                        |                                               78.97 |                                      -0.46% |
|                 | requeue (ms)                                             |                                                0.12 |                                  (R) -2.31% |
|                 | wake (ms)                                                |                                                0.29 |                                      -1.93% |
|                 | wake-parallel (ms)                                       |                                              0.0026 |                                       0.64% |
+-----------------+----------------------------------------------------------+-----------------------------------------------------+---------------------------------------------+
| perf/mem        | memset -k 1GB -f default -s 16GB (bytes/sec)             |                                      34089991443.74 |                                      -0.26% |
|                 | mmap -p 1GB -f demand -s 32GB -l 5 (bytes/sec)           |                                      33956264487.33 |                                      -0.65% |
|                 | mmap -p 4KB -f demand -s 32GB -l 5 (bytes/sec)           |                                      25932063345.48 |                                      -0.18% |
+-----------------+----------------------------------------------------------+-----------------------------------------------------+---------------------------------------------+
| perf/sched      | messaging -l 10000 (sec)                                 |                                                0.98 |                                      -1.76% |
|                 | pipe (ops/sec)                                           |                                           314096.00 |                                       1.49% |
|                 | seccomp-notify (ops/sec)                                 |                                           130701.47 |                                      -0.34% |
+-----------------+----------------------------------------------------------+-----------------------------------------------------+---------------------------------------------+
| perf/syscall    | basic (ops/sec)                                          |                                          7365617.00 |                                  (R) -9.34% |
|                 | execve (ops/sec)                                         |                                             1580.90 |                                      -0.60% |
|                 | fork (ops/sec)                                           |                                             2456.27 |                                      -0.25% |
|                 | getpgid (ops/sec)                                        |                                          7676908.73 |                                 (R) -11.71% |
+-----------------+----------------------------------------------------------+-----------------------------------------------------+---------------------------------------------+
| micromm/fork    | fork: p:1, d:10 (seconds)                                |                                              0.0084 |                                      -0.57% |
|                 | fork: p:512, d:10 (seconds)                              |                                              0.0048 |                                      -0.39% |
+-----------------+----------------------------------------------------------+-----------------------------------------------------+---------------------------------------------+
| micromm/munmap  | munmap: p:1, d:10 (seconds)                              |                                               0.078 |                                      -2.66% |
|                 | munmap: p:512, d:10 (seconds)                            |                                               0.022 |                                       0.94% |
+-----------------+----------------------------------------------------------+-----------------------------------------------------+---------------------------------------------+
| micromm/vmalloc | fix_align_alloc_test: p:1, h:0, l:500000 (usec)          |                                           737247.67 |                                       5.13% |
|                 | fix_size_alloc_test: p:1, h:0, l:500000 (usec)           |                                           391932.17 |                                       1.25% |
|                 | fix_size_alloc_test: p:4, h:0, l:500000 (usec)           |                                           512435.17 |                                      -0.01% |
|                 | fix_size_alloc_test: p:16, h:0, l:500000 (usec)          |                                           970130.83 |                                      -1.02% |
|                 | fix_size_alloc_test: p:16, h:1, l:500000 (usec)          |                                           915661.00 |                                      -0.61% |
|                 | fix_size_alloc_test: p:64, h:0, l:100000 (usec)          |                                           590007.50 |                                      -0.94% |
|                 | fix_size_alloc_test: p:64, h:1, l:100000 (usec)          |                                           541856.50 |                                      -0.31% |
|                 | fix_size_alloc_test: p:256, h:0, l:100000 (usec)         |                                          2161700.17 |                                      -0.92% |
|                 | fix_size_alloc_test: p:256, h:1, l:100000 (usec)         |                                          1980042.17 |                                      -0.52% |
|                 | fix_size_alloc_test: p:512, h:0, l:100000 (usec)         |                                          4216675.67 |                                      -0.94% |
|                 | fix_size_alloc_test: p:512, h:1, l:100000 (usec)         |                                          2983924.67 |                                      -1.93% |
|                 | full_fit_alloc_test: p:1, h:0, l:500000 (usec)           |                                           613119.67 |                                      -0.41% |
|                 | kvfree_rcu_1_arg_vmalloc_test: p:1, h:0, l:500000 (usec) |                                           534642.83 |                                       2.30% |
|                 | kvfree_rcu_2_arg_vmalloc_test: p:1, h:0, l:500000 (usec) |                                           516561.67 |                                      -0.69% |
|                 | long_busy_list_alloc_test: p:1, h:0, l:500000 (usec)     |                                          4973543.83 |                                      -0.79% |
|                 | pcpu_alloc_test: p:1, h:0, l:500000 (usec)               |                                           484844.83 |                                       0.75% |
|                 | random_size_align_alloc_test: p:1, h:0, l:500000 (usec)  |                                           890200.83 |                                      -0.56% |
|                 | random_size_alloc_test: p:1, h:0, l:500000 (usec)        |                                          2625180.83 |                                      -0.75% |
|                 | vm_map_ram_test: p:1, h:0, l:500000 (usec)               |                                           107025.00 |                                       1.39% |
+-----------------+----------------------------------------------------------+-----------------------------------------------------+---------------------------------------------+

Perf variability (AWS m7g.metal)

For interested readers, these tables show the spread, 95% confidence interval, coefficient of variation (cv), and number of measurements for each perf result.

Before (baseline):

+---------------+------------------------------------------------+---------+-----------+----------------+-----------+--------+--------+---------+
| Benchmark     | Result Class                                   |     min |   ci95min |           mean |   ci95max |    max |     cv |   count |
+===============+================================================+=========+===========+================+===========+========+========+=========+
| perf/futex    | hash (ops/sec)                                 |  -0.96% |    -0.22% |     2633400.13 |     0.22% |  1.07% |  0.58% |      30 |
|               | lock-pi (ops/sec)                              |  -5.49% |    -0.39% |         280.40 |     0.39% |  0.21% |  1.05% |      30 |
|               | requeue (ms)                                   |  -6.82% |    -4.12% |          0.028 |     4.12% | 41.02% | 11.03% |      30 |
|               | wake (ms)                                      | -13.60% |    -2.96% |          0.075 |     2.96% | 11.90% |  7.92% |      30 |
|               | wake-parallel (ms)                             | -19.77% |    -5.83% |         0.0041 |     5.83% | 45.87% | 15.62% |      30 |
+---------------+------------------------------------------------+---------+-----------+----------------+-----------+--------+--------+---------+
| perf/mem      | memset -k 1GB -f default -s 16GB (bytes/sec)   |  -4.20% |    -0.46% | 65813285254.01 |     0.46% |  1.22% |  1.24% |      30 |
|               | mmap -p 1GB -f demand -s 32GB -l 5 (bytes/sec) | -13.28% |    -1.36% | 64096741623.58 |     1.36% |  3.54% |  3.64% |      30 |
|               | mmap -p 4KB -f demand -s 32GB -l 5 (bytes/sec) | -11.70% |    -1.31% | 49942401198.50 |     1.31% |  2.59% |  3.51% |      30 |
+---------------+------------------------------------------------+---------+-----------+----------------+-----------+--------+--------+---------+
| perf/sched:76 | messaging -l 10000 (sec)                       |  -0.87% |    -0.15% |           1.32 |     0.15% |  0.87% |  0.41% |      30 |
|               | pipe (ops/sec)                                 | -22.16% |    -8.94% |      588652.50 |     8.94% | 71.10% | 23.95% |      30 |
|               | seccomp-notify (ops/sec)                       |  -5.45% |    -1.08% |      243222.80 |     1.08% |  4.85% |  2.90% |      30 |
+---------------+------------------------------------------------+---------+-----------+----------------+-----------+--------+--------+---------+
| perf/syscall  | basic (ops/sec)                                |  -0.49% |    -0.19% |     5172519.93 |     0.19% |  2.60% |  0.52% |      30 |
|               | execve (ops/sec)                               |  -1.42% |    -0.28% |        2360.57 |     0.28% |  1.20% |  0.74% |      30 |
|               | fork (ops/sec)                                 |  -1.85% |    -0.37% |        4148.93 |     0.37% |  1.79% |  0.99% |      30 |
|               | getpgid (ops/sec)                              |  -0.40% |    -0.16% |     5188203.73 |     0.16% |  1.54% |  0.44% |      30 |
+---------------+------------------------------------------------+---------+-----------+----------------+-----------+--------+--------+---------+

After (patched):

+---------------+------------------------------------------------+---------+-----------+----------------+-----------+--------+--------+---------+
| Benchmark     | Result Class                                   |     min |   ci95min |           mean |   ci95max |    max |     cv |   count |
+===============+================================================+=========+===========+================+===========+========+========+=========+
| perf/futex    | hash (ops/sec)                                 |  -0.86% |    -0.17% |     2417106.13 |     0.17% |  0.86% |  0.46% |      30 |
|               | lock-pi (ops/sec)                              |   0.00% |     0.00% |         280.00 |     0.00% |  0.00% |  0.00% |      30 |
|               | requeue (ms)                                   |  -4.39% |    -1.85% |          0.029 |     1.85% | 23.81% |  4.95% |      30 |
|               | wake (ms)                                      | -21.73% |    -3.39% |          0.076 |     3.39% | 10.19% |  9.07% |      30 |
|               | wake-parallel (ms)                             | -24.94% |    -5.29% |         0.0044 |     5.29% | 22.82% | 14.18% |      30 |
+---------------+------------------------------------------------+---------+-----------+----------------+-----------+--------+--------+---------+
| perf/mem      | memset -k 1GB -f default -s 16GB (bytes/sec)   |  -3.31% |    -0.51% | 65483112434.86 |     0.51% |  1.80% |  1.38% |      30 |
|               | mmap -p 1GB -f demand -s 32GB -l 5 (bytes/sec) |  -4.05% |    -1.07% | 63556894478.20 |     1.07% |  4.41% |  2.87% |      30 |
|               | mmap -p 4KB -f demand -s 32GB -l 5 (bytes/sec) | -10.72% |    -1.06% | 49666606495.00 |     1.06% |  3.00% |  2.85% |      30 |
+---------------+------------------------------------------------+---------+-----------+----------------+-----------+--------+--------+---------+
| perf/sched:76 | messaging -l 10000 (sec)                       |  -0.71% |    -0.14% |           1.37 |     0.14% |  0.82% |  0.38% |      30 |
|               | pipe (ops/sec)                                 | -33.91% |    -6.59% |      697562.97 |     6.59% | 27.52% | 17.65% |      30 |
|               | seccomp-notify (ops/sec)                       |  -5.05% |    -0.84% |      226021.30 |     0.84% |  4.15% |  2.24% |      30 |
+---------------+------------------------------------------------+---------+-----------+----------------+-----------+--------+--------+---------+
| perf/syscall  | basic (ops/sec)                                |  -0.48% |    -0.15% |     4223503.80 |     0.15% |  1.11% |  0.39% |      30 |
|               | execve (ops/sec)                               |  -1.60% |    -0.23% |        2346.60 |     0.23% |  1.42% |  0.61% |      30 |
|               | fork (ops/sec)                                 |  -2.01% |    -0.43% |        4134.93 |     0.43% |  2.27% |  1.14% |      30 |
|               | getpgid (ops/sec)                              |  -0.92% |    -0.10% |     4203192.50 |     0.10% |  0.25% |  0.26% |      30 |
+---------------+------------------------------------------------+---------+-----------+----------------+-----------+--------+--------+---------+

Overall I like what I see, but can we split it in two... like first the patches up to fixing the overflow handling using SP_EL0 and finalize that?

Hopefully Usama can test just that part and see if it has a lesser performance impact as well.

Then we can think about the rest once that is merged and works.

Yours,
Linus Walleij

Yes, that makes sense (and thanks for going through it). The whole thing is bisectable, so lemme go through the feedback I've got so far and send out a prefix that we could consider merging as preparatory work.

I'd like to see those numbers at various points during the series, tbh, so we can rule out any stack-protector changes etc.

Will

I'll trigger run on this prefix.

Sure, I can trigger runs on different points if you can mention which would be more valuable.

Thanks,
Usama

I would say comparison after applying 1 thru
- patch 12
- patch 14
- patch 18 and
- all patches

would be interesting to see. Will do you agree?

Yours,
Linus Walleij