arm64: Move overflow sp into SP_EL1 and kernel sp into SP_EL0
Subsystems (11)
Will Deacon
maintainercover arm64: Move overflow sp into SP_EL1 and kernel sp into SP_EL0
Hi everyone,
This series is a bit of a complicated juggling act that, on its own, doesn't achieve an awful lot. However, it lays the ground work for sizing the kernel stack at runtime, e.g. via a cmdline option or even potentially on a per-task basis and so I would like to work towards getting it merged independently.
The series is based on v7.3-rc1 and structured as follows:
* The first 9 patches move the 'current' task pointer from SP_EL0 to TPIDRRO_EL0.
* The following 8 patches point the newly-freed SP_EL0 at the overflow stack and switch to it explicitly when we detect a kernel stack overflow.
* The final 4 patches turn everything on its head, so that the overflow stack and kernel stack are swapped, with the former now residing in SP_EL1 and the latter in SP_EL0.
At the end of all that, when we take an exception from EL1, we are immediately transitioned to the overflow stack (now renamed "exception stack") and can push registers right away. This also means that using SPINTMASK to control NMI masking becomes a possibility, although speaking to Mark, Vladimir and Ada, they all seem to prefer ALLINT.
Mostafa will soon post a follow-up series that allows the kernel stack size to be specified on the kernel cmdline, which we are hoping to use to configure an 8k stack size in Android. I will be talking more about all of this at LPC in the Memory Management MC:
https://lpc.events/event/20/contributions/2419/
Cheers,
Will
--->8
Mostafa Saleh (1):
KVM: arm64: Protect TPIDRRO_EL0 across guest entry/exit
Will Deacon (20):
arm64: entry: Defer setting of TPIDRRO_EL0 until exit to userspace
arm64: entry: Only check for stack overflow on exceptions from EL1
arm64: stackprotector: Temporarily disable per-task stackprotector
arm64: bpf: Add support for generating reads of TPIDRRO_EL0
arm64: Store 'current' in TPIDRRO_EL0 instead of SP_EL0 selftests/bpf: arm64: Use TPIDRRO_EL0 instead of SP_EL0 for 'current' scripts/gdb: arm64: Use TPIDRRO_EL0 instead of SP_EL0 for 'current'
arm64: stackprotector: Re-enable per-task stackprotector
arm64: percpu: Specialise set_my_cpu_offset() for the primary CPU
arm64: percpu: Annotate __kern_my_cpu_offset() as '__always_inline'
KVM: arm64: Preserve handler/thread bit of EL1 mode in
__finalise_el2()
arm64: sdei: Guard most of asm/sdei.h with CONFIG_ARM_SDE_INTERFACE
arm64: sdei: Support SDEI events from kernel handler and thread modes
arm64: entry: Point SP_EL0 at the overflow stack
arm64: entry: Implement EL1t exception handlers for overflow stack
arm64: entry: Use SPSel to switch to overflow stack
arm64: entry: Split up kernel_ventry macro into separate helper macros
arm64: entry: The great stack switcheroo
arm64: tracing: Advertise a mode of EL1t in synthetic kernel regs
arm64: Rename 'overflow_stack' and OVERFLOW_STACK_SIZE
arch/arm64/Kconfig | 2 +- arch/arm64/Makefile | 2 +- arch/arm64/include/asm/assembler.h | 2 +- arch/arm64/include/asm/current.h | 6 +- arch/arm64/include/asm/ftrace.h | 4 +- arch/arm64/include/asm/insn.h | 1 + arch/arm64/include/asm/kvm_asm.h | 26 +++++ arch/arm64/include/asm/memory.h | 10 +- arch/arm64/include/asm/percpu.h | 10 +- arch/arm64/include/asm/perf_event.h | 2 +- arch/arm64/include/asm/sdei.h | 2 + arch/arm64/include/asm/stacktrace.h | 9 +- arch/arm64/include/asm/stacktrace/nvhe.h | 3 +- arch/arm64/include/uapi/asm/ptrace.h | 1 + arch/arm64/kernel/asm-offsets.c | 2 + arch/arm64/kernel/entry-common.c | 51 +++++++-- arch/arm64/kernel/entry.S | 108 +++++++++--------- arch/arm64/kernel/head.S | 8 +- arch/arm64/kernel/hyp-stub.S | 7 +- arch/arm64/kernel/process.c | 21 +--- arch/arm64/kernel/sdei.c | 6 +- arch/arm64/kernel/smp.c | 19 ++- arch/arm64/kernel/stacktrace.c | 2 +- arch/arm64/kernel/sys_compat.c | 7 -- arch/arm64/kernel/traps.c | 14 +-- arch/arm64/kvm/hyp/entry.S | 15 ++- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 2 - arch/arm64/kvm/hyp/nvhe/host.S | 2 +- arch/arm64/kvm/hyp/nvhe/stacktrace.c | 4 +- arch/arm64/kvm/stacktrace.c | 4 +- arch/arm64/mm/proc.S | 3 + arch/arm64/net/bpf_jit.h | 2 + arch/arm64/net/bpf_jit_comp.c | 4 +- scripts/gdb/linux/cpus.py | 2 +- .../selftests/bpf/progs/verifier_jit_inline.c | 2 +- 35 files changed, 218 insertions(+), 147 deletions(-)
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 1/21 arm64: entry: Defer setting of TPIDRRO_EL0 until exit to userspace
In preparation for using TPIDRRO_EL0 to point at 'current' while running inside the kernel, defer its userspace initialisation from the context-switch patch to exception return. This has the added benefit of not having to worry about keeping the in-memory value and the register value in-sync during preemptible sections.
--- arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/entry.S | 4 +++- arch/arm64/kernel/process.c | 16 +--------------- arch/arm64/kernel/sys_compat.c | 7 ------- 4 files changed, 5 insertions(+), 23 deletions(-)
arch/arm64/kernel/asm-offsets.c +1 -0
@@ -42,6 +42,7 @@ int main(void)#endif BLANK (); DEFINE(THREAD_CPU_CONTEXT, offsetof(struct task_struct, thread.cpu_context)); + DEFINE(THREAD_TP_VALUE, offsetof(struct task_struct, thread.uw.tp_value)); DEFINE(THREAD_SCTLR_USER, offsetof(struct task_struct, thread.sctlr_user));#ifdef CONFIG_ARM64_PTR_AUTH DEFINE(THREAD_KEYS_USER, offsetof(struct task_struct, thread.keys_user));
arch/arm64/kernel/entry.S +3 -1
@@ -47,7 +47,6 @@ b .Lskip_tramp_vectors_cleanup\@ .if \regsize == 64 mrs x30, tpidrro_el0 - msr tpidrro_el0, xzr .else mov x30, xzr .endif
@@ -360,9 +359,12 @@ alternative_else_nop_endif .if \el == 0 ldr x23, [sp, #S_SP] // load return stack pointer msr sp_el0, x23 + msr tpidrro_el0, xzr tst x22, #PSR_MODE32_BIT // native task? b.eq 3f + ldr x0, [tsk, #THREAD_TP_VALUE] + msr tpidrro_el0, x0 #ifdef CONFIG_ARM64_ERRATUM_845719 alternative_if ARM64_WORKAROUND_845719 #ifdef CONFIG_PID_IN_CONTEXTIDR
arch/arm64/kernel/process.c +1 -15
@@ -256,17 +256,8 @@ static void tls_thread_flush(void)if (system_supports_tpidr2 ())write_sysreg_s (0 ,SYS_TPIDR2_EL0 ); -if (is_compat_task ()) { +if (is_compat_task ()) current->thread.uw.tp_value =0 ; - -/* -* We need to ensure ordering between the shadow state and the -* hardware state, so that we don't corrupt the hardware state -* with a stale shadow state during context switch. -*/ -barrier (); -write_sysreg (0 , tpidrro_el0); - } }static void flush_tagged_addr_state (void )
@@ -531,11 +522,6 @@ static void tls_thread_switch(struct task_struct *next) {tls_preserve_current_state (); -if (is_compat_thread (task_thread_info (next))) -write_sysreg (next->thread.uw.tp_value, tpidrro_el0); -else -write_sysreg (0 , tpidrro_el0); -write_sysreg (*task_user_tls (next), tpidr_el0);if (system_supports_tpidr2 ())write_sysreg_s (next->thread.tpidr2_el0,SYS_TPIDR2_EL0 );
arch/arm64/kernel/sys_compat.c +0 -7
@@ -89,13 +89,6 @@ long compat_arm_syscall(struct pt_regs *regs, int scno)case __ARM_NR_compat_set_tls: current->thread.uw.tp_value = regs->regs[0 ]; - -/* -* Protect against register corruption from context switch. -* See comment in tls_thread_flush. -*/ -barrier (); -write_sysreg (regs->regs[0 ], tpidrro_el0);return 0 ;default :
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Re: [PATCH 01/21] arm64: entry: Defer setting of TPIDRRO_EL0 until exit to userspace
Quoted 40 lines · Will Deacon
In preparation for using
TPIDRRO_EL0to point at 'current' while running inside the kernel, defer its userspace initialisation from the context-switch patch to exception return. This has the added benefit of not having to worry about keeping the in-memory value and the register value in-sync during preemptible sections.Cc Mark Rutland <[email protected]>Signed-off-by Will Deacon <[email protected]>--- arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/entry.S | 4 +++- arch/arm64/kernel/process.c | 16 +--------------- arch/arm64/kernel/sys_compat.c | 7 ------- 4 files changed, 5 insertions(+), 23 deletions(-)arch/arm64/kernel/asm-offsets.c +1
-0@@ -42,6 +42,7 @@ int main(void)#endif BLANK (); DEFINE(THREAD_CPU_CONTEXT, offsetof(struct task_struct, thread.cpu_context)); + DEFINE(THREAD_TP_VALUE, offsetof(struct task_struct, thread.uw.tp_value)); DEFINE(THREAD_SCTLR_USER, offsetof(struct task_struct, thread.sctlr_user));#ifdef CONFIG_ARM64_PTR_AUTH DEFINE(THREAD_KEYS_USER, offsetof(struct task_struct, thread.keys_user));arch/arm64/kernel/entry.S +0
-1@@ -47,7 +47,6 @@ b .Lskip_tramp_vectors_cleanup\@ .if \regsize == 64 mrs x30, tpidrro_el0 - msr tpidrro_el0, xzr .else mov x30, xzr .endif
arch/arm64/kernel/entry.S alternative_else_nop_endif · line 367
Quoted 9 lines
arch/arm64/kernel/entry.S +3
-0@@ -360,9 +359,12 @@ alternative_else_nop_endif .if \el == 0 ldr x23, [sp, #S_SP] // load return stack pointer msr sp_el0, x23 + msr tpidrro_el0, xzr tst x22, #PSR_MODE32_BIT // native task? b.eq 3f + ldr x0, [tsk, #THREAD_TP_VALUE] + msr tpidrro_el0, x0
Quoted 56 lines
arch/arm64/kernel/entry.S +0
-0#ifdef CONFIG_ARM64_ERRATUM_845719 alternative_if ARM64_WORKAROUND_845719 #ifdef CONFIG_PID_IN_CONTEXTIDRarch/arm64/kernel/process.c +1
-15@@ -256,17 +256,8 @@ static void tls_thread_flush(void)if (system_supports_tpidr2 ())write_sysreg_s (0 ,SYS_TPIDR2_EL0 ); -if (is_compat_task ()) { +if (is_compat_task ()) current->thread.uw.tp_value =0 ; - -/* -* We need to ensure ordering between the shadow state and the -* hardware state, so that we don't corrupt the hardware state -* with a stale shadow state during context switch. -*/ -barrier (); -write_sysreg (0 , tpidrro_el0); - } }static void flush_tagged_addr_state (void )@@ -531,11 +522,6 @@ static void tls_thread_switch(struct task_struct *next) {tls_preserve_current_state (); -if (is_compat_thread (task_thread_info (next))) -write_sysreg (next->thread.uw.tp_value, tpidrro_el0); -else -write_sysreg (0 , tpidrro_el0); -write_sysreg (*task_user_tls (next), tpidr_el0);if (system_supports_tpidr2 ())write_sysreg_s (next->thread.tpidr2_el0,SYS_TPIDR2_EL0 );arch/arm64/kernel/sys_compat.c +0
-7@@ -89,13 +89,6 @@ long compat_arm_syscall(struct pt_regs *regs, int scno)case __ARM_NR_compat_set_tls: current->thread.uw.tp_value = regs->regs[0 ]; - -/* -* Protect against register corruption from context switch. -* See comment in tls_thread_flush. -*/ -barrier (); -write_sysreg (regs->regs[0 ], tpidrro_el0);return 0 ;default :
Will Deacon
maintainerPATCH 2/21 arm64: entry: Only check for stack overflow on exceptions from EL1
In reply to Will DeaconChecking for kernel stack overflow on entry from EL0 is largely pointless and gets in the way of using SPSel to switch to the overflow stack. If the kernel stack overflows on entry from userspace, that implies that it was unbalanced on the last exception return which means we can't sensibly reason about the state of the system given that we could've returned to an arbitrary virtual address in an arbitrary mode.
Remove the stack overflow check from the EL0 exception entry path.
--- arch/arm64/kernel/entry.S | 6 ++++++ 1 file changed, 6 insertions(+)
arch/arm64/kernel/entry.S +6 -0
@@ -54,6 +54,8 @@ .endif sub sp, sp, #PT_REGS_SIZE + + .if \el == 1 /* * Test whether the SP has overflowed, without corrupting a GPR. * Task and IRQ stacks are aligned so that SP & (1 << THREAD_SHIFT)
@@ -64,8 +66,11 @@ tbnz x0, #THREAD_SHIFT, 0f sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp + .endif + b el\el\ht\()_\regsize\()_\label + .if \el == 1 0: /* * Either we've just detected an overflow, or we've taken an exception
@@ -96,6 +101,7 @@ sub sp, sp, x0 mrs x0, tpidrro_el0 b el\el\ht\()_\regsize\()_\label + .endif .org .Lventry_start\@ + 128 // Did we overflow the ventry slot? .endm
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 3/21 arm64: stackprotector: Temporarily disable per-task stackprotector
In reply to Will DeaconPer-task stackprotector relies on telling the compiler about the system register used to hold a pointer to the current task. As we're about to change that around, temporarily disable per-task stackprotector for now.
--- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/arm64/Kconfig +1 -0
@@ -2455,6 +2455,7 @@ config CC_HAVE_STACKPROTECTOR_SYSREG config STACKPROTECTOR_PER_TASK def_bool y depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_SYSREG + depends on BROKEN config UNWIND_PATCH_PAC_INTO_SCS bool "Enable shadow call stack dynamically using code patching"
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 4/21 arm64: bpf: Add support for generating reads of TPIDRRO_EL0
In reply to Will DeaconIn preparation for storing 'current' in TPIDRRO_EL0, add support for the instruction encoding of "MRS Rt, TPIDRRO_EL0" to the BPF JIT.
--- arch/arm64/include/asm/insn.h | 1 + arch/arm64/net/bpf_jit.h | 2 ++ 2 files changed, 3 insertions(+)
arch/arm64/include/asm/insn.h +1 -0
@@ -136,6 +136,7 @@ enum aarch64_insn_special_register { };enum aarch64_insn_system_register { +AARCH64_INSN_SYSREG_TPIDRRO_EL0 =0x5e83 ,AARCH64_INSN_SYSREG_TPIDR_EL1 =0x4684 ,AARCH64_INSN_SYSREG_TPIDR_EL2 =0x6682 ,AARCH64_INSN_SYSREG_SP_EL0 =0x4208 ,
arch/arm64/net/bpf_jit.h +2 -0
@@ -331,6 +331,8 @@aarch64_insn_gen_adr (0 , offset, Rd,AARCH64_INSN_ADR_TYPE_ADR ) /* MRS */ +#define A64_MRS_TPIDRRO_EL0 (Rt) \ + aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_TPIDRRO_EL0)#define A64_MRS_TPIDR_EL1 (Rt) \ aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_TPIDR_EL1)#define A64_MRS_TPIDR_EL2 (Rt) \
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 5/21 KVM: arm64: Protect TPIDRRO_EL0 across guest entry/exit
In reply to Will DeaconFrom: Mostafa Saleh <[email protected]>
To prepare for using TPIDRRO_EL0 to hold 'current' and repurposing SP_EL0 as an overflow stack pointer, eagerly switch both registers when entering and exiting the guest.
--- arch/arm64/include/asm/kvm_asm.h | 10 ++++++++++ arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kvm/hyp/entry.S | 12 ++++++++---- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 2 -- 4 files changed, 19 insertions(+), 6 deletions(-)
arch/arm64/include/asm/kvm_asm.h +10 -0
@@ -408,6 +408,16 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt, msr sp_el0, \tmp .endm +.macro save_tpidrro_el0 ctxt, tmp + mrs \tmp, tpidrro_el0 + str \tmp, [\ctxt, #CPU_TPIDRRO_EL0_OFFSET] +.endm + +.macro restore_tpidrro_el0 ctxt, tmp + ldr \tmp, [\ctxt, #CPU_TPIDRRO_EL0_OFFSET] + msr tpidrro_el0, \tmp +.endm + #endif #endif /* __ARM_KVM_ASM_H__ */
arch/arm64/kernel/asm-offsets.c +1 -0
@@ -108,6 +108,7 @@ int main(void) DEFINE(VCPU_HCR_EL2, offsetof(struct kvm_vcpu, arch.hcr_el2));DEFINE (CPU_USER_PT_REGS , offsetof(struct kvm_cpu_context , regs)); DEFINE(CPU_ELR_EL2, offsetof(struct kvm_cpu_context, sys_regs[ELR_EL2])); + DEFINE(CPU_TPIDRRO_EL0_OFFSET, offsetof(struct kvm_cpu_context, sys_regs[TPIDRRO_EL0])); DEFINE(CPU_RGSR_EL1, offsetof(struct kvm_cpu_context, sys_regs[RGSR_EL1])); DEFINE(CPU_GCR_EL1, offsetof(struct kvm_cpu_context, sys_regs[GCR_EL1])); DEFINE(CPU_APIAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APIAKEYLO_EL1]));
arch/arm64/kvm/hyp/entry.S +8 -4
@@ -30,8 +30,9 @@ SYM_FUNC_START(__guest_enter) // Store the hyp regs save_callee_saved_regs x1 - // Save hyp's sp_el0 + // Save hyp's sp_el0 and tpidrro_el0 save_sp_el0 x1, x2 + save_tpidrro_el0 x1, x2 // Now the hyp state is stored if we have a pending RAS SError it must // affect the host or hyp. If any asynchronous exception is pending we
@@ -66,8 +67,9 @@ alternative_else_nop_endif // when this feature is enabled for kernel code. ptrauth_switch_to_guest x29, x0, x1, x2 - // Restore the guest's sp_el0 + // Restore the guest's sp_el0 and tpidrro_el0 restore_sp_el0 x29, x0 + restore_tpidrro_el0 x29, x0 // Restore guest regs x0-x17 ldp x0, x1, [x29, #CPU_XREG_OFFSET(0)]
@@ -146,8 +148,9 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL) // Store the guest regs x18-x29, lr save_callee_saved_regs x1 - // Store the guest's sp_el0 + // Store the guest's sp_el0 and tpidrro_el0 save_sp_el0 x1, x2 + save_tpidrro_el0 x1, x2 adr_this_cpu x2, kvm_hyp_ctxt, x3
@@ -161,8 +164,9 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL) // mte_switch_to_hyp(g_ctxt, h_ctxt, reg1) mte_switch_to_hyp x1, x2, x3 - // Restore hyp's sp_el0 + // Restore hyp's sp_el0 and tpidrro_el0 restore_sp_el0 x2, x3 + restore_tpidrro_el0 x2, x3 // Now restore the hyp regs restore_callee_saved_regs x2
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +0 -2
@@ -66,7 +66,6 @@ static inline void __sysreg_save_common_state(struct kvm_cpu_context *ctxt)static inline void __sysreg_save_user_state (struct kvm_cpu_context *ctxt) {ctxt_sys_reg (ctxt,TPIDR_EL0 ) =read_sysreg (tpidr_el0); -ctxt_sys_reg (ctxt,TPIDRRO_EL0 ) =read_sysreg (tpidrro_el0); }static inline bool ctxt_has_mte (struct kvm_cpu_context *ctxt)
@@ -209,7 +208,6 @@ static inline void __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)static inline void __sysreg_restore_user_state (struct kvm_cpu_context *ctxt) {write_sysreg (ctxt_sys_reg (ctxt,TPIDR_EL0 ), tpidr_el0); -write_sysreg (ctxt_sys_reg (ctxt,TPIDRRO_EL0 ), tpidrro_el0); } static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt,
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 6/21 arm64: Store 'current' in TPIDRRO_EL0 instead of SP_EL0
In reply to Will DeaconTo make SP_EL0 available to point at the per-cpu kernel overflow stack, we must relocate 'current' elsewhere. With TPIDRRO_EL0 now restored late in the return-to-user path, we can otherwise use it to hold 'current' and update the various accessors accordingly.
Move 'current' from SP_EL0 to TPIDRRO_EL0.
--- arch/arm64/include/asm/assembler.h | 2 +- arch/arm64/include/asm/current.h | 6 +++--- arch/arm64/kernel/entry.S | 20 ++++++++++---------- arch/arm64/kernel/head.S | 2 +- arch/arm64/kernel/process.c | 5 +++-- arch/arm64/net/bpf_jit_comp.c | 4 ++-- 6 files changed, 20 insertions(+), 19 deletions(-)
arch/arm64/include/asm/assembler.h +1 -1
@@ -587,7 +587,7 @@ alternative_else_nop_endif* Return the current task_struct. */ .macro get_current_task, rd - mrs \rd, sp_el0 + mrs \rd, tpidrro_el0 .endm/*
arch/arm64/include/asm/current.h +3 -3
@@ -14,11 +14,11 @@ struct task_struct;*/ static __always_inline struct task_struct *get_current(void) { -unsigned long sp_el0; +unsigned long tpidrro_el0; - asm ("mrs %0, sp_el0" :"=r" (sp_el0)); + asm ("mrs %0, tpidrro_el0" :"=r" (tpidrro_el0)); -return (struct task_struct *)sp_el0; +return (struct task_struct *)tpidrro_el0; }#define current get_current()
arch/arm64/kernel/entry.S +10 -10
@@ -81,9 +81,9 @@ /* Stash the original SP (minus PT_REGS_SIZE) in tpidr_el0. */ msr tpidr_el0, x0 - /* Recover the original x0 value and stash it in tpidrro_el0 */ + /* Recover the original x0 value and stash it in sp_el0 */ sub x0, sp, x0 - msr tpidrro_el0, x0 + msr sp_el0, x0 /* Switch to the overflow stack */ adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0
@@ -99,7 +99,7 @@ /* We were already on the overflow stack. Restore sp/x0 and carry on. */ sub sp, sp, x0 - mrs x0, tpidrro_el0 + mrs x0, sp_el0 b el\el\ht\()_\regsize\()_\label .endif .org .Lventry_start\@ + 128 // Did we overflow the ventry slot?
@@ -226,7 +226,7 @@ alternative_cb_end clear_gp_regs mrs x21, sp_el0 ldr_this_cpu tsk, __entry_task, x20 - msr sp_el0, tsk + msr tpidrro_el0, tsk /* * Ensure MDSCR_EL1.SS is clear, since we can unmask debug exceptions
@@ -548,7 +548,7 @@ SYM_CODE_START_LOCAL(__bad_stack) */ /* Restore the original x0 value */ - mrs x0, tpidrro_el0 + mrs x0, sp_el0 /* * Store the original GPRs to the new stack. The orginal SP (minus
@@ -843,7 +843,7 @@ SYM_FUNC_START(cpu_switch_to) ldp x29, x9, [x8], #16 ldr lr, [x8] mov sp, x9 - msr sp_el0, x1 + msr tpidrro_el0, x1 ptrauth_keys_install_kernel x1, x8, x9, x10 scs_save x0 scs_load_current
@@ -1031,11 +1031,11 @@ SYM_CODE_START(__sdei_asm_handler) /* * We may have interrupted userspace, or a guest, or exit-from or - * return-to either of these. We can't trust sp_el0, restore it. + * return-to either of these. We can't trust tpidrro_el0, restore it. */ - mrs x28, sp_el0 + mrs x28, tpidrro_el0 ldr_this_cpu dst=x0, sym=__entry_task, tmp=x1 - msr sp_el0, x0 + msr tpidrro_el0, x0 /* If we interrupted the kernel point to the previous stack/frame. */ and x0, x3, #0xc
@@ -1051,7 +1051,7 @@ SYM_CODE_START(__sdei_asm_handler) mov x1, x19 bl __sdei_handler - msr sp_el0, x28 + msr tpidrro_el0, x28 /* restore regs >x17 that we clobbered */ mov x4, x19 // keep x4 for __sdei_asm_exit_trampoline ldp x28, x29, [x4, #SDEI_EVENT_INTREGS + 16 * 14]
arch/arm64/kernel/head.S +1 -1
@@ -193,7 +193,7 @@ SYM_CODE_END(preserve_boot_args) * for consistency with user tasks and kthreads. */ .macro init_cpu_task tsk, tmp1, tmp2 - msr sp_el0, \tsk + msr tpidrro_el0, \tsk ldr \tmp1, [\tsk, #TSK_STACK] add sp, \tmp1, #THREAD_SIZE
arch/arm64/kernel/process.c +3 -2
@@ -551,8 +551,9 @@ static void ssbs_thread_switch(struct task_struct *next) }/* -* We store our current task in sp_el0, which is clobbered by userspace. Keep a -* shadow copy so that we can restore this upon entry from userspace. +* We store our current task in tpidrro_el0, which is clobbered when returning +* to userspace. Keep a shadow copy so that we can restore this upon entry from +* userspace. * * This is *only* for exception entry from EL0, and is not valid until we * __switch_to() a user task.
arch/arm64/net/bpf_jit_comp.c +2 -2
@@ -1694,7 +1694,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insnif (insn->src_reg ==0 && insn->imm == BPF_FUNC_get_smp_processor_id) { cpu_offset = offsetof(struct thread_info , cpu); -emit (A64_MRS_SP_EL0 (tmp), ctx); +emit (A64_MRS_TPIDRRO_EL0 (tmp), ctx);if (is_lsi_offset (cpu_offset,2 )) {emit (A64_LDR32I (r0, tmp, cpu_offset), ctx); }else {
@@ -1707,7 +1707,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn/* Implement helper call to bpf_get_current_task/_btf() inline */ if (insn->src_reg ==0 && (insn->imm == BPF_FUNC_get_current_task || insn->imm == BPF_FUNC_get_current_task_btf)) { -emit (A64_MRS_SP_EL0 (r0), ctx); +emit (A64_MRS_TPIDRRO_EL0 (r0), ctx);break ; }
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Re: [PATCH 06/21] arm64: Store 'current' in TPIDRRO_EL0 instead of SP_EL0
Quoted 6 lines · On Mon, 7 Sep 2026 17:42:31 +0100 Will Deacon <[email protected]> wrote:
To make
SP_EL0available to point at the per-cpu kernel overflow stack, we must relocate 'current' elsewhere. WithTPIDRRO_EL0now restored late in the return-to-user path, we can otherwise use it to hold 'current' and update the various accessors accordingly.Move 'current' from
SP_EL0toTPIDRRO_EL0.
Is it worth having a #define for the name of the register that stores 'current'? (And probably the other similar registers.)
That would avoid the bisection problems with the following two patches and might make other code a bit more readable.
David
Quoted 171 lines
Cc Mark Rutland <[email protected]>Signed-off-by Will Deacon <[email protected]>--- arch/arm64/include/asm/assembler.h | 2 +- arch/arm64/include/asm/current.h | 6 +++--- arch/arm64/kernel/entry.S | 20 ++++++++++---------- arch/arm64/kernel/head.S | 2 +- arch/arm64/kernel/process.c | 5 +++-- arch/arm64/net/bpf_jit_comp.c | 4 ++-- 6 files changed, 20 insertions(+), 19 deletions(-)arch/arm64/include/asm/assembler.h +1
-1@@ -587,7 +587,7 @@ alternative_else_nop_endif* Return the current task_struct. */ .macro get_current_task, rd - mrs \rd, sp_el0 + mrs \rd, tpidrro_el0 .endm/* arch/arm64/include/asm/current.h +3
-3@@ -14,11 +14,11 @@ struct task_struct;*/ static __always_inline struct task_struct *get_current(void) { -unsigned long sp_el0; +unsigned long tpidrro_el0; - asm ("mrs %0, sp_el0" :"=r" (sp_el0)); + asm ("mrs %0, tpidrro_el0" :"=r" (tpidrro_el0)); -return (struct task_struct *)sp_el0; +return (struct task_struct *)tpidrro_el0; }#define current get_current()arch/arm64/kernel/entry.S +10
-10@@ -81,9 +81,9 @@ /* Stash the original SP (minus PT_REGS_SIZE) in tpidr_el0. */ msr tpidr_el0, x0 - /* Recover the original x0 value and stash it in tpidrro_el0 */ + /* Recover the original x0 value and stash it in sp_el0 */ sub x0, sp, x0 - msr tpidrro_el0, x0 + msr sp_el0, x0 /* Switch to the overflow stack */ adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0@@ -99,7 +99,7 @@ /* We were already on the overflow stack. Restore sp/x0 and carry on. */ sub sp, sp, x0 - mrs x0, tpidrro_el0 + mrs x0, sp_el0 b el\el\ht\()_\regsize\()_\label .endif .org .Lventry_start\@ + 128 // Did we overflow the ventry slot?@@ -226,7 +226,7 @@ alternative_cb_end clear_gp_regs mrs x21, sp_el0 ldr_this_cpu tsk, __entry_task, x20 - msr sp_el0, tsk + msr tpidrro_el0, tsk /* * Ensure MDSCR_EL1.SS is clear, since we can unmask debug exceptions@@ -548,7 +548,7 @@ SYM_CODE_START_LOCAL(__bad_stack) */ /* Restore the original x0 value */ - mrs x0, tpidrro_el0 + mrs x0, sp_el0 /* * Store the original GPRs to the new stack. The orginal SP (minus@@ -843,7 +843,7 @@ SYM_FUNC_START(cpu_switch_to) ldp x29, x9, [x8], #16 ldr lr, [x8] mov sp, x9 - msr sp_el0, x1 + msr tpidrro_el0, x1 ptrauth_keys_install_kernel x1, x8, x9, x10 scs_save x0 scs_load_current@@ -1031,11 +1031,11 @@ SYM_CODE_START(__sdei_asm_handler) /* * We may have interrupted userspace, or a guest, or exit-from or - * return-to either of these. We can't trust sp_el0, restore it. + * return-to either of these. We can't trust tpidrro_el0, restore it. */ - mrs x28, sp_el0 + mrs x28, tpidrro_el0 ldr_this_cpu dst=x0, sym=__entry_task, tmp=x1 - msr sp_el0, x0 + msr tpidrro_el0, x0 /* If we interrupted the kernel point to the previous stack/frame. */ and x0, x3, #0xc@@ -1051,7 +1051,7 @@ SYM_CODE_START(__sdei_asm_handler) mov x1, x19 bl __sdei_handler - msr sp_el0, x28 + msr tpidrro_el0, x28 /* restore regs >x17 that we clobbered */ mov x4, x19 // keep x4 for __sdei_asm_exit_trampoline ldp x28, x29, [x4, #SDEI_EVENT_INTREGS + 16 * 14]arch/arm64/kernel/head.S +1
-1@@ -193,7 +193,7 @@ SYM_CODE_END(preserve_boot_args) * for consistency with user tasks and kthreads. */ .macro init_cpu_task tsk, tmp1, tmp2 - msr sp_el0, \tsk + msr tpidrro_el0, \tsk ldr \tmp1, [\tsk, #TSK_STACK] add sp, \tmp1, #THREAD_SIZEarch/arm64/kernel/process.c +3
-2@@ -551,8 +551,9 @@ static void ssbs_thread_switch(struct task_struct *next) }/* -* We store our current task in sp_el0, which is clobbered by userspace. Keep a -* shadow copy so that we can restore this upon entry from userspace. +* We store our current task in tpidrro_el0, which is clobbered when returning +* to userspace. Keep a shadow copy so that we can restore this upon entry from +* userspace. * * This is *only* for exception entry from EL0, and is not valid until we * __switch_to() a user task. arch/arm64/net/bpf_jit_comp.c +2
-2@@ -1694,7 +1694,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insnif (insn->src_reg ==0 && insn->imm == BPF_FUNC_get_smp_processor_id) { cpu_offset = offsetof(struct thread_info , cpu); -emit (A64_MRS_SP_EL0 (tmp), ctx); +emit (A64_MRS_TPIDRRO_EL0 (tmp), ctx);if (is_lsi_offset (cpu_offset,2 )) {emit (A64_LDR32I (r0, tmp, cpu_offset), ctx); }else {@@ -1707,7 +1707,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn/* Implement helper call to bpf_get_current_task/_btf() inline */ if (insn->src_reg ==0 && (insn->imm == BPF_FUNC_get_current_task || insn->imm == BPF_FUNC_get_current_task_btf)) { -emit (A64_MRS_SP_EL0 (r0), ctx); +emit (A64_MRS_TPIDRRO_EL0 (r0), ctx);break ; }
Will Deacon
maintainerRe: [PATCH 06/21] arm64: Store 'current' in TPIDRRO_EL0 instead of SP_EL0
Quoted 8 lines · David Laight
Quoted 5 lines · On Mon, 7 Sep 2026 17:42:31 +0100 Will Deacon <[email protected]> wrote:
To make
SP_EL0available to point at the per-cpu kernel overflow stack, we must relocate 'current' elsewhere. WithTPIDRRO_EL0now restored late in the return-to-user path, we can otherwise use it to hold 'current' and update the various accessors accordingly.
Quoted 6 lines
Quoted 1 line
Move 'current' from
SP_EL0toTPIDRRO_EL0.Is it worth having a #define for the name of the register that stores 'current'? (And probably the other similar registers.)
That would avoid the bisection problems with the following two patches and might make other code a bit more readable.
We don't tend to reallocate 'current' all that often, so I don't think it's worth the indirection tbh. The following patches are to a GDB script and a bpf selftest, so I'm not going to lose sleep about bisection.
Will
Will Deacon
maintainerPATCH 7/21 selftests/bpf: arm64: Use TPIDRRO_EL0 instead of SP_EL0 for 'current'
In reply to Will DeaconThe current task pointer now lives in TPIDRRO_EL0 instead of SP_EL0, so update the bpf selftests with the new register allocation.
--- tools/testing/selftests/bpf/progs/verifier_jit_inline.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/bpf/progs/verifier_jit_inline.c +1 -1
@@ -9,7 +9,7 @@ __success __retval(0) __arch_x86_64 __jited(" addq %gs:{{.*}}, %rax") __arch_arm64 -__jited(" mrs x8, SP_EL0") +__jited(" mrs x8, TPIDRRO_EL0") __arch_riscv64 __jited(" mv a5, tp")__arch_loongarch
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 8/21 scripts/gdb: arm64: Use TPIDRRO_EL0 instead of SP_EL0 for 'current'
In reply to Will DeaconThe current task pointer now lives in TPIDRRO_EL0 instead of SP_EL0, so update the gdb 'cpus.py' helper script with the new register allocation.
--- scripts/gdb/linux/cpus.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/gdb/linux/cpus.py +1 -1
@@ -185,7 +185,7 @@ def get_current_task(cpu): var_ptr = gdb.parse_and_eval("¤t_task") return per_cpu(var_ptr, cpu).dereference() elif utils.is_target_arch("aarch64"): - current_task_addr = gdb.parse_and_eval("(unsigned long)$SP_EL0") + current_task_addr = gdb.parse_and_eval("(unsigned long)$TPIDRRO_EL0") if (current_task_addr >> 63) != 0: current_task = current_task_addr.cast(task_ptr_type) return current_task.dereference()
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 9/21 arm64: stackprotector: Re-enable per-task stackprotector
In reply to Will DeaconNow that 'current' resides in TPIDRRO_EL0, update the per-task stackprotector compiler invocation to use the new register.
Note that this requires a very recent build of clang [1], as (unlike GCC), it historically rejected all registers other than SP_EL0.
--- arch/arm64/Kconfig | 3 +-- arch/arm64/Makefile | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-)
arch/arm64/Kconfig +1 -2
@@ -2450,12 +2450,11 @@ config RANDOMIZE_MODULE_REGION_FULL exhaustion, modules might be able to fall back to a larger 2GB area. config CC_HAVE_STACKPROTECTOR_SYSREG - def_bool $(cc-option,-mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=0) + def_bool $(cc-option,-mstack-protector-guard=sysreg -mstack-protector-guard-reg=tpidrro_el0 -mstack-protector-guard-offset=0) config STACKPROTECTOR_PER_TASK def_bool y depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_SYSREG - depends on BROKEN config UNWIND_PATCH_PAC_INTO_SCS bool "Enable shadow call stack dynamically using code patching"
arch/arm64/Makefile +1 -1
@@ -72,7 +72,7 @@ ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) prepare: stack_protector_prepare stack_protector_prepare: prepare0 $(eval KBUILD_CFLAGS += -mstack-protector-guard=sysreg \ - -mstack-protector-guard-reg=sp_el0 \ + -mstack-protector-guard-reg=tpidrro_el0 \ -mstack-protector-guard-offset=$(shell \ awk '{if ($$2 == "TSK_STACK_CANARY") print $$3;}' \ $(objtree)/include/generated/asm-offsets.h))
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 10/21 arm64: percpu: Specialise set_my_cpu_offset() for the primary CPU
In reply to Will Deaconset_my_cpu_offset() is only called from smp_prepare_boot_cpu() which runs once on the primary CPU and before alternatives have been applied.
Rework the function so that it operates only on the boot CPU offset, drop the unused alternative code sequence and internalise it into smp.c alongside its only caller.
--- arch/arm64/include/asm/percpu.h | 8 -------- arch/arm64/kernel/smp.c | 8 +++++++- 2 files changed, 7 insertions(+), 9 deletions(-)
arch/arm64/include/asm/percpu.h +0 -8
@@ -12,14 +12,6 @@#include <asm/stack_pointer.h> #include <asm/sysreg.h> -static inline void set_my_cpu_offset (unsigned long off) -{ - asm volatile(ALTERNATIVE("msr tpidr_el1, %0", - "msr tpidr_el2, %0", - ARM64_HAS_VIRT_HOST_EXTN) - :: "r" (off) : "memory"); -} -static inline unsigned long __hyp_my_cpu_offset (void ) {/*
arch/arm64/kernel/smp.c +7 -1
@@ -446,6 +446,12 @@ void __init smp_cpus_done(unsigned int max_cpus)mark_linear_text_alias_ro (); } +static void __init set_boot_cpu_offset (void ) +{ + asmvolatile ("msr tpidr_el1, %0" + ::"r" (per_cpu_offset (0 )) :"memory" ); +} +void __init smp_prepare_boot_cpu (void ) {/*
@@ -453,7 +459,7 @@ void __init smp_prepare_boot_cpu(void)* setup_per_cpu_areas(), and CPU0's boot time per-cpu area will be * freed shortly, so we must move over to the runtime per-cpu area. */ -set_my_cpu_offset (per_cpu_offset (smp_processor_id ())); +set_boot_cpu_offset ();cpuinfo_store_boot_cpu ();setup_boot_cpu_features ();
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 11/21 arm64: percpu: Annotate __kern_my_cpu_offset() as '__always_inline'
In reply to Will DeaconAllow the use of per-cpu variables from noinstr functions by marking __kern_my_cpu_offset() as '__always_inline'. This will later be relied upon to detect stack overflow of the overflow stack from noinstr C code.
--- arch/arm64/include/asm/percpu.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/percpu.h +1 -1
@@ -21,7 +21,7 @@ static inline unsigned long __hyp_my_cpu_offset(void)return read_sysreg (tpidr_el2); } -static inline unsigned long __kern_my_cpu_offset (void ) +static __always_inline unsigned long __kern_my_cpu_offset (void ) {unsigned long off;
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 12/21 KVM: arm64: Preserve handler/thread bit of EL1 mode in __finalise_el2()
In reply to Will Deacon__finalise_el2() unconditionally returns to EL2h when VHE is being used. In preparation for handling callers from both EL1h and EL1t, preserve the handler/thread bit of the calling mode when rewriting the exception level in the SPSR.
--- arch/arm64/include/uapi/asm/ptrace.h | 1 + arch/arm64/kernel/hyp-stub.S | 7 ++++--- 2 files changed, 5 insertions(+), 3 deletions(-)
arch/arm64/include/uapi/asm/ptrace.h +1 -0
@@ -37,6 +37,7 @@#define PSR_MODE_EL3t 0x0000000c#define PSR_MODE_EL3h 0x0000000d#define PSR_MODE_MASK 0x0000000f +#define PSR_MODE_h_BIT 0x00000001/* AArch32 CPSR bits */ #define PSR_MODE32_BIT 0x00000010
arch/arm64/kernel/hyp-stub.S +4 -3
@@ -149,10 +149,11 @@ SYM_CODE_START_LOCAL(__finalise_el2) isb - // Hack the exception return to stay at EL2 + // Hack the exception return to stay at EL2 ... mrs x0, spsr_el1 - and x0, x0, #~PSR_MODE_MASK - mov x1, #PSR_MODE_EL2h + // ... preserving the 'h' bit so we don't swizzle SP. + and x0, x0, #(~PSR_MODE_MASK | PSR_MODE_h_BIT) + mov x1, #PSR_MODE_EL2t orr x0, x0, x1 msr spsr_el1, x0
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 13/21 arm64: sdei: Guard most of asm/sdei.h with CONFIG_ARM_SDE_INTERFACE
In reply to Will DeaconThe SDEI header includes are a bit of a mess: linux/arm_sdei.h only includes asm/sdei.h if CONFIG_ARM_SDE_INTERFACE=y and then goes on to provide a definition of the sdei_arch_get_entry_point() macro if it hasn't already been defined. asm/sdei.h, however, defines the same macro unconditionally and so this breaks the build if CONFIG_ARM_SDE_INTERFACE is not set and asm/sdei.h is included after linux/arm_sdei.h.
The kbuild robot ran into this situation with randconfig testing, thanks to the transitive inclusion of asm/sdei.h via asm/stacktrace.h.
Guard most of asm/sdei.h with CONFIG_ARM_SDE_INTERFACE to avoid redefining macros provided by linux/arm_sdei.h.
--- arch/arm64/include/asm/sdei.h | 2 ++ 1 file changed, 2 insertions(+)
arch/arm64/include/asm/sdei.h +2 -0
@@ -9,6 +9,7 @@#define SDEI_STACK_SIZE IRQ_STACK_SIZE +#ifdef CONFIG_ARM_SDE_INTERFACE #ifndef __ASSEMBLER__#include <linux/linkage.h>
@@ -50,4 +51,5 @@ unsigned long sdei_arch_get_entry_point(int conduit);#define sdei_arch_get_entry_point (x) sdei_arch_get_entry_point(x)#endif /* __ASSEMBLER__ */ +#endif /* CONFIG_ARM_SDE_INTERFACE */ #endif /* __ASM_SDEI_H */
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 14/21 arm64: sdei: Support SDEI events from kernel handler and thread modes
In reply to Will DeaconSection 5.2.2 ("Event resume context") of the SDEI specification states that, as a result of an SDEI_EVENT_COMPLETE_AND_RESUME call:
| SPSR_ELc is set to the PSTATE when the event was taken
In preparation for making use of both thread and handler modes in the kernel, adjust the SDEI event handler so that we return to the correct kernel vector for the interrupted mode.
--- arch/arm64/kernel/sdei.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/arm64/kernel/sdei.c +3 -3
@@ -208,7 +208,7 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,int i, err =0 ;int clobbered_registers =4 ;u64 elr =read_sysreg (elr_el1); -u32 kernel_mode =read_sysreg (CurrentEL) |1 ;/* +SPSel */ +u32 kernel_mode =read_sysreg (CurrentEL);unsigned long vbar =read_sysreg (vbar_el1);if (arm64_kernel_unmapped_at_el0 ())
@@ -233,7 +233,7 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,pr_warn ("unsafe: exception during handler\n" ); } - mode = regs->pstate & (PSR_MODE32_BIT |PSR_MODE_MASK ); + mode = regs->pstate & (PSR_MODE32_BIT |PSR_MODE_MASK ) & ~PSR_MODE_h_BIT;/* * If we interrupted the kernel with interrupts masked, we always go
@@ -251,7 +251,7 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,* address'. */ if (mode == kernel_mode) -return vbar +0x280 ; +return vbar + (regs->pstate & PSR_MODE_h_BIT ?0x280 :0x80 );else if (mode &PSR_MODE32_BIT )return vbar +0x680 ;
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 15/21 arm64: entry: Point SP_EL0 at the overflow stack
In reply to Will DeaconWith SP_EL0 now available for use in the kernel, point it at the overflow stack so that exception entry from EL1h will later be able to switch to the overflow stack by writing to SPSel.
--- arch/arm64/kernel/entry.S | 2 ++ arch/arm64/kernel/head.S | 3 +++ arch/arm64/kernel/smp.c | 12 ++++++++++-- 3 files changed, 15 insertions(+), 2 deletions(-)
arch/arm64/kernel/entry.S +2 -0
@@ -227,6 +227,8 @@ alternative_cb_end mrs x21, sp_el0 ldr_this_cpu tsk, __entry_task, x20 msr tpidrro_el0, tsk + adr_this_cpu x19, overflow_stack + OVERFLOW_STACK_SIZE, x20 + msr sp_el0, x19 /* * Ensure MDSCR_EL1.SS is clear, since we can unmask debug exceptions
arch/arm64/kernel/head.S +3 -0
@@ -210,6 +210,9 @@ SYM_CODE_END(preserve_boot_args) ldr w\tmp2, [\tsk, #TSK_TI_CPU] ldr \tmp1, [\tmp1, \tmp2, lsl #3] set_this_cpu_offset \tmp1 + + adr_this_cpu \tmp1, overflow_stack + OVERFLOW_STACK_SIZE, \tmp2 + msr sp_el0, \tmp1 .endm /*
arch/arm64/kernel/smp.c +10 -2
@@ -51,6 +51,7 @@#include <asm/processor.h> #include <asm/smp_plat.h> #include <asm/sections.h> +#include <asm/stacktrace.h> #include <asm/tlbflush.h> #include <asm/ptrace.h> #include <asm/virt.h>
@@ -448,8 +449,15 @@ void __init smp_cpus_done(unsigned int max_cpus)static void __init set_boot_cpu_offset (void ) { - asmvolatile ("msr tpidr_el1, %0" - ::"r" (per_cpu_offset (0 )) :"memory" ); +u64 ovf_sp = (u64 )raw_cpu_ptr (overflow_stack) +OVERFLOW_STACK_SIZE ; + + asmvolatile ( +" msr tpidr_el1, %1\n" +" add %0, %0, %1\n" +" msr sp_el0, %0" /* Update the overflow stack pointer */ + :"+r" (ovf_sp) + :"r" (per_cpu_offset (0 )) + :"memory" ); }void __init smp_prepare_boot_cpu (void )
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 16/21 arm64: entry: Implement EL1t exception handlers for overflow stack
In reply to Will DeaconIn preparation for using EL1t to handle exceptions taken whilst running on the overflow stack, implement some simple wrappers around the EL1h handlers which perform a best-effort stack overflow check before proceeding.
--- arch/arm64/kernel/entry-common.c | 43 +++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 4 deletions(-)
arch/arm64/kernel/entry-common.c +39 -4
@@ -327,10 +327,45 @@ static void debug_exception_exit(struct pt_regs *regs) }NOKPROBE_SYMBOL (debug_exception_exit); -UNHANDLED(el1t, 64, sync) -UNHANDLED(el1t, 64, irq) -UNHANDLED(el1t, 64, fiq) -UNHANDLED(el1t, 64, error) +static void noinstr el1t_64_check_overflow_stack(struct pt_regs *regs) +{ +unsigned long sp =kernel_stack_pointer (regs) -sizeof (*regs); +unsigned long ovf_stack = (unsigned long )this_cpu_ptr (overflow_stack); + +/* +* We're in big trouble if we've overflowed the overflow stack +* so perform a best-effort check before we proceed. If our SP +* is outside of the overflow stack for this CPU then presumably +* we're already corrupting memory, so park ourselves here in an +* attempt to contain the damage. +*/ +if (sp < ovf_stack || sp > ovf_stack +OVERFLOW_STACK_SIZE ) +cpu_park_loop (); +} + +asmlinkage void noinstr el1t_64_sync_handler(struct pt_regs *regs) +{ +el1t_64_check_overflow_stack (regs); +el1h_64_sync_handler (regs); +} + +asmlinkage void noinstr el1t_64_irq_handler(struct pt_regs *regs) +{ +el1t_64_check_overflow_stack (regs); +el1h_64_irq_handler (regs); +} + +asmlinkage void noinstr el1t_64_fiq_handler(struct pt_regs *regs) +{ +el1t_64_check_overflow_stack (regs); +el1h_64_fiq_handler (regs); +} + +asmlinkage void noinstr el1t_64_error_handler(struct pt_regs *regs) +{ +el1t_64_check_overflow_stack (regs); +el1h_64_error_handler (regs); +} static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr) {
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 17/21 arm64: entry: Use SPSel to switch to overflow stack
In reply to Will DeaconWhen detecting a stack overflow on exception entry from EL1, use SPSel to switch to the overflow stack without corrupting any GPRs. Not only is this simpler than the previous logic, but it also opens the door to more complicated overflow checks (for example, based on per-task stack sizes or stacks which are not aligned to a specific page order) as well as the possibility of returning from a stack fault if we were able to resolve it.
--- arch/arm64/kernel/entry.S | 69 ++++++++++++--------------------------- 1 file changed, 21 insertions(+), 48 deletions(-)
arch/arm64/kernel/entry.S +21 -48
@@ -51,11 +51,11 @@ mov x30, xzr .endif .Lskip_tramp_vectors_cleanup\@: - .endif - + sub sp, sp, #PT_REGS_SIZE + .else /* \el == 1 */ + .ifc \ht, h sub sp, sp, #PT_REGS_SIZE - .if \el == 1 /* * Test whether the SP has overflowed, without corrupting a GPR. * Task and IRQ stacks are aligned so that SP & (1 << THREAD_SHIFT)
@@ -63,45 +63,16 @@ */ add sp, sp, x0 // sp' = sp + x0 sub x0, sp, x0 // x0' = sp' - x0 = (sp + x0) - x0 = sp - tbnz x0, #THREAD_SHIFT, 0f + tbnz x0, #THREAD_SHIFT, __bad_stack sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp + .else /* EL1t */ + msr spsel, #0 // Stay on the overflow stack + sub sp, sp, #PT_REGS_SIZE + .endif .endif b el\el\ht\()_\regsize\()_\label - - .if \el == 1 -0: - /* - * Either we've just detected an overflow, or we've taken an exception - * while on the overflow stack. Either way, we won't return to - * userspace, and can clobber EL0 registers to free up GPRs. - */ - - /* Stash the original SP (minus PT_REGS_SIZE) in tpidr_el0. */ - msr tpidr_el0, x0 - - /* Recover the original x0 value and stash it in sp_el0 */ - sub x0, sp, x0 - msr sp_el0, x0 - - /* Switch to the overflow stack */ - adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0 - - /* - * Check whether we were already on the overflow stack. This may happen - * after panic() re-enables interrupts. - */ - mrs x0, tpidr_el0 // sp of interrupted context - sub x0, sp, x0 // delta with top of overflow stack - tst x0, #~(OVERFLOW_STACK_SIZE - 1) // within range? - b.ne __bad_stack // no? -> bad stack pointer - - /* We were already on the overflow stack. Restore sp/x0 and carry on. */ - sub sp, sp, x0 - mrs x0, sp_el0 - b el\el\ht\()_\regsize\()_\label - .endif .org .Lventry_start\@ + 128 // Did we overflow the ventry slot? .endm
@@ -544,22 +515,24 @@ SYM_CODE_END(vectors) SYM_CODE_START_LOCAL(__bad_stack) /* - * We detected an overflow in kernel_ventry, which switched to the - * overflow stack. Stash the exception regs, and head to our overflow - * handler. + * We detected an overflow in kernel_ventry. + * Restore SP and X0. */ + sub x0, sp, x0 + sub sp, sp, x0 + add sp, sp, #PT_REGS_SIZE - /* Restore the original x0 value */ - mrs x0, sp_el0 + /* Switch to the overflow stack */ + msr spsel, #0 - /* - * Store the original GPRs to the new stack. The orginal SP (minus - * PT_REGS_SIZE) was stashed in tpidr_el0 by kernel_ventry. - */ + /* Stash the exception regs */ sub sp, sp, #PT_REGS_SIZE kernel_entry 1 - mrs x0, tpidr_el0 - add x0, x0, #PT_REGS_SIZE + + /* Fix-up the saved SP */ + msr spsel, #1 + mov x0, sp + msr spsel, #0 str x0, [sp, #S_SP] /* Stash the regs for handle_bad_stack */
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 18/21 arm64: entry: Split up kernel_ventry macro into separate helper macros
In reply to Will DeaconHandling entry from EL0, EL1h and EL1t in the 'kernel_ventry' assembly macro is getting a bit unwieldy as the control flow becomes increasingly dependent on the macro arguments.
Split 'kernel_ventry' into three leaf helper macros for EL0, EL1h and EL1t, which are then dispatched by simple wrappers.
--- arch/arm64/kernel/entry.S | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-)
arch/arm64/kernel/entry.S +19 -10
@@ -36,10 +36,7 @@ .endr .endm - .macro kernel_ventry, el:req, ht:req, regsize:req, label:req - .align 7 -.Lventry_start\@: - .if \el == 0 + .macro kernel_ventry_el0, ht:req, regsize:req, label:req /* * This must be the first instruction of the EL0 vector entries. It is * skipped by the trampoline vectors, to trigger the cleanup.
@@ -52,8 +49,10 @@ .endif .Lskip_tramp_vectors_cleanup\@: sub sp, sp, #PT_REGS_SIZE - .else /* \el == 1 */ - .ifc \ht, h + b el0\ht\()_\regsize\()_\label + .endm + + .macro kernel_ventry_el1h, regsize:req, label:req sub sp, sp, #PT_REGS_SIZE /*
@@ -66,13 +65,23 @@ tbnz x0, #THREAD_SHIFT, __bad_stack sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp - .else /* EL1t */ + b el1h_\regsize\()_\label + .endm + + .macro kernel_ventry_el1t, regsize:req, label:req msr spsel, #0 // Stay on the overflow stack sub sp, sp, #PT_REGS_SIZE - .endif - .endif + b el1t_\regsize\()_\label + .endm - b el\el\ht\()_\regsize\()_\label + .macro kernel_ventry, el:req, ht:req, regsize:req, label:req + .align 7 +.Lventry_start\@: + .if \el == 0 + kernel_ventry_el0 \ht, \regsize, \label + .else + kernel_ventry_el1\ht \regsize, \label + .endif .org .Lventry_start\@ + 128 // Did we overflow the ventry slot? .endm
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 19/21 arm64: entry: The great stack switcheroo
In reply to Will DeaconWith the kernel stack pointer in SP_EL1 and the overflow stack pointer in SP_EL0, it is now straightforward to switch between the two on exception entry from EL1 by writing to SPSel. However, since exception entry sets PSTATE.SP to 1 (selecting SP_EL1 as the stack pointer), repurposing the overflow stack as a more general kernel exception stack would require writing to SPSel on every exception entry from the kernel.
Switch things around so that the overflow stack resides in SP_EL1, with the kernel stack residing in SP_EL0.
--- arch/arm64/include/asm/kvm_asm.h | 16 +++++++++++++++ arch/arm64/kernel/entry-common.c | 34 ++++++++++++++++---------------- arch/arm64/kernel/entry.S | 26 +++++++++++++----------- arch/arm64/kernel/head.S | 5 ++++- arch/arm64/kernel/smp.c | 8 +++++--- arch/arm64/kvm/hyp/entry.S | 3 +++ arch/arm64/mm/proc.S | 3 +++ 7 files changed, 63 insertions(+), 32 deletions(-)
arch/arm64/include/asm/kvm_asm.h +16 -0
@@ -398,6 +398,22 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt, ldp x29, lr, [\ctxt, #CPU_XREG_OFFSET(29)] .endm +#ifdef __KVM_VHE_HYPERVISOR__ +.macro activate_exception_stack +msr spsel, #1 +.endm + +.macro deactivate_exception_stack +msr spsel, #0 +.endm +#else +.macro activate_exception_stack +.endm + +.macro deactivate_exception_stack +.endm +#endif + .macro save_sp_el0 ctxt, tmp mrs \tmp, sp_el0 str \tmp, [\ctxt, #CPU_SP_EL0_OFFSET]
arch/arm64/kernel/entry-common.c +17 -17
@@ -327,7 +327,7 @@ static void debug_exception_exit(struct pt_regs *regs) }NOKPROBE_SYMBOL (debug_exception_exit); -static void noinstr el1t_64_check_overflow_stack(struct pt_regs *regs) +static void noinstr el1h_64_check_overflow_stack(struct pt_regs *regs) {unsigned long sp =kernel_stack_pointer (regs) -sizeof (*regs);unsigned long ovf_stack = (unsigned long )this_cpu_ptr (overflow_stack);
@@ -343,28 +343,28 @@ static void noinstr el1t_64_check_overflow_stack(struct pt_regs *regs)cpu_park_loop (); } -asmlinkage void noinstr el1t_64_sync_handler(struct pt_regs *regs) +asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs) { -el1t_64_check_overflow_stack (regs); -el1h_64_sync_handler (regs); +el1h_64_check_overflow_stack (regs); +el1t_64_sync_handler (regs); } -asmlinkage void noinstr el1t_64_irq_handler(struct pt_regs *regs) +asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs) { -el1t_64_check_overflow_stack (regs); -el1h_64_irq_handler (regs); +el1h_64_check_overflow_stack (regs); +el1t_64_irq_handler (regs); } -asmlinkage void noinstr el1t_64_fiq_handler(struct pt_regs *regs) +asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs) { -el1t_64_check_overflow_stack (regs); -el1h_64_fiq_handler (regs); +el1h_64_check_overflow_stack (regs); +el1t_64_fiq_handler (regs); } -asmlinkage void noinstr el1t_64_error_handler(struct pt_regs *regs) +asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs) { -el1t_64_check_overflow_stack (regs); -el1h_64_error_handler (regs); +el1h_64_check_overflow_stack (regs); +el1t_64_error_handler (regs); } static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
@@ -494,7 +494,7 @@ static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr)arm64_exit_to_kernel_mode (regs, state); } -asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs) +asmlinkage void noinstr el1t_64_sync_handler(struct pt_regs *regs) {unsigned long esr =read_sysreg (esr_el1);
@@ -578,17 +578,17 @@ static void noinstr el1_interrupt(struct pt_regs *regs,__el1_irq (regs, handler); } -asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs) +asmlinkage void noinstr el1t_64_irq_handler(struct pt_regs *regs) {el1_interrupt (regs, handle_arch_irq); } -asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs) +asmlinkage void noinstr el1t_64_fiq_handler(struct pt_regs *regs) {el1_interrupt (regs, handle_arch_fiq); } -asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs) +asmlinkage void noinstr el1t_64_error_handler(struct pt_regs *regs) {unsigned long esr =read_sysreg (esr_el1);irqentry_state_t state;
arch/arm64/kernel/entry.S +15 -11
@@ -54,6 +54,12 @@ .macro kernel_ventry_el1h, regsize:req, label:req sub sp, sp, #PT_REGS_SIZE + b el1h_\regsize\()_\label + .endm + + .macro kernel_ventry_el1t, regsize:req, label:req + msr spsel, #0 // Stay on the kernel stack + sub sp, sp, #PT_REGS_SIZE /* * Test whether the SP has overflowed, without corrupting a GPR.
@@ -65,12 +71,6 @@ tbnz x0, #THREAD_SHIFT, __bad_stack sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp - b el1h_\regsize\()_\label - .endm - - .macro kernel_ventry_el1t, regsize:req, label:req - msr spsel, #0 // Stay on the overflow stack - sub sp, sp, #PT_REGS_SIZE b el1t_\regsize\()_\label .endm
@@ -208,7 +208,10 @@ alternative_cb_end ldr_this_cpu tsk, __entry_task, x20 msr tpidrro_el0, tsk adr_this_cpu x19, overflow_stack + OVERFLOW_STACK_SIZE, x20 - msr sp_el0, x19 + mov x20, sp + msr sp_el0, x20 + mov sp, x19 + msr spsel, #0 /* * Ensure MDSCR_EL1.SS is clear, since we can unmask debug exceptions
@@ -346,6 +349,9 @@ alternative_else_nop_endif .if \el == 0 ldr x23, [sp, #S_SP] // load return stack pointer + mov x0, sp + msr spsel, #1 + mov sp, x0 msr sp_el0, x23 msr tpidrro_el0, xzr tst x22, #PSR_MODE32_BIT // native task?
@@ -532,16 +538,14 @@ SYM_CODE_START_LOCAL(__bad_stack) add sp, sp, #PT_REGS_SIZE /* Switch to the overflow stack */ - msr spsel, #0 + msr spsel, #1 /* Stash the exception regs */ sub sp, sp, #PT_REGS_SIZE kernel_entry 1 /* Fix-up the saved SP */ - msr spsel, #1 - mov x0, sp - msr spsel, #0 + mrs x0, sp_el0 str x0, [sp, #S_SP] /* Stash the regs for handle_bad_stack */
arch/arm64/kernel/head.S +4 -1
@@ -211,8 +211,11 @@ SYM_CODE_END(preserve_boot_args) ldr \tmp1, [\tmp1, \tmp2, lsl #3] set_this_cpu_offset \tmp1 - adr_this_cpu \tmp1, overflow_stack + OVERFLOW_STACK_SIZE, \tmp2 + mov \tmp1, sp msr sp_el0, \tmp1 + adr_this_cpu \tmp1, overflow_stack + OVERFLOW_STACK_SIZE, \tmp2 + mov sp, \tmp1 + msr spsel, #0 .endm /*
arch/arm64/kernel/smp.c +5 -3
@@ -447,14 +447,16 @@ void __init smp_cpus_done(unsigned int max_cpus)mark_linear_text_alias_ro (); } -static void __init set_boot_cpu_offset (void ) +static void __init update_boot_cpu_offset_and_overflow_stack (void ) {u64 ovf_sp = (u64 )raw_cpu_ptr (overflow_stack) +OVERFLOW_STACK_SIZE ; asmvolatile (" msr tpidr_el1, %1\n" " add %0, %0, %1\n" -" msr sp_el0, %0" /* Update the overflow stack pointer */ +" msr spsel, #1\n" +" mov sp, %0\n" /* Update the overflow stack pointer */ +" msr spsel, #0" :"+r" (ovf_sp) :"r" (per_cpu_offset (0 )) :"memory" );
@@ -467,7 +469,7 @@ void __init smp_prepare_boot_cpu(void)* setup_per_cpu_areas(), and CPU0's boot time per-cpu area will be * freed shortly, so we must move over to the runtime per-cpu area. */ -set_boot_cpu_offset (); +update_boot_cpu_offset_and_overflow_stack ();cpuinfo_store_boot_cpu ();setup_boot_cpu_features ();
arch/arm64/kvm/hyp/entry.S +3 -0
@@ -31,6 +31,7 @@ SYM_FUNC_START(__guest_enter) save_callee_saved_regs x1 // Save hyp's sp_el0 and tpidrro_el0 + activate_exception_stack save_sp_el0 x1, x2 save_tpidrro_el0 x1, x2
@@ -50,6 +51,7 @@ alternative_else_nop_endif // that would usually be synchonized by the ERET. isb mov x0, #ARM_EXCEPTION_IRQ + deactivate_exception_stack ret 1:
@@ -167,6 +169,7 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL) // Restore hyp's sp_el0 and tpidrro_el0 restore_sp_el0 x2, x3 restore_tpidrro_el0 x2, x3 + deactivate_exception_stack // Now restore the hyp regs restore_callee_saved_regs x2
arch/arm64/mm/proc.S +3 -0
@@ -98,6 +98,7 @@ SYM_FUNC_START(cpu_do_suspend) mrs x10, oslsr_el1 mrs x11, sctlr_el1 get_this_cpu_offset x12 + msr spsel, #1 mrs x13, sp_el0 stp x2, x3, [x0] stp x4, x5, [x0, #16]
@@ -156,6 +157,8 @@ alternative_else_nop_endif msr sctlr_el1, x12 set_this_cpu_offset x13 msr sp_el0, x14 + msr spsel, #0 + /* * Restore oslsr_el1 by writing oslar_el1 */
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerRe: [PATCH 19/21] arm64: entry: The great stack switcheroo
Quoted 20 lines · Will Deacon
With the kernel stack pointer in
SP_EL1and the overflow stack pointer inSP_EL0, it is now straightforward to switch between the two on exception entry from EL1 by writing to SPSel. However, since exception entry sets PSTATE.SP to 1 (selectingSP_EL1as the stack pointer), repurposing the overflow stack as a more general kernel exception stack would require writing to SPSel on every exception entry from the kernel.Switch things around so that the overflow stack resides in
SP_EL1, with the kernel stack residing inSP_EL0.Signed-off-by Will Deacon <[email protected]>--- arch/arm64/include/asm/kvm_asm.h | 16 +++++++++++++++ arch/arm64/kernel/entry-common.c | 34 ++++++++++++++++---------------- arch/arm64/kernel/entry.S | 26 +++++++++++++----------- arch/arm64/kernel/head.S | 5 ++++- arch/arm64/kernel/smp.c | 8 +++++--- arch/arm64/kvm/hyp/entry.S | 3 +++ arch/arm64/mm/proc.S | 3 +++ 7 files changed, 63 insertions(+), 32 deletions(-)
[...]
Quoted 13 lines
arch/arm64/mm/proc.S +1
-0@@ -98,6 +98,7 @@ SYM_FUNC_START(cpu_do_suspend) mrs x10, oslsr_el1 mrs x11, sctlr_el1 get_this_cpu_offset x12 + msr spsel, #1 mrs x13, sp_el0 stp x2, x3, [x0] stp x4, x5, [x0, #16]
arch/arm64/mm/proc.S alternative_else_nop_endif · line 160
Quoted 4 lines
arch/arm64/mm/proc.S +1
-0@@ -156,6 +157,8 @@ alternative_else_nop_endif msr sctlr_el1, x12 set_this_cpu_offset x13 msr sp_el0, x14 + msr spsel, #0
Sashiko rightfully points out that I've got this slightly wrong because I've pushed the spsel toggle too deep in the callchain for the suspend/resume paths. My original rationale was that the current code already saves/restores SP_EL{0,1}, so we shouldn't need to change anything. However, that's not quite right because it's the caller (e.g. __cpu_suspend_enter()) which saves SP_EL1 as 'sp', so the stack toggle should happen there.
I'll fix that in v2.
Will
Will Deacon
maintainerPATCH 20/21 arm64: tracing: Advertise a mode of EL1t in synthetic kernel regs
In reply to Will DeaconWhen synthesising a kernel regs structure for ftrace or perf, advertise a mode of EL1t to reflect the mode in which the kernel runs for the vast majority of the time.
--- arch/arm64/include/asm/ftrace.h | 4 ++-- arch/arm64/include/asm/perf_event.h | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
arch/arm64/include/asm/ftrace.h +2 -2
@@ -153,7 +153,7 @@ ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs) regs->pc = afregs->pc; regs->regs[29 ] = afregs->fp; regs->regs[30 ] = afregs->lr; - regs->pstate = PSR_MODE_EL1h; + regs->pstate = PSR_MODE_EL1t;return regs; }
@@ -161,7 +161,7 @@ ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs) (_regs)->pc =arch_ftrace_regs (fregs)->pc; \ (_regs)->regs[29 ] =arch_ftrace_regs (fregs)->fp; \ (_regs)->sp =arch_ftrace_regs (fregs)->sp; \ - (_regs)->pstate = PSR_MODE_EL1h; \ + (_regs)->pstate = PSR_MODE_EL1t; \ }while (0 ) int ftrace_regs_query_register_offset(const char *name);
arch/arm64/include/asm/perf_event.h +1 -1
@@ -17,7 +17,7 @@ (regs)->pc = (__ip); \ (regs)->regs[29 ] = (unsigned long )__builtin_frame_address (0 ); \ (regs)->sp = current_stack_pointer; \ - (regs)->pstate = PSR_MODE_EL1h; \ + (regs)->pstate = PSR_MODE_EL1t; \ }#endif
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Will Deacon
maintainerPATCH 21/21 arm64: Rename 'overflow_stack' and OVERFLOW_STACK_SIZE
In reply to Will DeaconNow that the kernel "overflow" stack resides in SP_EL1, it is the active stack for any exception taken from the kernel. Even though we currently switch immediately to the kernel stack in the EL1t entry code, rename the "overflow_stack" to "kernel_exception_stack" to better reflect its usage. Introduce NVHE_OVERFLOW_STACK_SIZE to keep the EL2 overflow stack size distinct from the EL1 kernel exception stack size.
--- arch/arm64/include/asm/memory.h | 10 ++++++---- arch/arm64/include/asm/stacktrace.h | 9 +++++---- arch/arm64/include/asm/stacktrace/nvhe.h | 3 +-- arch/arm64/kernel/entry-common.c | 18 +++++++++--------- arch/arm64/kernel/entry.S | 2 +- arch/arm64/kernel/head.S | 2 +- arch/arm64/kernel/smp.c | 9 +++++---- arch/arm64/kernel/stacktrace.c | 2 +- arch/arm64/kernel/traps.c | 14 +++++++------- arch/arm64/kvm/hyp/nvhe/host.S | 2 +- arch/arm64/kvm/hyp/nvhe/stacktrace.c | 4 ++-- arch/arm64/kvm/stacktrace.c | 4 ++-- 12 files changed, 41 insertions(+), 38 deletions(-)
arch/arm64/include/asm/memory.h +6 -4
@@ -139,10 +139,11 @@#define IRQ_STACK_SIZE THREAD_SIZE -#define OVERFLOW_STACK_SIZE SZ_4K +#define KERNEL_EXC_STACK_SIZE SZ_4K -#define NVHE_STACK_SHIFT PAGE_SHIFT -#define NVHE_STACK_SIZE (UL(1) << NVHE_STACK_SHIFT) +#define NVHE_OVERFLOW_STACK_SIZE SZ_4K +#define NVHE_STACK_SHIFT PAGE_SHIFT +#define NVHE_STACK_SIZE (UL(1) << NVHE_STACK_SHIFT)/* * With the minimum frame size of [x29, x30], exactly half the combined
@@ -150,7 +151,8 @@* save the unwinded stacktrace; plus an additional entry to delimit the * end. */ -#define NVHE_STACKTRACE_SIZE ((OVERFLOW_STACK_SIZE + NVHE_STACK_SIZE) / 2 + sizeof(long)) +#define NVHE_STACKTRACE_SIZE ((NVHE_OVERFLOW_STACK_SIZE + NVHE_STACK_SIZE) \ + / 2 + sizeof(long))/* * Alignment of kernel segments (e.g. .text, .data).
arch/arm64/include/asm/stacktrace.h +5 -4
@@ -59,12 +59,13 @@ static inline bool on_task_stack(const struct task_struct *tsk,#define on_thread_stack () (on_task_stack(current, current_stack_pointer, 1)) -DECLARE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack); +DECLARE_PER_CPU(unsigned long [KERNEL_EXC_STACK_SIZE/sizeof(long)], + kernel_exception_stack); -static inline struct stack_info stackinfo_get_overflow (void ) +static inline struct stack_info stackinfo_get_kernel_exception (void ) { -unsigned long low = (unsigned long )raw_cpu_ptr (overflow_stack); -unsigned long high = low +OVERFLOW_STACK_SIZE ; +unsigned long low = (unsigned long )raw_cpu_ptr (kernel_exception_stack); +unsigned long high = low +KERNEL_EXC_STACK_SIZE ;return (struct stack_info ) { .low = low,
arch/arm64/include/asm/stacktrace/nvhe.h +1 -2
@@ -44,8 +44,7 @@ static inline void kvm_nvhe_unwind_init(struct unwind_state *state,* In non-protected mode, the unwinding is done from kernel proper context * (by the host in EL1). */ - -DECLARE_KVM_NVHE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack); +DECLARE_KVM_NVHE_PER_CPU(unsigned long [NVHE_OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack); DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_stacktrace_info, kvm_stacktrace_info); DECLARE_PER_CPU(unsigned long, kvm_arm_hyp_stack_base);
arch/arm64/kernel/entry-common.c +9 -9
@@ -327,43 +327,43 @@ static void debug_exception_exit(struct pt_regs *regs) }NOKPROBE_SYMBOL (debug_exception_exit); -static void noinstr el1h_64_check_overflow_stack(struct pt_regs *regs) +static void noinstr el1h_64_check_exception_stack(struct pt_regs *regs) {unsigned long sp =kernel_stack_pointer (regs) -sizeof (*regs); -unsigned long ovf_stack = (unsigned long )this_cpu_ptr (overflow_stack); +unsigned long exc_stack = (unsigned long )this_cpu_ptr (kernel_exception_stack);/* -* We're in big trouble if we've overflowed the overflow stack +* We're in big trouble if we've overflowed the exception stack * so perform a best-effort check before we proceed. If our SP -* is outside of the overflow stack for this CPU then presumably +* is outside of the exceptoin stack for this CPU then presumably * we're already corrupting memory, so park ourselves here in an * attempt to contain the damage. */ -if (sp < ovf_stack || sp > ovf_stack +OVERFLOW_STACK_SIZE ) +if (sp < exc_stack || sp > exc_stack +KERNEL_EXC_STACK_SIZE )cpu_park_loop (); } asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs) { -el1h_64_check_overflow_stack (regs); +el1h_64_check_exception_stack (regs);el1t_64_sync_handler (regs); } asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs) { -el1h_64_check_overflow_stack (regs); +el1h_64_check_exception_stack (regs);el1t_64_irq_handler (regs); } asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs) { -el1h_64_check_overflow_stack (regs); +el1h_64_check_exception_stack (regs);el1t_64_fiq_handler (regs); } asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs) { -el1h_64_check_overflow_stack (regs); +el1h_64_check_exception_stack (regs);el1t_64_error_handler (regs); }
arch/arm64/kernel/entry.S +1 -1
@@ -207,7 +207,7 @@ alternative_cb_end mrs x21, sp_el0 ldr_this_cpu tsk, __entry_task, x20 msr tpidrro_el0, tsk - adr_this_cpu x19, overflow_stack + OVERFLOW_STACK_SIZE, x20 + adr_this_cpu x19, kernel_exception_stack + KERNEL_EXC_STACK_SIZE, x20 mov x20, sp msr sp_el0, x20 mov sp, x19
arch/arm64/kernel/head.S +1 -1
@@ -213,7 +213,7 @@ SYM_CODE_END(preserve_boot_args) mov \tmp1, sp msr sp_el0, \tmp1 - adr_this_cpu \tmp1, overflow_stack + OVERFLOW_STACK_SIZE, \tmp2 + adr_this_cpu \tmp1, kernel_exception_stack + KERNEL_EXC_STACK_SIZE, \tmp2 mov sp, \tmp1 msr spsel, #0 .endm
arch/arm64/kernel/smp.c +5 -4
@@ -447,9 +447,10 @@ void __init smp_cpus_done(unsigned int max_cpus)mark_linear_text_alias_ro (); } -static void __init update_boot_cpu_offset_and_overflow_stack (void ) +static void __init update_boot_cpu_offset_and_exception_stack (void ) { -u64 ovf_sp = (u64 )raw_cpu_ptr (overflow_stack) +OVERFLOW_STACK_SIZE ; +u64 exc_sp = (u64 )raw_cpu_ptr (kernel_exception_stack) + +KERNEL_EXC_STACK_SIZE ; asmvolatile ( " msr tpidr_el1, %1\n"
@@ -457,7 +458,7 @@ static void __init update_boot_cpu_offset_and_overflow_stack(void)" msr spsel, #1\n" " mov sp, %0\n" /* Update the overflow stack pointer */ " msr spsel, #0" - : "+r" (ovf_sp) + : "+r" (exc_sp) : "r" (per_cpu_offset(0)) : "memory"); }
@@ -469,7 +470,7 @@ void __init smp_prepare_boot_cpu(void)* setup_per_cpu_areas(), and CPU0's boot time per-cpu area will be * freed shortly, so we must move over to the runtime per-cpu area. */ -update_boot_cpu_offset_and_overflow_stack (); +update_boot_cpu_offset_and_exception_stack ();cpuinfo_store_boot_cpu ();setup_boot_cpu_features ();
arch/arm64/kernel/stacktrace.c +1 -1
@@ -337,7 +337,7 @@ kunwind_stack_walk(kunwind_consume_fn consume_state,struct stack_info stacks[] = {stackinfo_get_task (task),STACKINFO_CPU (irq), -STACKINFO_CPU (overflow), +STACKINFO_CPU (kernel_exception), #if defined(CONFIG_ARM_SDE_INTERFACE)STACKINFO_SDEI (normal), STACKINFO_SDEI(critical),
arch/arm64/kernel/traps.c +7 -7
@@ -897,14 +897,14 @@ void bad_el0_sync(struct pt_regs *regs, int reason, unsigned long esr) "Bad EL0 synchronous exception"); } -DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack) -__aligned(16) ; +DEFINE_PER_CPU(unsigned long [KERNEL_EXC_STACK_SIZE/sizeof(long)], + kernel_exception_stack)__aligned(16) ;void __noreturn panic_bad_stack (struct pt_regs *regs,unsigned long esr,unsigned long far) {unsigned long tsk_stk = (unsigned long )current->stack;unsigned long irq_stk = (unsigned long )this_cpu_read (irq_stack_ptr); -unsigned long ovf_stk = (unsigned long )this_cpu_ptr (overflow_stack); +unsigned long exc_stk = (unsigned long )this_cpu_ptr (kernel_exception_stack);console_verbose ();pr_emerg ("Insufficient stack space to handle exception!" );
@@ -912,12 +912,12 @@ void __noreturn panic_bad_stack(struct pt_regs *regs, unsigned long esr, unsignepr_emerg ("ESR: 0x%016lx -- %s\n" , esr,esr_get_class_string (esr));pr_emerg ("FAR: 0x%016lx\n" , far); -pr_emerg ("Task stack: [0x%016lx..0x%016lx]\n" , +pr_emerg ("Task stack: [0x%016lx..0x%016lx]\n" , tsk_stk, tsk_stk +THREAD_SIZE ); -pr_emerg ("IRQ stack: [0x%016lx..0x%016lx]\n" , +pr_emerg ("IRQ stack: [0x%016lx..0x%016lx]\n" , irq_stk, irq_stk +IRQ_STACK_SIZE ); -pr_emerg ("Overflow stack: [0x%016lx..0x%016lx]\n" , - ovf_stk, ovf_stk +OVERFLOW_STACK_SIZE ); +pr_emerg ("Exception stack: [0x%016lx..0x%016lx]\n" , + exc_stk, exc_stk +KERNEL_EXC_STACK_SIZE );__show_regs (regs);
arch/arm64/kvm/hyp/nvhe/host.S +1 -1
@@ -205,7 +205,7 @@ SYM_FUNC_END(__host_hvc) .L__hyp_sp_overflow\@: /* Switch to the overflow stack */ - adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0 + adr_this_cpu sp, overflow_stack + NVHE_OVERFLOW_STACK_SIZE, x0 b hyp_panic_bad_stack ASM_BUG()
arch/arm64/kvm/hyp/nvhe/stacktrace.c +2 -2
@@ -9,7 +9,7 @@#include <asm/memory.h> #include <asm/percpu.h> -DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack) +DEFINE_PER_CPU(unsigned long [NVHE_OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)__aligned(16) ; DEFINE_PER_CPU(struct kvm_nvhe_stacktrace_info, kvm_stacktrace_info);
@@ -42,7 +42,7 @@ DEFINE_PER_CPU(unsigned long [NVHE_STACKTRACE_SIZE/sizeof(long)], pkvm_stacktracstatic struct stack_info stackinfo_get_overflow (void ) {unsigned long low = (unsigned long )this_cpu_ptr (overflow_stack); -unsigned long high = low +OVERFLOW_STACK_SIZE ; +unsigned long high = low +NVHE_OVERFLOW_STACK_SIZE ;return (struct stack_info ) { .low = low,
arch/arm64/kvm/stacktrace.c +2 -2
@@ -27,7 +27,7 @@ static struct stack_info stackinfo_get_overflow(void)struct kvm_nvhe_stacktrace_info *stacktrace_info =this_cpu_ptr_nvhe_sym (kvm_stacktrace_info);unsigned long low = (unsigned long )stacktrace_info->overflow_stack_base; -unsigned long high = low +OVERFLOW_STACK_SIZE ; +unsigned long high = low +NVHE_OVERFLOW_STACK_SIZE ;return (struct stack_info ) { .low = low,
@@ -38,7 +38,7 @@ static struct stack_info stackinfo_get_overflow(void)static struct stack_info stackinfo_get_overflow_kern_va (void ) {unsigned long low = (unsigned long )this_cpu_ptr_nvhe_sym (overflow_stack); -unsigned long high = low +OVERFLOW_STACK_SIZE ; +unsigned long high = low +NVHE_OVERFLOW_STACK_SIZE ;return (struct stack_info ) { .low = low,
Signature · 1 line
2.55.0.979.g7e5102b832-goog
Hi Will,
Quoted 38 lines · Will Deacon
Hi everyone,
This series is a bit of a complicated juggling act that, on its own, doesn't achieve an awful lot. However, it lays the ground work for sizing the kernel stack at runtime, e.g. via a cmdline option or even potentially on a per-task basis and so I would like to work towards getting it merged independently.
The series is based on v7.3-rc1 and structured as follows:
* The first 9 patches move the 'current' task pointer from
SP_EL0toTPIDRRO_EL0.* The following 8 patches point the newly-freed
SP_EL0at the overflow stack and switch to it explicitly when we detect a kernel stack overflow.* The final 4 patches turn everything on its head, so that the overflow stack and kernel stack are swapped, with the former now residing in
SP_EL1and the latter inSP_EL0.At the end of all that, when we take an exception from EL1, we are immediately transitioned to the overflow stack (now renamed "exception stack") and can push registers right away. This also means that using SPINTMASK to control NMI masking becomes a possibility, although speaking to Mark, Vladimir and Ada, they all seem to prefer ALLINT.
Mostafa will soon post a follow-up series that allows the kernel stack size to be specified on the kernel cmdline, which we are hoping to use to configure an 8k stack size in Android. I will be talking more about all of this at LPC in the Memory Management MC:
https://lpc.events/event/20/contributions/2419/
Cheers,
Will
Quoted 13 lines
Cc Arnd Bergmann <[email protected]>Cc Ard Biesheuvel <[email protected]>Cc Ada Couprie Diaz <[email protected]>Cc David Hildenbrand <[email protected]>Cc Catalin Marinas <[email protected]>Cc Vladimir Murzin <[email protected]>Cc Mark Rutland <[email protected]>Cc Mostafa Saleh <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc Oliver Upton <[email protected]>Cc Linus Walleij <[email protected]>Cc Marc Zyngier <[email protected]>
I gave it a try and I observe splat:
Unable to handle kernel execute from non-executable memory at virtual address ffff000970e81148
Mem abort info:
ESR = 0x000000008600000f
EC = 0x21: IABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
FSC = 0x0f: level 3 permission fault
swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000008129f000
[ffff000970e81148] pgd=0000000000000000, p4d=18000009f1dff403, pud=18000009f1079403, pmd=18000009f0ef1403, pte=00e80009f0e81707
Internal error: Oops: 000000008600000f [#1] SMP Modules linked in: CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.3.0-rc2-7fb054f87+ #3627 PREEMPT(lazy) Hardware name: Generated (DT) pstate: 1634023c9 (nZCv DAIF +ALLINT +PAN -UAO +TCO +DIT -SSBS BTYPE=--) pc : 0xffff000970e81148 lr : 0xffff000970e81148
sp : ffff000970e81150
x29: ffff800080f85fc0 x28: 0000000000000001 x27: 0000000000002000
x26: 0000000000000000 x25: 00000000000000c0 x24: 0000000040000023
x23: ffff800080c0e1d8 x22: 00000000000003c0 x21: ffff800080b58300
x20: cfa2800080b58300 x19: 00000000200003c0 x18: 0000000000000000
x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000028
x14: 0000000000000000 x13: ffff8000814f3ac0 x12: ffff8008efcac000
x11: b2000c3eb474d99d x10: 0000000000000008 x9 : 0000000000001000 x8 : ffff800080010800 x7 : 055001f2b5503510 x6 : 0000000001310000 x5 : 0000000000000000 x4 : 0000000000000000 x3 : ffff800081502d80 x2 : 0000000000000802 x1 : ffff000800106a70 x0 : 0000000000000001
Call trace: 0xffff000970e81148 (P) Code: 00000000 00000000 00000000 00000000 (00000002) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Oops: Fatal exception
SMP: stopping secondary CPUs
Kernel Offset: disabled
CPU features: 0x0,00000000,034bfc7d,ff728b42,7ffce667 Memory Limit: none
---[ end Kernel panic - not syncing: Oops: Fatal exception ]---
I suspect it is related to power management, since it can be triggered with the sleep command, though I haven't debugged it. I noticed that Sashiko has reported issues related to suspend/resume, so if you provide fixups for the relevant commits, I can give them another try. Otherwise, I'll wait for v2 :)
Cheers
Vladimir
Will Deacon
maintainerHi Vladimir,
Quoted 6 lines · Vladimir Murzin
Quoted 5 lines · Will Deacon
This series is a bit of a complicated juggling act that, on its own, doesn't achieve an awful lot. However, it lays the ground work for sizing the kernel stack at runtime, e.g. via a cmdline option or even potentially on a per-task basis and so I would like to work towards getting it merged independently.
[...]
Quoted 1 line
I gave it a try and I observe splat:
Thanks for taking it for a spin!
Quoted 36 lines
Unable to handle kernel execute from non-executable memory at virtual address ffff000970e81148Mem abort info:
ESR =0x000000008600000f
EC =0x21: IABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
FSC =0x0f: level 3 permission fault
swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000008129f000
[ffff000970e81148] pgd=0000000000000000, p4d=18000009f1dff403, pud=18000009f1079403, pmd=18000009f0ef1403, pte=00e80009f0e81707Internal error: Oops: 000000008600000f [#1] SMP Modules linked in: CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.3.0-rc2-7fb054f87+ #3627 PREEMPT(lazy) Hardware name: Generated (DT) pstate: 1634023c9 (nZCv DAIF +ALLINT +PAN -UAO +TCO +DIT -SSBS BTYPE=--) pc : 0xffff000970e81148 lr : 0xffff000970e81148sp :
ffff000970e81150
x29:ffff800080f85fc0x28: 0000000000000001 x27: 0000000000002000
x26: 0000000000000000 x25:00000000000000c0x24: 0000000040000023
x23:ffff800080c0e1d8x22:00000000000003c0x21:ffff800080b58300
x20:cfa2800080b58300x19:00000000200003c0x18: 0000000000000000
x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000028
x14: 0000000000000000 x13:ffff8000814f3ac0x12:ffff8008efcac000
x11:b2000c3eb474d99dx10: 0000000000000008 x9 : 0000000000001000 x8 :ffff800080010800x7 :055001f2b5503510x6 : 0000000001310000 x5 : 0000000000000000 x4 : 0000000000000000 x3 :ffff800081502d80x2 : 0000000000000802 x1 :ffff000800106a70x0 : 0000000000000001Call trace: 0xffff000970e81148 (P) Code: 00000000 00000000 00000000 00000000 (00000002) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Oops: Fatal exceptionSMP: stopping secondary CPUs
Kernel Offset: disabled
CPU features: 0x0,00000000,034bfc7d,ff728b42,7ffce667 Memory Limit: none
Quoted 7 lines
---[ end Kernel panic - not syncing: Oops: Fatal exception ]---I suspect it is related to power management, since it can be triggered with the sleep command, though I haven't debugged it. I noticed that Sashiko has reported issues related to suspend/resume, so if you provide fixups for the relevant commits, I can give them another try. Otherwise, I'll wait for v2 :)
It's fiddly to envisage how we end up trying to execute from non-executable memory, but there are two bugs in the suspend/resume code:
1. I don't save/restore the stack pointers correctly (I suppose this could explain almost any crash, tbh)
2. I don't restore the pauth keys properly
I've hacked up an untested diff below, please can you take it for a spin?
Cheers,
Will
--->8
arch/arm64/kernel/sleep.S +2 -1
@@ -133,7 +133,8 @@ SYM_FUNC_START(_cpu_resume) add x0, x0, #SLEEP_STACK_DATA_SYSTEM_REGS /* load sp from context */ ldr x2, [x0, #CPU_CTX_SP] - mov sp, x2 + msr sp_el0, x2 + /* * cpu_do_resume expects x0 to contain context address pointer */
arch/arm64/mm/proc.S +12 -5
@@ -87,6 +87,7 @@ * This must be kept in sync with struct cpu_suspend_ctx in <asm/suspend.h>. */ SYM_FUNC_START(cpu_do_suspend) + msr spsel, #1 mrs x2, tpidr_el0 mrs x3, tpidrro_el0 mrs x4, contextidr_el1
@@ -98,8 +99,7 @@ SYM_FUNC_START(cpu_do_suspend) mrs x10, oslsr_el1 mrs x11, sctlr_el1 get_this_cpu_offset x12 - msr spsel, #1 - mrs x13, sp_el0 + mov x13, sp // SP_EL1 stp x2, x3, [x0] stp x4, x5, [x0, #16] stp x6, x7, [x0, #32]
@@ -115,6 +115,7 @@ alternative_if ARM64_HAS_TCR2 mrs x2, REG_TCR2_EL1 str x2, [x0, #104] alternative_else_nop_endif + msr spsel, #0 ret SYM_FUNC_END(cpu_do_suspend)
@@ -122,6 +123,8 @@ SYM_FUNC_END(cpu_do_suspend) * cpu_do_resume - restore CPU register context * * x0: Address of context pointer + * + * Entered with SPSel == 1, returns with SPSel == 0. */ SYM_FUNC_START(cpu_do_resume) ldp x2, x3, [x0]
@@ -130,6 +133,10 @@ SYM_FUNC_START(cpu_do_resume) ldp x9, x10, [x0, #48] ldp x11, x12, [x0, #64] ldp x13, x14, [x0, #80] + + /* Move 'current' somewhere safe */ + mov x15, x3 + /* * Restore x18, as it may be used as a platform register, and clear * the buffer to minimize the risk of exposure when used for shadow
@@ -156,8 +163,7 @@ alternative_else_nop_endif msr sctlr_el1, x12 set_this_cpu_offset x13 - msr sp_el0, x14 - msr spsel, #0 + mov sp, x14 // SP_EL1 /* * Restore oslsr_el1 by writing oslar_el1
@@ -179,8 +185,9 @@ alternative_if ARM64_HAS_GIC_PRIO_MASKING alternative_else_nop_endif #endif - ptrauth_keys_install_kernel_nosync x14, x1, x2, x3 + ptrauth_keys_install_kernel_nosync x15, x1, x2, x3 isb + msr spsel, #0 ret SYM_FUNC_END(cpu_do_resume) #endif
Hi Will,
Quoted 60 lines · Will Deacon
Hi Vladimir,
Quoted 6 lines · Vladimir Murzin
Quoted 5 lines · Will Deacon
This series is a bit of a complicated juggling act that, on its own, doesn't achieve an awful lot. However, it lays the ground work for sizing the kernel stack at runtime, e.g. via a cmdline option or even potentially on a per-task basis and so I would like to work towards getting it merged independently.
[...]
Quoted 1 line
I gave it a try and I observe splat:
Thanks for taking it for a spin!
Quoted 36 lines
Unable to handle kernel execute from non-executable memory at virtual address ffff000970e81148Mem abort info:
ESR =0x000000008600000f
EC =0x21: IABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
FSC =0x0f: level 3 permission fault
swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000008129f000
[ffff000970e81148] pgd=0000000000000000, p4d=18000009f1dff403, pud=18000009f1079403, pmd=18000009f0ef1403, pte=00e80009f0e81707Internal error: Oops: 000000008600000f [#1] SMP Modules linked in: CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.3.0-rc2-7fb054f87+ #3627 PREEMPT(lazy) Hardware name: Generated (DT) pstate: 1634023c9 (nZCv DAIF +ALLINT +PAN -UAO +TCO +DIT -SSBS BTYPE=--) pc : 0xffff000970e81148 lr : 0xffff000970e81148sp :
ffff000970e81150
x29:ffff800080f85fc0x28: 0000000000000001 x27: 0000000000002000
x26: 0000000000000000 x25:00000000000000c0x24: 0000000040000023
x23:ffff800080c0e1d8x22:00000000000003c0x21:ffff800080b58300
x20:cfa2800080b58300x19:00000000200003c0x18: 0000000000000000
x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000028
x14: 0000000000000000 x13:ffff8000814f3ac0x12:ffff8008efcac000
x11:b2000c3eb474d99dx10: 0000000000000008 x9 : 0000000000001000 x8 :ffff800080010800x7 :055001f2b5503510x6 : 0000000001310000 x5 : 0000000000000000 x4 : 0000000000000000 x3 :ffff800081502d80x2 : 0000000000000802 x1 :ffff000800106a70x0 : 0000000000000001Call trace: 0xffff000970e81148 (P) Code: 00000000 00000000 00000000 00000000 (00000002) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Oops: Fatal exceptionSMP: stopping secondary CPUs
Kernel Offset: disabled
CPU features: 0x0,00000000,034bfc7d,ff728b42,7ffce667 Memory Limit: noneQuoted 7 lines
---[ end Kernel panic - not syncing: Oops: Fatal exception ]---I suspect it is related to power management, since it can be triggered with the sleep command, though I haven't debugged it. I noticed that Sashiko has reported issues related to suspend/resume, so if you provide fixups for the relevant commits, I can give them another try. Otherwise, I'll wait for v2 :)
It's fiddly to envisage how we end up trying to execute from non-executable memory, but there are two bugs in the suspend/resume code:
Quoted 7 lines
1. I don't save/restore the stack pointers correctly (I suppose this could explain almost any crash, tbh)
2. I don't restore the pauth keys properly
I've hacked up an untested diff below, please can you take it for a spin?
With fixup applied I do not see splat anymore :)
Thanks
Vladimir
Quoted 92 lines
Cheers,
Will
--->8
arch/arm64/kernel/sleep.S +2
-1@@ -133,7 +133,8 @@ SYM_FUNC_START(_cpu_resume) add x0, x0, #SLEEP_STACK_DATA_SYSTEM_REGS /* load sp from context */ ldr x2, [x0, #CPU_CTX_SP] - mov sp, x2 + msr sp_el0, x2 + /* * cpu_do_resume expects x0 to contain context address pointer */arch/arm64/mm/proc.S +12
-5@@ -87,6 +87,7 @@ * This must be kept in sync with struct cpu_suspend_ctx in <asm/suspend.h>. */ SYM_FUNC_START(cpu_do_suspend) + msr spsel, #1 mrs x2, tpidr_el0 mrs x3, tpidrro_el0 mrs x4, contextidr_el1@@ -98,8 +99,7 @@ SYM_FUNC_START(cpu_do_suspend) mrs x10, oslsr_el1 mrs x11, sctlr_el1 get_this_cpu_offset x12 - msr spsel, #1 - mrs x13, sp_el0 + mov x13, sp // SP_EL1 stp x2, x3, [x0] stp x4, x5, [x0, #16] stp x6, x7, [x0, #32]@@ -115,6 +115,7 @@ alternative_if ARM64_HAS_TCR2 mrs x2, REG_TCR2_EL1 str x2, [x0, #104] alternative_else_nop_endif + msr spsel, #0 ret SYM_FUNC_END(cpu_do_suspend)@@ -122,6 +123,8 @@ SYM_FUNC_END(cpu_do_suspend) * cpu_do_resume - restore CPU register context * * x0: Address of context pointer + * + * Entered with SPSel == 1, returns with SPSel == 0. */ SYM_FUNC_START(cpu_do_resume) ldp x2, x3, [x0]@@ -130,6 +133,10 @@ SYM_FUNC_START(cpu_do_resume) ldp x9, x10, [x0, #48] ldp x11, x12, [x0, #64] ldp x13, x14, [x0, #80] + + /* Move 'current' somewhere safe */ + mov x15, x3 + /* * Restore x18, as it may be used as a platform register, and clear * the buffer to minimize the risk of exposure when used for shadow@@ -156,8 +163,7 @@ alternative_else_nop_endif msr sctlr_el1, x12 set_this_cpu_offset x13 - msr sp_el0, x14 - msr spsel, #0 + mov sp, x14 // SP_EL1 /* * Restore oslsr_el1 by writing oslar_el1@@ -179,8 +185,9 @@ alternative_if ARM64_HAS_GIC_PRIO_MASKING alternative_else_nop_endif #endif - ptrauth_keys_install_kernel_nosync x14, x1, x2, x3 + ptrauth_keys_install_kernel_nosync x15, x1, x2, x3 isb + msr spsel, #0 ret SYM_FUNC_END(cpu_do_resume) #endif
Will Deacon
maintainerQuoted 10 lines · Vladimir Murzin
Quoted 7 lines · Will Deacon
On Wed, Sep 09, 2026 at 11:39:24AM +0100, Vladimir Murzin wrote:
[...]Quoted 1 line
I gave it a try and I observe splat:
I've hacked up an untested diff below, please can you take it for a spin?
With fixup applied I do not see splat anymore :)
Excellent, thanks for testing it!
Will