riscv: SCS support
cover riscv: SCS support
Hi folks,
This series adds Shadow Call Stack (SCS) support for RISC-V. SCS uses compiler instrumentation to store return addresses in a separate shadow stack to protect them against accidental or malicious overwrites. More information about SCS can be found here:
https://clang.llvm.org/docs/ShadowCallStack.html
Patch 1 is from Deepak, and it simplifies VMAP_STACK overflow handling by adding support for accessing per-CPU variables directly in assembly. The patch is included in this series to make IRQ stack switching cleaner with SCS, and I've simply rebased it. Patch 2 uses this functionality to clean up the stack switching by moving duplicate code into a single function. On RISC-V, the compiler uses the gp register for storing the current shadow call stack pointer, which is incompatible with global pointer relaxation. Patch 3 moves global pointer loading into a macro that can be easily disabled with SCS. Patch 4 implements SCS register loading and switching, and allows the feature to be enabled, and patch 5 adds separate per-CPU IRQ shadow call stacks when CONFIG_IRQ_STACKS is enabled.
Note that this series requires Clang 17. Earlier Clang versions support SCS on RISC-V, but use the x18 register instead of gp, which isn't ideal. gcc has SCS support for arm64, but I'm not aware of plans to support RISC-V. Once the Zicfiss extension is ratified, it's probably preferable to use hardware-backed shadow stacks instead of SCS on hardware that supports the extension, and we may want to consider implementing CONFIG_DYNAMIC_SCS to patch between the implementation at runtime (similarly to the arm64 implementation, which switches to SCS when hardware PAC support isn't available).
Sami
Deepak Gupta (1):
riscv: VMAP_STACK overflow detection thread-safe
Sami Tolvanen (4):
riscv: Deduplicate IRQ stack switching
riscv: Move global pointer loading to a macro
riscv: Implement Shadow Call Stack
riscv: Use separate IRQ shadow call stacks
arch/riscv/Kconfig | 6 ++ arch/riscv/Makefile | 4 + arch/riscv/include/asm/asm.h | 35 ++++++++ arch/riscv/include/asm/irq_stack.h | 3 + arch/riscv/include/asm/scs.h | 54 ++++++++++++ arch/riscv/include/asm/thread_info.h | 16 +++- arch/riscv/kernel/asm-offsets.c | 4 + arch/riscv/kernel/entry.S | 126 +++++++++++++-------------- arch/riscv/kernel/head.S | 19 ++-- arch/riscv/kernel/irq.c | 53 ++++++----- arch/riscv/kernel/suspend_entry.S | 5 +- arch/riscv/kernel/traps.c | 65 ++------------ arch/riscv/kernel/vdso/Makefile | 2 +- arch/riscv/purgatory/Makefile | 4 + 14 files changed, 228 insertions(+), 168 deletions(-) create mode 100644 arch/riscv/include/asm/scs.h
base-commit: 52a93d39b17dc7eb98b6aa3edb93943248e03b2f
Signature · 1 line
2.41.0.640.ga95def55d0-goog
PATCH 1/5 riscv: VMAP_STACK overflow detection thread-safe
From: Deepak Gupta <[email protected]>
commit 31da94c25aea ("riscv: add VMAP_STACK overflow detection") added support for CONFIG_VMAP_STACK. If overflow is detected, CPU switches to shadow_stack temporarily before switching finally to per-cpu overflow_stack.
If two CPUs/harts are racing and end up in over flowing kernel stack, one or both will end up corrupting each other state because shadow_stack is not per-cpu. This patch optimizes per-cpu overflow stack switch by directly picking per-cpu overflow_stack and gets rid of shadow_stack.
Following are the changes in this patch
- Defines an asm macro to obtain per-cpu symbols in destination register.
- In entry.S, when overflow is detected, per-cpu overflow stack is located using per-cpu asm macro. Computing per-cpu symbol requires a temporary register. x31 is saved away into CSR_SCRATCH (CSR_SCRATCH is anyways zero since we're in kernel).
Please see Links for additional relevant disccussion and alternative solution.
Tested by echo EXHAUST_STACK > /sys/kernel/debug/provoke-crash/DIRECT Kernel crash log below
Insufficient stack space to handle exception!/debug/provoke-crash/DIRECT
Task stack: [0xff20000010a98000..0xff20000010a9c000]
Overflow stack: [0xff600001f7d98370..0xff600001f7d99370]
CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) epc : __memset+0x60/0xfc ra : recursive_loop+0x48/0xc6 [lkdtm] epc : ffffffff808de0e4 ra : ffffffff0163a752 sp : ff20000010a97e80 gp : ffffffff815c0330 tp : ff600000820ea280 t0 : ff20000010a97e88 t1 : 000000000000002e t2 : 3233206874706564 s0 : ff20000010a982b0 s1 : 0000000000000012 a0 : ff20000010a97e88 a1 : 0000000000000000 a2 : 0000000000000400 a3 : ff20000010a98288 a4 : 0000000000000000 a5 : 0000000000000000 a6 : fffffffffffe43f0 a7 : 00007fffffffffff s2 : ff20000010a97e88 s3 : ffffffff01644680 s4 : ff20000010a9be90 s5 : ff600000842ba6c0 s6 : 00aaaaaac29e42b0 s7 : 00fffffff0aa3684 s8 : 00aaaaaac2978040 s9 : 0000000000000065 s10: 00ffffff8a7cad10 s11: 00ffffff8a76a4e0 t3 : ffffffff815dbaf4 t4 : ffffffff815dbaf4 t5 : ffffffff815dbab8 t6 : ff20000010a9bb48 status: 0000000200000120 badaddr: ff20000010a97e88 cause: 000000000000000f Kernel panic - not syncing: Kernel stack overflow CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) Call Trace: [<ffffffff80006754>] dump_backtrace+0x30/0x38 [<ffffffff808de798>] show_stack+0x40/0x4c [<ffffffff808ea2a8>] dump_stack_lvl+0x44/0x5c [<ffffffff808ea2d8>] dump_stack+0x18/0x20 [<ffffffff808dec06>] panic+0x126/0x2fe [<ffffffff800065ea>] walk_stackframe+0x0/0xf0 [<ffffffff0163a752>] recursive_loop+0x48/0xc6 [lkdtm] SMP: stopping secondary CPUs ---[ end Kernel panic - not syncing: Kernel stack overflow ]---
--- arch/riscv/include/asm/asm.h | 16 +++++++ arch/riscv/include/asm/thread_info.h | 3 -- arch/riscv/kernel/asm-offsets.c | 1 + arch/riscv/kernel/entry.S | 70 ++++------------------------ arch/riscv/kernel/traps.c | 36 +------------- 5 files changed, 28 insertions(+), 98 deletions(-)
arch/riscv/include/asm/asm.h +16 -0
@@ -82,6 +82,22 @@ .endr .endm +#ifdef CONFIG_32BIT +#define PER_CPU_OFFSET_SHIFT 2 +#else +#define PER_CPU_OFFSET_SHIFT 3 +#endif + +.macro asm_per_cpu dst sym tmp + REG_L \tmp, TASK_TI_CPU_NUM(tp) + slli \tmp, \tmp, PER_CPU_OFFSET_SHIFT + la \dst, __per_cpu_offset + add \dst, \dst, \tmp + REG_L \tmp, 0(\dst) + la \dst, \sym + add \dst, \dst, \tmp +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp)
arch/riscv/include/asm/thread_info.h +0 -3
@@ -34,9 +34,6 @@#ifndef __ASSEMBLY__ -extern long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE /sizeof (long )]; -extern unsigned long spin_shadow_stack; -#include <asm/processor.h> #include <asm/csr.h>
arch/riscv/kernel/asm-offsets.c +1 -0
@@ -39,6 +39,7 @@ void asm_offsets(void)OFFSET (TASK_TI_KERNEL_SP , task_struct, thread_info.kernel_sp);OFFSET (TASK_TI_USER_SP , task_struct, thread_info.user_sp); +OFFSET (TASK_TI_CPU_NUM , task_struct, thread_info.cpu);OFFSET (TASK_THREAD_F0 , task_struct, thread.fstate.f[0 ]);OFFSET (TASK_THREAD_F1 , task_struct, thread.fstate.f[1 ]);OFFSET (TASK_THREAD_F2 , task_struct, thread.fstate.f[2 ]);
arch/riscv/kernel/entry.S +10 -60
86 lines not on this page · raw message
arch/riscv/kernel/traps.c +1 -35
@@ -397,48 +397,14 @@ int is_valid_bugaddr(unsigned long pc)#endif /* CONFIG_GENERIC_BUG */ #ifdef CONFIG_VMAP_STACK -/* -* Extra stack space that allows us to provide panic messages when the kernel -* has overflowed its stack. -*/ -static DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], +DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)__aligned(16); -/* -* A temporary stack for use by handle_kernel_stack_overflow. This is used so -* we can call into C code to get the per-hart overflow stack. Usage of this -* stack must be protected by spin_shadow_stack. -*/ -long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE /sizeof (long )]__aligned(16) ; - -/* -* A pseudo spinlock to protect the shadow stack from being used by multiple -* harts concurrently. This isn't a real spinlock because the lock side must -* be taken without a valid stack and only a single register, it's only taken -* while in the process of panicing anyway so the performance and error -* checking a proper spinlock gives us doesn't matter. -*/ -unsigned long spin_shadow_stack; - -asmlinkage unsigned long get_overflow_stack (void ) -{ -return (unsigned long )this_cpu_ptr (overflow_stack) + -OVERFLOW_STACK_SIZE ; -} asmlinkage void handle_bad_stack(struct pt_regs *regs) {unsigned long tsk_stk = (unsigned long )current->stack;unsigned long ovf_stk = (unsigned long )this_cpu_ptr (overflow_stack); -/* -* We're done with the shadow stack by this point, as we're on the -* overflow stack. Tell any other concurrent overflowing harts that -* they can proceed with panicing by releasing the pseudo-spinlock. -* -* This pairs with an amoswap.aq in handle_kernel_stack_overflow. -*/ -smp_store_release (&spin_shadow_stack,0 ); -console_verbose ();pr_emerg ("Insufficient stack space to handle exception!\n" );
Signature · 1 line
2.41.0.640.ga95def55d0-goog
kernel test robot
build botRe: [PATCH 1/5] riscv: VMAP_STACK overflow detection thread-safe
kernel test robot noticed the following build errors:
Hi Sami,
kernel test robot noticed the following build errors:
[auto build test ERROR on 52a93d39b17dc7eb98b6aa3edb93943248e03b2f]
url: https://github.com/intel-lab-lkp/linux/commits/Sami-Tolvanen/riscv-VMAP_STACK-overflow-detection-thread-safe/20230812-073751 base: 52a93d39b17dc7eb98b6aa3edb93943248e03b2f patch link: https://lore.kernel.org/r/20230811233556.97161-8-samitolvanen%40google.com patch subject: [PATCH 1/5] riscv: VMAP_STACK overflow detection thread-safe config: riscv-randconfig-r042-20230812 (https://download.01.org/0day-ci/archive/20230812/[email protected]/config) compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a) reproduce: (https://download.01.org/0day-ci/archive/20230812/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> ld.lld: error: undefined symbol: __per_cpu_offset >>> referenced by arch/riscv/kernel/entry.o:(handle_kernel_stack_overflow) in archive vmlinux.a
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH 1/5] riscv: VMAP_STACK overflow detection thread-safe
Quoted 15 lines · kernel test robot
Hi Sami,
kernel test robot noticed the following build errors:
[auto build test ERROR on
52a93d39b17dc7eb98b6aa3edb93943248e03b2f]url: https://github.com/intel-lab-lkp/linux/commits/Sami-Tolvanen/riscv-VMAP_STACK-overflow-detection-thread-safe/20230812-073751 base: 52a93d39b17dc7eb98b6aa3edb93943248e03b2f patch link: https://lore.kernel.org/r/20230811233556.97161-8-samitolvanen%40google.com patch subject: [PATCH 1/5] riscv: VMAP_STACK overflow detection thread-safe config: riscv-randconfig-r042-20230812 (https://download.01.org/0day-ci/archive/20230812/[email protected]/config) compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a) reproduce: (https://download.01.org/0day-ci/archive/20230812/[email protected]/reproduce)
Quoted 9 lines
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/All errors (new ones prefixed by >>):
Quoted 2 lines
Quoted 2 lines
ld.lld: error: undefined symbol:
__per_cpu_offsetQuoted 1 line
referenced by arch/riscv/kernel/entry.o:(handle_kernel_stack_overflow) in archive vmlinux.a
!CONFIG_SMP missed
Quoted 4 lines
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
Signature · 2 lines
Best Regards Guo Ren
Re: [PATCH 1/5] riscv: VMAP_STACK overflow detection thread-safe
Quoted 22 lines · Guo Ren
Quoted 20 lines · kernel test robot
Hi Sami,
kernel test robot noticed the following build errors:
[auto build test ERROR on
52a93d39b17dc7eb98b6aa3edb93943248e03b2f]url: https://github.com/intel-lab-lkp/linux/commits/Sami-Tolvanen/riscv-VMAP_STACK-overflow-detection-thread-safe/20230812-073751 base: 52a93d39b17dc7eb98b6aa3edb93943248e03b2f patch link: https://lore.kernel.org/r/20230811233556.97161-8-samitolvanen%40google.com patch subject: [PATCH 1/5] riscv: VMAP_STACK overflow detection thread-safe config: riscv-randconfig-r042-20230812 (https://download.01.org/0day-ci/archive/20230812/[email protected]/config) compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a) reproduce: (https://download.01.org/0day-ci/archive/20230812/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
Quoted 5 lines
Quoted 4 lines
All errors (new ones prefixed by >>):
Quoted 2 lines
Quoted 2 lines
ld.lld: error: undefined symbol:
__per_cpu_offsetQuoted 1 line
referenced by arch/riscv/kernel/entry.o:(handle_kernel_stack_overflow) in archive vmlinux.a
!CONFIG_SMP missed
Indeed. I'll fix this in v2.
Sami
PATCH 2/5 riscv: Deduplicate IRQ stack switching
In reply to Sami TolvanenWith CONFIG_IRQ_STACKS, we switch to a separate per-CPU IRQ stack before calling handle_riscv_irq or __do_softirq. We currently have duplicate inline assembly snippets for stack switching in both code paths. Now that we can access per-CPU variables in assembly, implement call_on_irq_stack in assembly, and use that instead of redudant inline assembly.
--- arch/riscv/include/asm/asm.h | 5 +++++ arch/riscv/include/asm/irq_stack.h | 3 +++ arch/riscv/kernel/entry.S | 32 ++++++++++++++++++++++++++++++ arch/riscv/kernel/irq.c | 32 ++++++++---------------------- arch/riscv/kernel/traps.c | 29 ++++----------------------- 5 files changed, 52 insertions(+), 49 deletions(-)
arch/riscv/include/asm/asm.h +5 -0
@@ -98,6 +98,11 @@ add \dst, \dst, \tmp .endm +.macro load_per_cpu dst ptr tmp + asm_per_cpu \dst \ptr \tmp + REG_L \dst, 0(\dst) +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp)
arch/riscv/include/asm/irq_stack.h +3 -0
@@ -12,6 +12,9 @@ DECLARE_PER_CPU(ulong *, irq_stack_ptr); +asmlinkage void call_on_irq_stack(struct pt_regs *regs, +void (*func)(struct pt_regs *)); +#ifdef CONFIG_VMAP_STACK /* * To ensure that VMAP'd stack overflow detection works correctly, all VMAP'd
arch/riscv/kernel/entry.S +32 -0
@@ -218,6 +218,38 @@ SYM_CODE_START(ret_from_fork) tail syscall_exit_to_user_mode SYM_CODE_END(ret_from_fork) +#ifdef CONFIG_IRQ_STACKS +/* + * void call_on_irq_stack(struct pt_regs *regs, + * void (*func)(struct pt_regs *)); + * + * Calls func(regs) using the per-CPU IRQ stack. + */ +SYM_FUNC_START(call_on_irq_stack) + /* Create a frame record to save ra and s0 (fp) */ + addi sp, sp, -RISCV_SZPTR + REG_S ra, (sp) + addi sp, sp, -RISCV_SZPTR + REG_S s0, (sp) + addi s0, sp, 2*RISCV_SZPTR + + /* Switch to the per-CPU IRQ stack and call the handler */ + load_per_cpu t0, irq_stack_ptr, t1 + li t1, IRQ_STACK_SIZE + add sp, t0, t1 + jalr a1 + + /* Switch back to the thread stack and restore ra and s0 */ + addi sp, s0, -2*RISCV_SZPTR + REG_L s0, (sp) + addi sp, sp, RISCV_SZPTR + REG_L ra, (sp) + addi sp, sp, RISCV_SZPTR + + ret +SYM_FUNC_END(call_on_irq_stack) +#endif /* CONFIG_IRQ_STACKS */ + /* * Integer register context switch * The callee-saved registers must be saved and restored.
arch/riscv/kernel/irq.c +8 -24
@@ -61,32 +61,16 @@ static void init_irq_stacks(void)#endif /* CONFIG_VMAP_STACK */ #ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK +static void ___do_softirq (struct pt_regs *regs) +{ +__do_softirq (); +} +void do_softirq_own_stack (void ) { -#ifdef CONFIG_IRQ_STACKS -if (on_thread_stack ()) { -ulong *sp =per_cpu (irq_stack_ptr,smp_processor_id ()) - +IRQ_STACK_SIZE /sizeof (ulong); - __asm__ __volatile( -"addi sp, sp, -" RISCV_SZPTR "\n" -REG_S " ra, (sp) \n" -"addi sp, sp, -" RISCV_SZPTR "\n" -REG_S " s0, (sp) \n" -"addi s0, sp, 2*" RISCV_SZPTR "\n" -"move sp, %[sp] \n" -"call __do_softirq \n" -"addi sp, s0, -2*" RISCV_SZPTR "\n" -REG_L " s0, (sp) \n" -"addi sp, sp, " RISCV_SZPTR "\n" -REG_L " ra, (sp) \n" -"addi sp, sp, " RISCV_SZPTR "\n" - : - : [sp] "r" (sp) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", -"t0" ,"t1" ,"t2" ,"t3" ,"t4" ,"t5" ,"t6" , -"memory" ); - } else -#endif +if (on_thread_stack ()) +call_on_irq_stack (NULL , ___do_softirq); +else __do_softirq (); }#endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */
arch/riscv/kernel/traps.c +4 -25
@@ -350,31 +350,10 @@ static void noinstr handle_riscv_irq(struct pt_regs *regs) asmlinkage void noinstr do_irq(struct pt_regs *regs) {irqentry_state_t state =irqentry_enter (regs); -#ifdef CONFIG_IRQ_STACKS -if (on_thread_stack ()) { -ulong *sp =per_cpu (irq_stack_ptr,smp_processor_id ()) - +IRQ_STACK_SIZE /sizeof (ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "move a0, %[regs] \n" - "call handle_riscv_irq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp), [regs] "r" (regs) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", -" t0", " t1", " t2", " t3", " t4", " t5", " t6", - "memory"); - } else -#endif + +if (IS_ENABLED (CONFIG_IRQ_STACKS ) &&on_thread_stack ()) +call_on_irq_stack (regs, handle_riscv_irq); +else handle_riscv_irq (regs);irqentry_exit (regs, state);
Signature · 1 line
2.41.0.640.ga95def55d0-goog
PATCH 3/5 riscv: Move global pointer loading to a macro
In reply to Sami TolvanenIn Clang 17, -fsanitize=shadow-call-stack uses the newly declared platform register gp for storing shadow call stack pointers. As this is obviously incompatible with gp relaxation, in preparation for CONFIG_SHADOW_CALL_STACK support, move global pointer loading to a single macro, which we can cleanly disable when SCS is used instead.
--- arch/riscv/include/asm/asm.h | 8 ++++++++ arch/riscv/kernel/entry.S | 6 ++---- arch/riscv/kernel/head.S | 15 +++------------ arch/riscv/kernel/suspend_entry.S | 5 +---- 4 files changed, 14 insertions(+), 20 deletions(-)
arch/riscv/include/asm/asm.h +8 -0
@@ -103,6 +103,14 @@ REG_L \dst, 0(\dst) .endm +/* load __global_pointer to gp */ +.macro load_global_pointer +.option push +.option norelax + la gp, __global_pointer$ +.option pop +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp)
arch/riscv/kernel/entry.S +2 -4
@@ -75,10 +75,8 @@ _save_context: csrw CSR_SCRATCH, x0 /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer + move a0, sp /* pt_regs */ la ra, ret_from_exception
arch/riscv/kernel/head.S +3 -12
@@ -110,10 +110,7 @@ relocate_enable_mmu: csrw CSR_TVEC, a0 /* Reload the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer /* * Switch to kernel page tables. A full fence is necessary in order to
@@ -134,10 +131,7 @@ secondary_start_sbi: csrw CSR_IP, zero /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer /* * Disable FPU & VECTOR to detect illegal usage of
@@ -228,10 +222,7 @@ pmp_done: #endif /* CONFIG_RISCV_M_MODE */ /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer /* * Disable FPU & VECTOR to detect illegal usage of
arch/riscv/kernel/suspend_entry.S +1 -4
@@ -60,10 +60,7 @@ END(__cpu_suspend_enter) ENTRY(__cpu_resume_enter) /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer #ifdef CONFIG_MMU /* Save A0 and A1 */
Signature · 1 line
2.41.0.640.ga95def55d0-goog
PATCH 4/5 riscv: Implement Shadow Call Stack
In reply to Sami TolvanenImplement CONFIG_SHADOW_CALL_STACK for RISC-V. When enabled, the compiler injects instructions to all non-leaf C functions to store the return address to the shadow stack and unconditionally load it again before returning, which makes it harder to corrupt the return address through a stack overflow, for example.
The active shadow call stack pointer is stored in the gp register, which makes SCS incompatible with gp relaxation. Use --no-relax-gp to ensure gp relaxation is disabled and disable global pointer loading. Add SCS pointers to struct thread_info, implement SCS initialization, and task switching
--- arch/riscv/Kconfig | 6 ++++ arch/riscv/Makefile | 4 +++ arch/riscv/include/asm/asm.h | 6 ++++ arch/riscv/include/asm/scs.h | 47 ++++++++++++++++++++++++++++ arch/riscv/include/asm/thread_info.h | 13 ++++++++ arch/riscv/kernel/asm-offsets.c | 3 ++ arch/riscv/kernel/entry.S | 11 +++++++ arch/riscv/kernel/head.S | 4 +++ arch/riscv/kernel/vdso/Makefile | 2 +- arch/riscv/purgatory/Makefile | 4 +++ 10 files changed, 99 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/include/asm/scs.h
arch/riscv/Kconfig +6 -0
@@ -46,6 +46,7 @@ config RISCV select ARCH_SUPPORTS_HUGETLBFS if MMU select ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU select ARCH_SUPPORTS_PER_VMA_LOCK if MMU + select ARCH_SUPPORTS_SHADOW_CALL_STACK if HAVE_SHADOW_CALL_STACK select ARCH_USE_MEMTEST select ARCH_USE_QUEUED_RWLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU
@@ -169,6 +170,11 @@ config GCC_SUPPORTS_DYNAMIC_FTRACE def_bool CC_IS_GCC depends on $(cc-option,-fpatchable-function-entry=8) +config HAVE_SHADOW_CALL_STACK + def_bool $(cc-option,-fsanitize=shadow-call-stack) + # https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e6eeb51f0cb7b8819e50da6d2444d769 + depends on $(ld-option,--no-relax-gp) + config ARCH_MMAP_RND_BITS_MIN default 18 if 64BIT default 8
arch/riscv/Makefile +4 -0
@@ -55,6 +55,10 @@ endif endif endif +ifeq ($(CONFIG_SHADOW_CALL_STACK),y) + KBUILD_LDFLAGS += --no-relax-gp +endif + # ISA string setting riscv-march-$(CONFIG_ARCH_RV32I) := rv32ima riscv-march-$(CONFIG_ARCH_RV64I) := rv64ima
arch/riscv/include/asm/asm.h +6 -0
@@ -103,6 +103,11 @@ REG_L \dst, 0(\dst) .endm +#ifdef CONFIG_SHADOW_CALL_STACK +/* gp is used as the shadow call stack pointer instead */ +.macro load_global_pointer +.endm +#else /* load __global_pointer to gp */ .macro load_global_pointer .option push
@@ -110,6 +115,7 @@ la gp, __global_pointer$ .option pop .endm +#endif /* CONFIG_SHADOW_CALL_STACK */ /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31
arch/riscv/include/asm/scs.h new file +47 -0
47 lines not on this page · raw message
arch/riscv/include/asm/thread_info.h +13 -0
@@ -57,8 +57,20 @@ struct thread_info {long user_sp;/* User stack pointer */ int cpu;unsigned long syscall_work;/* SYSCALL_WORK_ flags */ +#ifdef CONFIG_SHADOW_CALL_STACK +void *scs_base; +void *scs_sp; +#endif }; +#ifdef CONFIG_SHADOW_CALL_STACK +#define INIT_SCS \ + .scs_base = init_shadow_call_stack, \ + .scs_sp = init_shadow_call_stack, +#else +#define INIT_SCS +#endif +/* * macros/functions for gaining access to the thread information structure *
@@ -68,6 +80,7 @@ struct thread_info { { \ .flags = 0, \ .preempt_count = INIT_PREEMPT_COUNT, \ + INIT_SCS \ }void arch_release_task_struct (struct task_struct *tsk);
arch/riscv/kernel/asm-offsets.c +3 -0
@@ -38,6 +38,9 @@ void asm_offsets(void)OFFSET (TASK_TI_PREEMPT_COUNT , task_struct, thread_info.preempt_count);OFFSET (TASK_TI_KERNEL_SP , task_struct, thread_info.kernel_sp);OFFSET (TASK_TI_USER_SP , task_struct, thread_info.user_sp); +#ifdef CONFIG_SHADOW_CALL_STACK +OFFSET (TASK_TI_SCS_SP , task_struct, thread_info.scs_sp); +#endif OFFSET (TASK_TI_CPU_NUM , task_struct, thread_info.cpu);OFFSET (TASK_THREAD_F0 , task_struct, thread.fstate.f[0 ]);
arch/riscv/kernel/entry.S +11 -0
@@ -9,6 +9,7 @@ #include <asm/asm.h> #include <asm/csr.h> +#include <asm/scs.h> #include <asm/unistd.h> #include <asm/page.h> #include <asm/thread_info.h>
@@ -77,6 +78,9 @@ _save_context: /* Load the global pointer */ load_global_pointer + /* Load the kernel shadow call stack pointer if coming from userspace */ + scs_load_current_if_task_changed s5 + move a0, sp /* pt_regs */ la ra, ret_from_exception
@@ -123,6 +127,9 @@ SYM_CODE_START_NOALIGN(ret_from_exception) addi s0, sp, PT_SIZE_ON_STACK REG_S s0, TASK_TI_KERNEL_SP(tp) + /* Save the kernel shadow call stack pointer */ + scs_save_current + /* * Save TP into the scratch register , so we can find the kernel data * structures again.
@@ -277,6 +284,8 @@ SYM_FUNC_START(__switch_to) REG_S s9, TASK_THREAD_S9_RA(a3) REG_S s10, TASK_THREAD_S10_RA(a3) REG_S s11, TASK_THREAD_S11_RA(a3) + /* Save the kernel shadow call stack pointer */ + scs_save_current /* Restore context from next->thread */ REG_L ra, TASK_THREAD_RA_RA(a4) REG_L sp, TASK_THREAD_SP_RA(a4)
@@ -294,6 +303,8 @@ SYM_FUNC_START(__switch_to) REG_L s11, TASK_THREAD_S11_RA(a4) /* The offset of thread_info in task_struct is zero. */ move tp, a1 + /* Switch to the next shadow call stack */ + scs_load_current ret SYM_FUNC_END(__switch_to)
arch/riscv/kernel/head.S +4 -0
@@ -14,6 +14,7 @@ #include <asm/cpu_ops_sbi.h> #include <asm/hwcap.h> #include <asm/image.h> +#include <asm/scs.h> #include <asm/xip_fixup.h> #include "efi-header.S"
@@ -153,6 +154,7 @@ secondary_start_sbi: XIP_FIXUP_OFFSET a3 add a3, a3, a1 REG_L sp, (a3) + scs_load_current .Lsecondary_start_common:
@@ -293,6 +295,7 @@ clear_bss_done: la sp, init_thread_union + THREAD_SIZE XIP_FIXUP_OFFSET sp addi sp, sp, -PT_SIZE_ON_STACK + scs_load_init_stack #ifdef CONFIG_BUILTIN_DTB la a0, __dtb_start XIP_FIXUP_OFFSET a0
@@ -311,6 +314,7 @@ clear_bss_done: la tp, init_task la sp, init_thread_union + THREAD_SIZE addi sp, sp, -PT_SIZE_ON_STACK + scs_load_init_stack #ifdef CONFIG_KASAN call kasan_early_init
arch/riscv/kernel/vdso/Makefile +1 -1
@@ -36,7 +36,7 @@ CPPFLAGS_vdso.lds += -DHAS_VGETTIMEOFDAY endif # Disable -pg to prevent insert call site -CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) # Disable profiling and instrumentation for VDSO code GCOV_PROFILE := n
arch/riscv/purgatory/Makefile +4 -0
@@ -77,6 +77,10 @@ ifdef CONFIG_STACKPROTECTOR_STRONG PURGATORY_CFLAGS_REMOVE += -fstack-protector-strong endif +ifdef CONFIG_SHADOW_CALL_STACK +PURGATORY_CFLAGS_REMOVE += $(CC_FLAGS_SCS) +endif + CFLAGS_REMOVE_purgatory.o += $(PURGATORY_CFLAGS_REMOVE) CFLAGS_purgatory.o += $(PURGATORY_CFLAGS)
Signature · 1 line
2.41.0.640.ga95def55d0-goog
PATCH 5/5 riscv: Use separate IRQ shadow call stacks
In reply to Sami TolvanenWhen both CONFIG_IRQ_STACKS and SCS are enabled, also use a separate per-CPU shadow call stack.
--- arch/riscv/include/asm/scs.h | 7 +++++++ arch/riscv/kernel/entry.S | 7 +++++++ arch/riscv/kernel/irq.c | 21 +++++++++++++++++++++ 3 files changed, 35 insertions(+)
arch/riscv/include/asm/scs.h +7 -0
@@ -13,6 +13,11 @@ XIP_FIXUP_OFFSET gp .endm +/* Load the per-CPU IRQ shadow call stack to gp. */ +.macro scs_load_irq_stack tmp + load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp +.endm + /* Load task_scs_sp(current) to gp. */ .macro scs_load_current REG_L gp, TASK_TI_SCS_SP(tp)
@@ -34,6 +39,8 @@ .macro scs_load_init_stack .endm +.macro scs_load_irq_stack tmp +.endm .macro scs_load_current .endm .macro scs_load_current_if_task_changed prev
arch/riscv/kernel/entry.S +7 -0
@@ -238,12 +238,19 @@ SYM_FUNC_START(call_on_irq_stack) REG_S s0, (sp) addi s0, sp, 2*RISCV_SZPTR + /* Switch to the per-CPU shadow call stack */ + scs_save_current + scs_load_irq_stack t0 + /* Switch to the per-CPU IRQ stack and call the handler */ load_per_cpu t0, irq_stack_ptr, t1 li t1, IRQ_STACK_SIZE add sp, t0, t1 jalr a1 + /* Switch back to the thread shadow call stack */ + scs_load_current + /* Switch back to the thread stack and restore ra and s0 */ addi sp, s0, -2*RISCV_SZPTR REG_L s0, (sp)
arch/riscv/kernel/irq.c +21 -0
@@ -9,6 +9,7 @@#include <linux/irqchip.h> #include <linux/irqdomain.h> #include <linux/module.h> +#include <linux/scs.h> #include <linux/seq_file.h> #include <asm/sbi.h> #include <asm/smp.h>
@@ -34,6 +35,24 @@ EXPORT_SYMBOL_GPL(riscv_get_intc_hwnode);#ifdef CONFIG_IRQ_STACKS #include <asm/irq_stack.h> +DECLARE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); + +#ifdef CONFIG_SHADOW_CALL_STACK +DEFINE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); +#endif + +static void init_irq_scs (void ) +{ +int cpu; + +if (!scs_is_enabled ()) +return ; + +for_each_possible_cpu (cpu) +per_cpu (irq_shadow_call_stack_ptr, cpu) = +scs_alloc (cpu_to_node (cpu)); +} + DEFINE_PER_CPU(ulong *, irq_stack_ptr);#ifdef CONFIG_VMAP_STACK
@@ -76,6 +95,7 @@ void do_softirq_own_stack(void)#endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ #else +static void init_irq_scs (void ) {}static void init_irq_stacks (void ) {}#endif /* CONFIG_IRQ_STACKS */
@@ -87,6 +107,7 @@ int arch_show_interrupts(struct seq_file *p, int prec)void __init init_IRQ (void ) { +init_irq_scs ();init_irq_stacks ();irqchip_init ();if (!handle_arch_irq)
Signature · 1 line
2.41.0.640.ga95def55d0-goog
Hi Sami,
Quoted 24 lines · Sami Tolvanen
Hi folks,
This series adds Shadow Call Stack (SCS) support for RISC-V. SCS uses compiler instrumentation to store return addresses in a separate shadow stack to protect them against accidental or malicious overwrites. More information about SCS can be found here:
https://clang.llvm.org/docs/ShadowCallStack.html
Patch 1 is from Deepak, and it simplifies
VMAP_STACKoverflow handling by adding support for accessing per-CPU variables directly in assembly. The patch is included in this series to make IRQ stack switching cleaner with SCS, and I've simply rebased it. Patch 2 uses this functionality to clean up the stack switching by moving duplicate code into a single function. On RISC-V, the compiler uses the gp register for storing the current shadow call stack pointer, which is incompatible with global pointer relaxation. Patch 3 moves global pointer loading into a macro that can be easily disabled with SCS. Patch 4 implements SCS register loading and switching, and allows the feature to be enabled, and patch 5 adds separate per-CPU IRQ shadow call stacks whenCONFIG_IRQ_STACKSis enabled.
Quoted 10 lines
Note that this series requires Clang 17. Earlier Clang versions support SCS on RISC-V, but use the x18 register instead of gp, which isn't ideal. gcc has SCS support for arm64, but I'm not aware of plans to support RISC-V. Once the Zicfiss extension is ratified, it's probably preferable to use hardware-backed shadow stacks instead of SCS on hardware that supports the extension, and we may want to consider implementing
CONFIG_DYNAMIC_SCSto patch between the implementation at runtime (similarly to the arm64 implementation, which switches to SCS when hardware PAC support isn't available).
I took this series for a spin on top of 6.5-rc6 with both LLVM 18 (built within the past couple of days) and LLVM 17.0.0-rc2 but it seems that the CFI_BACKWARDS LKDTM test does not pass with CONFIG_SHADOW_CALL_STACK=y.
[ 73.324652] lkdtm: Performing direct entry CFI_BACKWARD [ 73.324900] lkdtm: Attempting unchecked stack return address redirection ... [ 73.325178] lkdtm: Eek: return address mismatch! 0000000000000002 != ffffffff80614982 [ 73.325478] lkdtm: FAIL: stack return address manipulation failed!
Does the test need to be adjusted or is there some other issue?
Cheers,
Nathan
Quoted 43 lines · Nathan Chancellor
Hi Sami,
Quoted 24 lines · Sami Tolvanen
Hi folks,
This series adds Shadow Call Stack (SCS) support for RISC-V. SCS uses compiler instrumentation to store return addresses in a separate shadow stack to protect them against accidental or malicious overwrites. More information about SCS can be found here:
https://clang.llvm.org/docs/ShadowCallStack.html
Patch 1 is from Deepak, and it simplifies
VMAP_STACKoverflow handling by adding support for accessing per-CPU variables directly in assembly. The patch is included in this series to make IRQ stack switching cleaner with SCS, and I've simply rebased it. Patch 2 uses this functionality to clean up the stack switching by moving duplicate code into a single function. On RISC-V, the compiler uses the gp register for storing the current shadow call stack pointer, which is incompatible with global pointer relaxation. Patch 3 moves global pointer loading into a macro that can be easily disabled with SCS. Patch 4 implements SCS register loading and switching, and allows the feature to be enabled, and patch 5 adds separate per-CPU IRQ shadow call stacks whenCONFIG_IRQ_STACKSis enabled.Quoted 10 lines
Note that this series requires Clang 17. Earlier Clang versions support SCS on RISC-V, but use the x18 register instead of gp, which isn't ideal. gcc has SCS support for arm64, but I'm not aware of plans to support RISC-V. Once the Zicfiss extension is ratified, it's probably preferable to use hardware-backed shadow stacks instead of SCS on hardware that supports the extension, and we may want to consider implementing
CONFIG_DYNAMIC_SCSto patch between the implementation at runtime (similarly to the arm64 implementation, which switches to SCS when hardware PAC support isn't available).I took this series for a spin on top of 6.5-rc6 with both LLVM 18 (built within the past couple of days) and LLVM 17.0.0-rc2 but it seems that the
CFI_BACKWARDSLKDTM test does not pass withCONFIG_SHADOW_CALL_STACK=y.
Quoted 6 lines
[ 73.324652] lkdtm: Performing direct entry CFI_BACKWARD [ 73.324900] lkdtm: Attempting unchecked stack return address redirection ... [ 73.325178] lkdtm: Eek: return address mismatch! 0000000000000002 != ffffffff80614982 [ 73.325478] lkdtm: FAIL: stack return address manipulation failed!Does the test need to be adjusted or is there some other issue?
Does it pass without the series? I tried to write it to be arch-agnostic, but I never tested it on RISC-V. It's very possible that test needs adjusting for the architecture. Besides the label horrors, the use of __builtin_frame_address may not work there either...
Signature · 1 line
Kees Cook
Quoted 50 lines · Kees Cook
Quoted 48 lines · Nathan Chancellor
Hi Sami,
Quoted 24 lines · Sami Tolvanen
Hi folks,
This series adds Shadow Call Stack (SCS) support for RISC-V. SCS uses compiler instrumentation to store return addresses in a separate shadow stack to protect them against accidental or malicious overwrites. More information about SCS can be found here:
https://clang.llvm.org/docs/ShadowCallStack.html
Patch 1 is from Deepak, and it simplifies
VMAP_STACKoverflow handling by adding support for accessing per-CPU variables directly in assembly. The patch is included in this series to make IRQ stack switching cleaner with SCS, and I've simply rebased it. Patch 2 uses this functionality to clean up the stack switching by moving duplicate code into a single function. On RISC-V, the compiler uses the gp register for storing the current shadow call stack pointer, which is incompatible with global pointer relaxation. Patch 3 moves global pointer loading into a macro that can be easily disabled with SCS. Patch 4 implements SCS register loading and switching, and allows the feature to be enabled, and patch 5 adds separate per-CPU IRQ shadow call stacks whenCONFIG_IRQ_STACKSis enabled.Quoted 10 lines
Note that this series requires Clang 17. Earlier Clang versions support SCS on RISC-V, but use the x18 register instead of gp, which isn't ideal. gcc has SCS support for arm64, but I'm not aware of plans to support RISC-V. Once the Zicfiss extension is ratified, it's probably preferable to use hardware-backed shadow stacks instead of SCS on hardware that supports the extension, and we may want to consider implementing
CONFIG_DYNAMIC_SCSto patch between the implementation at runtime (similarly to the arm64 implementation, which switches to SCS when hardware PAC support isn't available).I took this series for a spin on top of 6.5-rc6 with both LLVM 18 (built within the past couple of days) and LLVM 17.0.0-rc2 but it seems that the
CFI_BACKWARDSLKDTM test does not pass withCONFIG_SHADOW_CALL_STACK=y.[ 73.324652] lkdtm: Performing direct entry CFI_BACKWARD [ 73.324900] lkdtm: Attempting unchecked stack return address redirection ... [ 73.325178] lkdtm: Eek: return address mismatch! 0000000000000002 != ffffffff80614982 [ 73.325478] lkdtm: FAIL: stack return address manipulation failed!
Quoted 6 lines
Quoted 1 line
Does the test need to be adjusted or is there some other issue?
Does it pass without the series? I tried to write it to be arch-agnostic, but I never tested it on RISC-V. It's very possible that test needs adjusting for the architecture. Besides the label horrors, the use of
__builtin_frame_addressmay not work there either...
Looks like __builtin_frame_address behaves differently on RISC-V. After staring at the disassembly a bit, using __builtin_frame_address(0) - 1 instead of + 1 seems to yield correct results. WIth that change, here's the test without SCS:
# echo CFI_BACKWARD > /sys/kernel/debug/provoke-crash/DIRECT [ 16.272028] lkdtm: Performing direct entry CFI_BACKWARD [ 16.272368] lkdtm: Attempting unchecked stack return address redirection ... [ 16.272671] lkdtm: ok: redirected stack return address. [ 16.272885] lkdtm: Attempting checked stack return address redirection ... [ 16.273384] lkdtm: FAIL: stack return address was redirected! [ 16.273755] lkdtm: This is probably expected, since this kernel (6.5.0-rc5-00005-g5a1201f89265-dirty riscv64) was built *without* CONFIG_ARM64_PTR_AUTH_KERNEL=y nor CONFIG_SHADOW_CALL_STACK=y
And with SCS:
# echo CFI_BACKWARD > /sys/kernel/debug/provoke-crash/DIRECT [ 17.429551] lkdtm: Performing direct entry CFI_BACKWARD [ 17.430184] lkdtm: Attempting unchecked stack return address redirection ... [ 17.431402] lkdtm: ok: redirected stack return address. [ 17.432031] lkdtm: Attempting checked stack return address redirection ... [ 17.432861] lkdtm: ok: control flow unchanged.
Sami
Quoted 4 lines · Sami Tolvanen
Looks like
__builtin_frame_addressbehaves differently on RISC-V. After staring at the disassembly a bit, using__builtin_frame_address(0)- 1 instead of + 1 seems to yield correct results.
Elliott was kind enough to point out to me off-list that this behavior has been documented here:
I'll include a patch to fix the test on RISC-V in the next version.
Sami
Quoted 6 lines · Sami Tolvanen
Quoted 4 lines · Sami Tolvanen
Looks like
__builtin_frame_addressbehaves differently on RISC-V. After staring at the disassembly a bit, using__builtin_frame_address(0)- 1 instead of + 1 seems to yield correct results.
Quoted 6 lines
Elliott was kind enough to point out to me off-list that this behavior has been documented here:
I'll include a patch to fix the test on RISC-V in the next version.
Perfect! Thanks for figuring out the offset. :)
Signature · 1 line
Kees Cook
Quoted 11 lines · Nathan Chancellor
I took this series for a spin on top of 6.5-rc6 with both LLVM 18 (built within the past couple of days) and LLVM 17.0.0-rc2 but it seems that the
CFI_BACKWARDSLKDTM test does not pass withCONFIG_SHADOW_CALL_STACK=y.[ 73.324652] lkdtm: Performing direct entry CFI_BACKWARD [ 73.324900] lkdtm: Attempting unchecked stack return address redirection ... [ 73.325178] lkdtm: Eek: return address mismatch! 0000000000000002 != ffffffff80614982 [ 73.325478] lkdtm: FAIL: stack return address manipulation failed!Does the test need to be adjusted or is there some other issue?
The test doesn't work on RISC-V. set_return_addr_unchecked thinks 0x2 is the return address, so I assume the __builtin_frame_address logic isn't quite right here. Kees, any thoughts?
Sami