riscv: Add reliable stack unwinding for livepatch
Subsystems (7)
v5 cover riscv: Add reliable stack unwinding for livepatch
Hi,
This is v5 of the RISC-V reliable stack unwinding series for livepatch.
The series adds explicit frame-record metadata, preserves the frame pointer through dynamic ftrace, and replaces the open-coded stack walker with a conservative frame-pointer based unwinder. It then enables the RISC-V livepatch capability and adds the syscall-wrapper selftest prefix.
Changes since v4:
* Use sizeof(struct frame_record) for STACKFRAME_RECORD_SIZE, as requested by Shuai Xue.
* Guard IRQ stack accessors and IRQ stack entries with CONFIG_IRQ_STACKS, addressing the link-time issue reported by Shuai Xue and Sashiko.
* Add the corresponding Reviewed-by tag from Shuai Xue to 1/7.
* Rebase onto the latest riscv/for-next; no other functional changes.
The review replies have been sent in the v4 RESEND thread.
Base:
riscv/for-next ff492e321c47 ("Revert \"riscv: hwprobe: Fix stale vDSO data for late-initialized keys at boot\"")
Previous versions:
v4 RESEND: https://lore.kernel.org/r/[email protected]
v3: https://lore.kernel.org/r/[email protected]
v2: https://lore.kernel.org/r/[email protected]
v1: https://lore.kernel.org/r/[email protected]
Validation:
* Built Linux 7.2.0-rc6 with riscv64-unknown-linux-gnu-gcc 15.2.0
and binutils 2.46.
* Standard livepatch configuration built successfully, including Image, vmlinux, modules, compile_commands.json and GDB scripts.
* CONFIG_IRQ_STACKS=n variant linked successfully with GCC.
* QEMU 8.2.2, 1 CPU: livepatch selftests PASS: 7, SKIP: 1,
FAIL: 0; ftrace function graph 3 passed, 0 failed, 3 unsupported.
* QEMU 8.2.2, 4 CPUs: livepatch selftests PASS: 7, SKIP: 1,
FAIL: 0; ftrace function graph 3 passed, 0 failed, 3 unsupported.
* No BUG, WARNING, Oops, panic, unreliable stack or Call Trace found by the validation log scans. Expected negative livepatch errors are part of the selftests.
Wang Han (7):
riscv: stacktrace: Add frame record metadata
riscv: stacktrace: disable KASAN and KCOV instrumentation for
stacktrace.o
riscv: ftrace: always preserve s0 in dynamic ftrace register frame
riscv: stacktrace: introduce stack-bound tracking helpers
riscv: stacktrace: switch to frame-pointer based unwinder
riscv: Kconfig: enable HAVE_RELIABLE_STACKTRACE and HAVE_LIVEPATCH selftests/livepatch: Add RISC-V syscall wrapper prefix
arch/riscv/Kconfig | 4 + arch/riscv/include/asm/ptrace.h | 9 + arch/riscv/include/asm/stacktrace.h | 67 ++- arch/riscv/include/asm/stacktrace/common.h | 159 +++++ arch/riscv/include/asm/stacktrace/frame.h | 53 ++ arch/riscv/kernel/Makefile | 6 + arch/riscv/kernel/asm-offsets.c | 6 + arch/riscv/kernel/entry.S | 39 +- arch/riscv/kernel/ftrace.c | 6 +- arch/riscv/kernel/head.S | 23 + arch/riscv/kernel/mcount-dyn.S | 4 - arch/riscv/kernel/perf_callchain.c | 2 +- arch/riscv/kernel/process.c | 33 +- arch/riscv/kernel/stacktrace.c | 561 +++++++++++++++--- .../livepatch/test_modules/test_klp_syscall.c | 2 + 15 files changed, 868 insertions(+), 106 deletions(-) create mode 100644 arch/riscv/include/asm/stacktrace/common.h create mode 100644 arch/riscv/include/asm/stacktrace/frame.h
Range-diff against v4:
1: 8cef363cfed7 ! 1: cbb1dc004d0f riscv: stacktrace: Add frame record metadata
@@ Commit message
the secondary CPU path sets up s0 before smp_callin() so idle-task unwinding does not inherit an undefined caller frame;
* copy_thread creates matching final metadata records for new kernel
- and user tasks, and keeps s0 available for the frame-pointer chain;
- * call_on_irq_stack still reserves an aligned stack slot, but links the
- saved {fp, ra} with the raw frame-record size so s0 points at the
- RISC-V frame record rather than past the alignment padding.
+ and user tasks, and keeps s0 available for the frame-pointer chain.
- The call_on_irq_stack adjustment fixes a latent RV32 issue. On RV64,
- sizeof(struct stackframe) is equal to the stack alignment, so the old
- s0 value happened to point just above the saved {fp, ra}. On RV32, the
- raw frame record is 8 bytes while the reserved stack slot is 16-byte
- aligned, so the old s0 value pointed into the padding. Using the raw
- record size makes s0 point above the saved frame record on both RV32
- and RV64 while still reserving the aligned slot.
+ Keep the embedded metadata-record field offsets distinct from the
+ s0-relative STACKFRAME_* offsets used by call_on_irq_stack(), because
+ the latter describe a frame record relative to s0 rather than to the
+ record base.
These changes keep s0 reserved for the frame-pointer chain at task and
- stack-switch boundaries.
+ exception boundaries.
+ Reviewed-by: Shuai Xue <[email protected]>
## arch/riscv/include/asm/ptrace.h ##
@@ arch/riscv/kernel/asm-offsets.c: void asm_offsets(void)
OFFSET(HIBERN_PBE_ADDR, pbe, address);
@@ arch/riscv/kernel/asm-offsets.c: void asm_offsets(void)
- OFFSET(SBI_HART_BOOT_STACK_PTR_OFFSET, sbi_hart_boot_data, stack_ptr);
-
DEFINE(STACKFRAME_SIZE_ON_STACK, ALIGN(sizeof(struct stackframe), STACK_ALIGN));
-+ DEFINE(STACKFRAME_RECORD_SIZE, sizeof(struct stackframe));
- OFFSET(STACKFRAME_FP, stackframe, fp);
- OFFSET(STACKFRAME_RA, stackframe, ra);
+ DEFINE(STACKFRAME_FP, offsetof(struct stackframe, fp) - sizeof(struct stackframe));
+ DEFINE(STACKFRAME_RA, offsetof(struct stackframe, ra) - sizeof(struct stackframe));
++ DEFINE(STACKFRAME_RECORD_SIZE, sizeof(struct frame_record));
++ OFFSET(FRAME_RECORD_FP, frame_record, fp);
++ OFFSET(FRAME_RECORD_RA, frame_record, ra);
#ifdef CONFIG_FUNCTION_TRACER
+ DEFINE(FTRACE_OPS_FUNC, offsetof(struct ftrace_ops, func));
+ #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
## arch/riscv/kernel/entry.S ##
@@
@@ arch/riscv/kernel/entry.S: SYM_CODE_START(handle_exception)
+ * Create a metadata frame record. The unwinder will use this to
+ * identify and unwind exception boundaries.
+ */
-+ REG_S zero, (S_STACKFRAME + STACKFRAME_FP)(sp) /* stackframe.record.fp = 0 */
-+ REG_S zero, (S_STACKFRAME + STACKFRAME_RA)(sp) /* stackframe.record.ra = 0 */
++ REG_S zero, (S_STACKFRAME + FRAME_RECORD_FP)(sp) /* stackframe.record.fp = 0 */
++ REG_S zero, (S_STACKFRAME + FRAME_RECORD_RA)(sp) /* stackframe.record.ra = 0 */
+#ifdef CONFIG_RISCV_M_MODE
+ li t0, SR_MPP
+ and t0, s1, t0
@@ arch/riscv/kernel/entry.S: SYM_CODE_START_LOCAL(handle_kernel_stack_overflow)
+ * pt_regs boundary and the unwinder can resume from the pre-overflow
+ * frame pointer saved in PT_S0.
+ */
-+ REG_S zero, (S_STACKFRAME + STACKFRAME_FP)(sp)
-+ REG_S zero, (S_STACKFRAME + STACKFRAME_RA)(sp)
++ REG_S zero, (S_STACKFRAME + FRAME_RECORD_FP)(sp)
++ REG_S zero, (S_STACKFRAME + FRAME_RECORD_RA)(sp)
+ li t0, FRAME_META_TYPE_PT_REGS
+ REG_S t0, S_STACKFRAME_TYPE(sp)
+ addi s0, sp, S_STACKFRAME + STACKFRAME_RECORD_SIZE
@@ arch/riscv/kernel/entry.S: ASM_NOKPROBE(handle_kernel_stack_overflow)
move a2, sp /* pt_regs */
call ret_from_fork_kernel
j ret_from_exception
-@@ arch/riscv/kernel/entry.S: SYM_FUNC_START(call_on_irq_stack)
- addi sp, sp, -STACKFRAME_SIZE_ON_STACK
- REG_S ra, STACKFRAME_RA(sp)
- REG_S s0, STACKFRAME_FP(sp)
-- addi s0, sp, STACKFRAME_SIZE_ON_STACK
-+ addi s0, sp, STACKFRAME_RECORD_SIZE
-
- /* Switch to the per-CPU shadow call stack */
- scs_save_current
-@@ arch/riscv/kernel/entry.S: SYM_FUNC_START(call_on_irq_stack)
- scs_load_current
-
- /* Switch back to the thread stack and restore ra and s0 */
-- addi sp, s0, -STACKFRAME_SIZE_ON_STACK
-+ addi sp, s0, -STACKFRAME_RECORD_SIZE
- REG_L ra, STACKFRAME_RA(sp)
- REG_L s0, STACKFRAME_FP(sp)
- addi sp, sp, STACKFRAME_SIZE_ON_STACK
## arch/riscv/kernel/head.S ##
@@
@@ arch/riscv/kernel/head.S: SYM_CODE_START(_start_kernel)
+ * fp/s0 points above the metadata record (RISC-V
+ * convention).
+ */
-+ REG_S zero, (S_STACKFRAME + STACKFRAME_FP)(sp)
-+ REG_S zero, (S_STACKFRAME + STACKFRAME_RA)(sp)
++ REG_S zero, (S_STACKFRAME + FRAME_RECORD_FP)(sp)
++ REG_S zero, (S_STACKFRAME + FRAME_RECORD_RA)(sp)
+ li t0, FRAME_META_TYPE_FINAL
+ REG_S t0, S_STACKFRAME_TYPE(sp)
+ addi s0, sp, S_STACKFRAME + STACKFRAME_RECORD_SIZE
2: 237864b66d78 = 2: c63daa8759a6 riscv: stacktrace: disable KASAN and KCOV instrumentation for stacktrace.o
3: e6035966a35a = 3: 352269306907 riscv: ftrace: always preserve s0 in dynamic ftrace register frame
4: d132087ea01e ! 4: 2cc86f5728c1 riscv: stacktrace: introduce stack-bound tracking helpers
@@ arch/riscv/include/asm/stacktrace.h: extern void notrace walk_stackframe(struct
+/*
+ * IRQ stack accessors
+ */
++#ifdef CONFIG_IRQ_STACKS
+static inline struct stack_info stackinfo_get_irq(void)
+{
+ unsigned long low = (unsigned long)raw_cpu_read(irq_stack_ptr);
@@ arch/riscv/include/asm/stacktrace.h: extern void notrace walk_stackframe(struct
+
+ return stackinfo_on_stack(&info, sp, size);
+}
++#endif /* CONFIG_IRQ_STACKS */
+
+/*
+ * Task stack accessors
5: 02adea3ece82 ! 5: 57c5e7d08cf8 riscv: stacktrace: switch to frame-pointer based unwinder
@@ arch/riscv/kernel/stacktrace.c: unsigned long __get_wchan(struct task_struct *ta
+ struct task_struct *tsk = task ?: current;
+ struct stack_info stacks[] = {
+ stackinfo_get_task(tsk),
++#ifdef CONFIG_IRQ_STACKS
+ STACKINFO_CPU(tsk, irq),
++#endif
+#ifdef CONFIG_VMAP_STACK
+ STACKINFO_CPU(tsk, overflow),
+#endif
6: c7d7dbe7a8a1 = 6: 9b3379fe9320 riscv: Kconfig: enable HAVE_RELIABLE_STACKTRACE and HAVE_LIVEPATCH
7: ae94a234b34a < -: ------------ selftests/livepatch: Add RISC-V syscall wrapper prefix
-: ------------ > 7: 07dddd25aae1 selftests/livepatch: Add RISC-V syscall wrapper prefixbase-commit: ff492e321c477f0c8899ac2b875635a84e928d78
Signature · 1 line
2.43.0
PATCH v5 1/7 riscv: stacktrace: Add frame record metadata
Reliable frame-pointer unwinding needs an explicit way to identify exception boundaries and the final entry frame. The existing unwinder infers those boundaries from return addresses, which is too loose for a future reliable unwinder.
Add a small metadata frame record to pt_regs and initialize it on exception entry, kernel stack overflow, kernel thread fork, user fork, and early idle task setup. The record uses a zero {fp, ra} sentinel plus a type field so a later unwinder can distinguish a final user-to-kernel boundary from a nested kernel pt_regs boundary.
This follows the arm64 metadata frame-record model, adapted to the RISC-V {fp, ra} frame record convention.
The metadata is established at the RISC-V entry boundaries that need an explicit unwind marker:
* exception entry clears the metadata {fp, ra} pair and uses SPP (or MPP in M-mode) to record whether the pt_regs frame is the final user-to-kernel boundary or a nested kernel boundary;
* the kernel stack overflow path builds a nested pt_regs metadata record on the overflow stack so an unwinder can resume from the pre-overflow s0 saved in PT_S0;
* _start_kernel builds the init task's final metadata record, while the secondary CPU path sets up s0 before smp_callin() so idle-task unwinding does not inherit an undefined caller frame;
* copy_thread creates matching final metadata records for new kernel and user tasks, and keeps s0 available for the frame-pointer chain.
Keep the embedded metadata-record field offsets distinct from the s0-relative STACKFRAME_* offsets used by call_on_irq_stack(), because the latter describe a frame record relative to s0 rather than to the record base.
These changes keep s0 reserved for the frame-pointer chain at task and exception boundaries.
--- arch/riscv/include/asm/ptrace.h | 9 ++++ arch/riscv/include/asm/stacktrace/frame.h | 53 +++++++++++++++++++++++ arch/riscv/kernel/asm-offsets.c | 6 +++ arch/riscv/kernel/entry.S | 39 ++++++++++++++++- arch/riscv/kernel/head.S | 23 ++++++++++ arch/riscv/kernel/process.c | 33 +++++++++++++- 6 files changed, 159 insertions(+), 4 deletions(-) create mode 100644 arch/riscv/include/asm/stacktrace/frame.h
arch/riscv/include/asm/ptrace.h +9 -0
@@ -8,6 +8,7 @@#include <uapi/asm/ptrace.h> #include <asm/csr.h> +#include <asm/stacktrace/frame.h> #include <linux/compiler.h> #ifndef __ASSEMBLER__
@@ -53,6 +54,14 @@ struct pt_regs {unsigned long cause;/* a0 value before the syscall */ unsigned long orig_a0; + +/* +* This frame record is entirely zeroed on exception entry, allowing the +* unwinder to identify exception boundaries. The type field encodes +* whether the exception was taken from user (FINAL) or kernel (PT_REGS) +* mode. +*/ +struct frame_record_meta stackframe; };#define PTRACE_SYSEMU 0x1f
arch/riscv/include/asm/stacktrace/frame.h new file +53 -0
53 lines not on this page · raw message
arch/riscv/kernel/asm-offsets.c +6 -0
@@ -131,6 +131,9 @@ void asm_offsets(void)OFFSET (PT_BADADDR , pt_regs, badaddr);OFFSET (PT_CAUSE , pt_regs, cause); +DEFINE (S_STACKFRAME , offsetof(struct pt_regs , stackframe)); + DEFINE(S_STACKFRAME_TYPE, offsetof(struct pt_regs, stackframe.type)); +OFFSET (SUSPEND_CONTEXT_REGS , suspend_context, regs);OFFSET (HIBERN_PBE_ADDR , pbe, address);
@@ -503,6 +506,9 @@ void asm_offsets(void)DEFINE (STACKFRAME_SIZE_ON_STACK ,ALIGN (sizeof (struct stackframe ),STACK_ALIGN ));DEFINE (STACKFRAME_FP , offsetof(struct stackframe , fp) -sizeof (struct stackframe ));DEFINE (STACKFRAME_RA , offsetof(struct stackframe , ra) -sizeof (struct stackframe )); +DEFINE (STACKFRAME_RECORD_SIZE ,sizeof (struct frame_record )); +OFFSET (FRAME_RECORD_FP , frame_record, fp); +OFFSET (FRAME_RECORD_RA , frame_record, ra);#ifdef CONFIG_FUNCTION_TRACER DEFINE (FTRACE_OPS_FUNC , offsetof(struct ftrace_ops , func));#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
arch/riscv/kernel/entry.S +37 -2
@@ -11,6 +11,7 @@ #include <asm/asm.h> #include <asm/csr.h> #include <asm/scs.h> +#include <asm/stacktrace/frame.h> #include <asm/unistd.h> #include <asm/page.h> #include <asm/thread_info.h>
@@ -198,6 +199,27 @@ SYM_CODE_START(handle_exception) REG_S s4, PT_CAUSE(sp) REG_S s5, PT_TP(sp) + /* + * Create a metadata frame record. The unwinder will use this to + * identify and unwind exception boundaries. + */ + REG_S zero, (S_STACKFRAME + FRAME_RECORD_FP)(sp) /* stackframe.record.fp = 0 */ + REG_S zero, (S_STACKFRAME + FRAME_RECORD_RA)(sp) /* stackframe.record.ra = 0 */ +#ifdef CONFIG_RISCV_M_MODE + li t0, SR_MPP + and t0, s1, t0 +#else + andi t0, s1, SR_SPP +#endif + bnez t0, 1f + li t0, FRAME_META_TYPE_FINAL + j 2f +1: + li t0, FRAME_META_TYPE_PT_REGS +2: + REG_S t0, S_STACKFRAME_TYPE(sp) + addi s0, sp, S_STACKFRAME + STACKFRAME_RECORD_SIZE + /* * Set the scratch register to 0, so that if a recursive exception * occurs, the exception vector knows it came from the kernel
@@ -354,6 +376,19 @@ SYM_CODE_START_LOCAL(handle_kernel_stack_overflow) REG_S s3, PT_BADADDR(sp) REG_S s4, PT_CAUSE(sp) REG_S s5, PT_TP(sp) + + /* + * Create a metadata frame record for the overflow pt_regs. The + * overflow path is entered from kernel context, so this is a nested + * pt_regs boundary and the unwinder can resume from the pre-overflow + * frame pointer saved in PT_S0. + */ + REG_S zero, (S_STACKFRAME + FRAME_RECORD_FP)(sp) + REG_S zero, (S_STACKFRAME + FRAME_RECORD_RA)(sp) + li t0, FRAME_META_TYPE_PT_REGS + REG_S t0, S_STACKFRAME_TYPE(sp) + addi s0, sp, S_STACKFRAME + STACKFRAME_RECORD_SIZE + move a0, sp tail handle_bad_stack SYM_CODE_END(handle_kernel_stack_overflow)
@@ -362,8 +397,8 @@ ASM_NOKPROBE(handle_kernel_stack_overflow) SYM_CODE_START(ret_from_fork_kernel_asm) call schedule_tail - move a0, s1 /* fn_arg */ - move a1, s0 /* fn */ + move a0, s3 /* fn_arg */ + move a1, s2 /* fn */ move a2, sp /* pt_regs */ call ret_from_fork_kernel j ret_from_exception
arch/riscv/kernel/head.S +23 -0
@@ -14,6 +14,7 @@ #include <asm/hwcap.h> #include <asm/image.h> #include <asm/scs.h> +#include <asm/stacktrace/frame.h> #include <asm/usercfi.h> #include "efi-header.S"
@@ -177,6 +178,14 @@ secondary_start_sbi: REG_S a0, (a1) 1: #endif + + /* + * Set up the frame pointer for the secondary idle task so reliable + * stack unwinding terminates at the metadata frame in task_pt_regs(). + * Without this, the first frame records can inherit an undefined caller + * fp and unwind past smp_callin() into .Lsecondary_park. + */ + addi s0, sp, S_STACKFRAME + STACKFRAME_RECORD_SIZE scs_load_current call smp_callin #endif /* CONFIG_SMP */
@@ -305,6 +314,20 @@ SYM_CODE_START(_start_kernel) la tp, init_task la sp, init_thread_union + THREAD_SIZE addi sp, sp, -PT_SIZE_ON_STACK + + /* + * Set up a metadata frame record for the init task so that + * the unwinder can identify the outermost frame by its + * {fp, ra} = {0, 0} sentinel at the bottom of pt_regs. + * fp/s0 points above the metadata record (RISC-V + * convention). + */ + REG_S zero, (S_STACKFRAME + FRAME_RECORD_FP)(sp) + REG_S zero, (S_STACKFRAME + FRAME_RECORD_RA)(sp) + li t0, FRAME_META_TYPE_FINAL + REG_S t0, S_STACKFRAME_TYPE(sp) + addi s0, sp, S_STACKFRAME + STACKFRAME_RECORD_SIZE + #if defined(CONFIG_RISCV_SBI) && defined(CONFIG_RISCV_USER_CFI) li a7, SBI_EXT_FWFT li a6, SBI_EXT_FWFT_SET
arch/riscv/kernel/process.c +31 -2
@@ -258,8 +258,23 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)/* Supervisor/Machine, irqs on: */ childregs->status =SR_PP |SR_PIE ; - p->thread.s[0 ] = (unsigned long )args->fn; - p->thread.s[1 ] = (unsigned long )args->fn_arg; +/* +* Set up a metadata frame record at the bottom of the +* stack for the unwinder. Use FRAME_META_TYPE_FINAL +* since this is the outermost kernel entry for the new +* task. The frame_record::{fp,ra} are already zero from +* memset(). +* +* fp/s0 points above the metadata record (RISC-V +* convention). fn and fn_arg are passed via s2/s3, +* keeping s0 available for the frame pointer chain. +*/ + childregs->stackframe.type =FRAME_META_TYPE_FINAL ; + + p->thread.s[0 ] = (unsigned long )(&childregs->stackframe) + +sizeof (struct frame_record ); + p->thread.s[2 ] = (unsigned long )args->fn; + p->thread.s[3 ] = (unsigned long )args->fn_arg; p->thread.ra = (unsigned long )ret_from_fork_kernel_asm; }else {/* allocate new shadow stack if needed. In case of CLONE_VM we have to */
@@ -278,6 +293,20 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)if (clone_flags &CLONE_SETTLS ) childregs->tp = tls; childregs->a0 =0 ;/* Return value of fork() */ + +/* +* Set up the unwind boundary: ensure the metadata +* frame record has its {fp,ra} sentinel zeroed and +* point fp/s0 above the metadata record. Mark it as +* FINAL since this is the outermost kernel entry for +* the new task. +*/ + childregs->stackframe.record.fp =0 ; + childregs->stackframe.record.ra =0 ; + childregs->stackframe.type =FRAME_META_TYPE_FINAL ; + p->thread.s[0 ] = (unsigned long )(&childregs->stackframe) + +sizeof (struct frame_record ); + p->thread.ra = (unsigned long )ret_from_fork_user_asm; } p->thread.riscv_v_flags =0 ;
Signature · 1 line
2.43.0
PATCH v5 2/7 riscv: stacktrace: disable KASAN and KCOV instrumentation for stacktrace.o
In reply to Wang HanKASAN records stack traces for every alloc/free, which means it walks the unwinder very frequently. Instrumenting the stack trace collection code itself adds substantial overhead and makes the traces themselves noisier.
KCOV instruments every basic-block edge. The unwinder is a hot path, especially with KASAN enabled, so KCOV instrumentation has the same kind of cost and noise problem here.
Mark stacktrace.o as not KASAN- or KCOV-instrumented, matching the x86 treatment of its stack unwinding code. RISC-V keeps the relevant unwinder code in stacktrace.o, so a single translation-unit annotation covers the equivalent scope. This is a prerequisite preference for the upcoming reliable unwinder, but the change is valid on its own.
--- arch/riscv/kernel/Makefile | 6 ++++++ 1 file changed, 6 insertions(+)
arch/riscv/kernel/Makefile +6 -0
@@ -44,6 +44,12 @@ CFLAGS_REMOVE_return_address.o = $(CC_FLAGS_FTRACE) CFLAGS_REMOVE_sbi_ecall.o = $(CC_FLAGS_FTRACE) endif +# When KASAN is enabled, a stack trace is recorded for every alloc/free, which +# can significantly impact performance. Avoid instrumenting the stack trace +# collection code to minimize this impact. +KASAN_SANITIZE_stacktrace.o := n +KCOV_INSTRUMENT_stacktrace.o := n + always-$(KBUILD_BUILTIN) += vmlinux.lds obj-y += head.o
Signature · 1 line
2.43.0
PATCH v5 3/7 riscv: ftrace: always preserve s0 in dynamic ftrace register frame
In reply to Wang Hanstruct __arch_ftrace_regs declares s0 unconditionally, and both ftrace_regs_get_frame_pointer() and ftrace_partial_regs() read it unconditionally. But the SAVE_ABI_REGS / RESTORE_ABI_REGS macros in mcount-dyn.S only stored s0 under HAVE_FUNCTION_GRAPH_FP_TEST (CONFIG_FUNCTION_GRAPH_TRACER && CONFIG_FRAME_POINTER). With CONFIG_FRAME_POINTER=n the slot held whatever was on the stack before, so any callback going through ftrace_partial_regs() saw a garbage regs->s0. RISC-V kernels default to FRAME_POINTER=y, which is why this has not bitten in practice.
Save and restore s0 unconditionally in the dynamic ftrace ABI register frame. This fixes the latent garbage-s0 case, brings the dynamic ftrace path in line with the static _mcount path (mcount.S SAVE_ABI_STATE already saves s0 unconditionally), and matches the frame layout already documented in the comment above SAVE_ABI_REGS. It is also a prerequisite for the upcoming reliable unwinder, which reads ftrace_regs_get_frame_pointer(fregs) directly.
The cost is one extra REG_S/REG_L pair per traced call, negligible compared to the overall ftrace cost; the existing FREGS_SIZE_ON_STACK already reserved the slot, so no extra stack space is used.
--- arch/riscv/kernel/mcount-dyn.S | 4 ---- 1 file changed, 4 deletions(-)
arch/riscv/kernel/mcount-dyn.S +0 -4
@@ -85,9 +85,7 @@ addi sp, sp, -FREGS_SIZE_ON_STACK REG_S t0, FREGS_EPC(sp) REG_S x1, FREGS_RA(sp) -#ifdef HAVE_FUNCTION_GRAPH_FP_TEST REG_S x8, FREGS_S0(sp) -#endif REG_S x6, FREGS_T1(sp) #ifdef CONFIG_CC_IS_CLANG REG_S x7, FREGS_T2(sp)
@@ -113,9 +111,7 @@ .macro RESTORE_ABI_REGS REG_L t0, FREGS_EPC(sp) REG_L x1, FREGS_RA(sp) -#ifdef HAVE_FUNCTION_GRAPH_FP_TEST REG_L x8, FREGS_S0(sp) -#endif REG_L x6, FREGS_T1(sp) #ifdef CONFIG_CC_IS_CLANG REG_L x7, FREGS_T2(sp)
Signature · 1 line
2.43.0
PATCH v5 4/7 riscv: stacktrace: introduce stack-bound tracking helpers
In reply to Wang HanA reliable unwinder needs to validate that every frame record it reads is fully contained in a known kernel stack, and it needs to refuse to walk back into a stack it has already left. Add the building blocks for that:
* struct stack_info / struct unwind_state in a new asm/stacktrace/common.h, modelled on the arm64 reference implementation.
* stackinfo_get_irq() / stackinfo_get_task() / stackinfo_get_overflow() plus the corresponding on_*_stack() predicates in asm/stacktrace.h, so callers can ask "is this object on stack X?" by stack kind rather than open-coded address arithmetic.
* unwind_init_common(), unwind_find_stack() and unwind_consume_stack() helpers that enforce the forward-progress-only invariant required for reliability.
No existing user is wired up to these helpers in this commit; the unwinder switch comes in a follow-up. The header changes leave on_thread_stack() with the same semantics as before, just expressed in terms of the new helpers.
--- arch/riscv/include/asm/stacktrace.h | 67 ++++++++- arch/riscv/include/asm/stacktrace/common.h | 159 +++++++++++++++++++++ 2 files changed, 224 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/stacktrace/common.h
arch/riscv/include/asm/stacktrace.h +65 -2
@@ -3,8 +3,13 @@ #ifndef _ASM_RISCV_STACKTRACE_H #define _ASM_RISCV_STACKTRACE_H +#include <linux/percpu.h> #include <linux/sched.h> +#include <linux/sched/task_stack.h> + +#include <asm/irq_stack.h> #include <asm/ptrace.h> +#include <asm/stacktrace/common.h> struct stackframe { unsigned long fp;
@@ -16,14 +21,72 @@ extern void notrace walk_stackframe(struct task_struct *task, struct pt_regs *reextern void dump_backtrace (struct pt_regs *regs,struct task_struct *task,const char *loglvl); -static inline bool on_thread_stack (void ) +/* +* IRQ stack accessors +*/ +#ifdef CONFIG_IRQ_STACKS +static inline struct stack_info stackinfo_get_irq (void ) +{ +unsigned long low = (unsigned long )raw_cpu_read (irq_stack_ptr); +unsigned long high = low +IRQ_STACK_SIZE ; + +return (struct stack_info ) { + .low = low, + .high = high, + }; +} + +static inline bool on_irq_stack (unsigned long sp,unsigned long size) +{ +struct stack_info info =stackinfo_get_irq (); + +return stackinfo_on_stack (&info, sp, size); +} +#endif /* CONFIG_IRQ_STACKS */ + +/* +* Task stack accessors +*/ +static inline struct stack_info stackinfo_get_task (const struct task_struct *tsk) { -return !(((unsigned long )(current->stack) ^ current_stack_pointer) & ~(THREAD_SIZE -1 )); +unsigned long low = (unsigned long )task_stack_page (tsk); +unsigned long high = low +THREAD_SIZE ; + +return (struct stack_info ) { + .low = low, + .high = high, + }; +} + +static inline bool on_task_stack (const struct task_struct *tsk, +unsigned long sp,unsigned long size) +{ +struct stack_info info =stackinfo_get_task (tsk); + +return stackinfo_on_stack (&info, sp, size); } +/* +* Cast is necessary since current->stack is an opaque ptr. +*/ +#define on_thread_stack () (on_task_stack(current, current_stack_pointer, 1)) +/* +* Overflow stack accessors +*/ #ifdef CONFIG_VMAP_STACK DECLARE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack); + +static inline struct stack_info stackinfo_get_overflow (void ) +{ +unsigned long low = (unsigned long )raw_cpu_ptr (overflow_stack); +unsigned long high = low +OVERFLOW_STACK_SIZE ; + +return (struct stack_info ) { + .low = low, + .high = high, + }; +}#endif /* CONFIG_VMAP_STACK */ #endif /* _ASM_RISCV_STACKTRACE_H */
arch/riscv/include/asm/stacktrace/common.h new file +159 -0
159 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v5 5/7 riscv: stacktrace: switch to frame-pointer based unwinder
In reply to Wang HanReplace the open-coded frame-pointer walker in arch_stack_walk() with a robust kunwind state machine, modelled on arch/arm64/kernel/stacktrace.c and retargeted to the RISC-V {fp, ra} frame record convention. The new walker tracks stack bounds, consumes frame records monotonically, understands the metadata pt_regs records added in the previous frame record metadata patch, and recovers return addresses replaced by function graph tracing and kretprobes.
This commit introduces arch_stack_walk_reliable() but does not yet select HAVE_RELIABLE_STACKTRACE; that is done in a follow-up Kconfig patch so this commit can be reviewed and bisected as a pure unwinder replacement. Until that Kconfig change lands, livepatch is not yet enabled and arch_stack_walk_reliable() has no in-tree caller.
Three related callers are updated to keep the same frame-record assumptions everywhere:
* Function graph tracing: the old RISC-V unwinder matched function graph return-stack entries by the saved return-address slot. That was consistent with the static mcount path, but not with the dynamic ftrace path where the parent slot is ftrace_regs::ra. Use the architectural frame pointer as the function graph return-address cookie, matching the kunwind walker.
* Perf callchains: route kernel callchain collection through arch_stack_walk() so perf sees the same frame-pointer unwind behaviour as dump_stack() and the upcoming livepatch path.
* dump_backtrace() / __get_wchan() / show_stack(): these now go through arch_stack_walk(); the explicit "Call Trace:" header is moved into dump_backtrace() to preserve the original output.
The non-frame-pointer fallback walker is kept untouched for !CONFIG_FRAME_POINTER builds.
--- arch/riscv/kernel/ftrace.c | 6 +- arch/riscv/kernel/perf_callchain.c | 2 +- arch/riscv/kernel/stacktrace.c | 561 ++++++++++++++++++++++++----- 3 files changed, 473 insertions(+), 96 deletions(-)
arch/riscv/kernel/ftrace.c +4 -2
@@ -242,7 +242,8 @@ void prepare_ftrace_return(unsigned long *parent, unsigned long self_addr,*/ old = *parent; -if (!function_graph_enter (old, self_addr, frame_pointer, parent)) +if (!function_graph_enter (old, self_addr, frame_pointer, + (void *)frame_pointer)) *parent = return_hooker; }
@@ -264,7 +265,8 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,*/ old = *parent; -if (!function_graph_enter_regs (old, ip, frame_pointer, parent, fregs)) +if (!function_graph_enter_regs (old, ip, frame_pointer, + (void *)frame_pointer, fregs)) *parent = return_hooker; }#endif /* CONFIG_DYNAMIC_FTRACE */
arch/riscv/kernel/perf_callchain.c +1 -1
@@ -44,5 +44,5 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry,return ; } -walk_stackframe (NULL , regs, fill_callchain, entry); +arch_stack_walk (fill_callchain, entry,NULL , regs); }
arch/riscv/kernel/stacktrace.c +468 -93
602 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v5 6/7 riscv: Kconfig: enable HAVE_RELIABLE_STACKTRACE and HAVE_LIVEPATCH
In reply to Wang HanNow that the metadata frame records, the kunwind state machine and arch_stack_walk_reliable() are all in place, advertise the capability to the rest of the kernel:
* select HAVE_RELIABLE_STACKTRACE under FRAME_POINTER && 64BIT, so only the configurations with the tested metadata records and FP-based reliable walker enable it.
* select HAVE_LIVEPATCH under the same condition and source kernel/livepatch/Kconfig so the livepatch menu is reachable from the RISC-V configuration.
The 64BIT dependency is conservative scoping rather than a hard technical requirement: the metadata frame record, kunwind state machine and arch_stack_walk_reliable() also build on RV32, and the IRQ-stack frame-record adjustment fixes a latent RV32 issue. However, the syscall livepatch selftest and module relocation path have only been exercised on RV64 QEMU virt so far. The 64BIT gate can be relaxed in a follow-up once RV32 has equivalent coverage.
This is split out from the unwinder change so the policy decision and the implementation can be reviewed and reverted independently.
--- arch/riscv/Kconfig | 4 ++++ 1 file changed, 4 insertions(+)
arch/riscv/Kconfig +4 -0
@@ -186,6 +186,7 @@ config RISCV select HAVE_KRETPROBES # https://github.com/ClangBuiltLinux/linux/issues/1881 select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD + select HAVE_LIVEPATCH if FRAME_POINTER && 64BIT select HAVE_MOVE_PMD select HAVE_MOVE_PUD select HAVE_PAGE_SIZE_4KB
@@ -196,6 +197,7 @@ config RISCV select HAVE_POSIX_CPU_TIMERS_TASK_WORK select HAVE_PREEMPT_DYNAMIC_KEY select HAVE_REGS_AND_STACK_ACCESS_API + select HAVE_RELIABLE_STACKTRACE if FRAME_POINTER && 64BIT select HAVE_RETHOOK select HAVE_RSEQ select HAVE_RUST if RUSTC_SUPPORTS_RISCV && CC_IS_CLANG
@@ -1392,3 +1394,5 @@ endmenu # "CPU Power Management" source "arch/riscv/kvm/Kconfig" source "drivers/acpi/Kconfig" + +source "kernel/livepatch/Kconfig"
Signature · 1 line
2.43.0
PATCH v5 7/7 selftests/livepatch: Add RISC-V syscall wrapper prefix
In reply to Wang HanThe syscall livepatch selftest resolves and patches a syscall wrapper symbol. To use that test for RISC-V livepatch validation, add the RISC-V FN_PREFIX definition for ARCH_HAS_SYSCALL_WRAPPER.
Without this macro, the syscall livepatch selftest cannot resolve the RISC-V target symbol, and the syscall-related livepatch test fails on RISC-V.
--- .../testing/selftests/livepatch/test_modules/test_klp_syscall.c | 2 ++ 1 file changed, 2 insertions(+)
tools/testing/selftests/livepatch/test_modules/test_klp_syscall.c +2 -0
@@ -24,6 +24,8 @@#define FN_PREFIX __s390x_#elif defined(__aarch64__)#define FN_PREFIX __arm64_ +#elif defined(__riscv) +#define FN_PREFIX __riscv_#elif defined(__powerpc__)#define FN_PREFIX #else
Signature · 1 line
2.43.0
Re: [PATCH v5 7/7] selftests/livepatch: Add RISC-V syscall wrapper prefix
Quoted 9 lines · Wang Han
The syscall livepatch selftest resolves and patches a syscall wrapper symbol. To use that test for RISC-V livepatch validation, add the RISC-V
FN_PREFIXdefinition forARCH_HAS_SYSCALL_WRAPPER.Without this macro, the syscall livepatch selftest cannot resolve the RISC-V target symbol, and the syscall-related livepatch test fails on RISC-V.
[...]
Signature · 1 line
Miroslav