Skip to contentSkip to conversation

Add BPF Exceptions support for RISC-V

PATCH In review Varun R Mallya 5 participants 14 messages Jul 22 – Jul 30

Subsystems (6)
Kselftest BPF perf events RISC-V +2 more subsystems
lore
Share & tools
lore
#0 lore raw

v3 cover Add BPF Exceptions support for RISC-V

This patchset aims to add BPF exceptions supports for riscv64 by implementing the arch_bpf_stack_walk function and then updating the prologue and epilogue for BPF JIT on riscv. Also, bpf_jit_supports_exceptions() returns true now so that the verifier does not reject programs containing BPF exceptions on riscv64.

On riscv the unwinder used by arch_bpf_stack_walk() is the frame-pointer unwinder, so exception support is gated on CONFIG_FRAME_POINTER.

In the prologue and epilogue of the RISC-V JIT, I saved the return address and then the frame-pointer according to [1]. Also, according to [2], s0 to s11 are callee saved registers, which is why a new array (rv_exception_csave_regs) has been created to save these registers which contains all the required registers along with the frame pointer as well as return address.

The following demonstrates that all the selftests for BPF exceptions pass:

root@virtme-ng:~/bpf# ./test_progs -t exceptions
#114/1   exceptions/exception_throw_always_1:OK
#114/2   exceptions/exception_throw_always_2:OK
#114/3   exceptions/exception_throw_unwind_1:OK
#114/4   exceptions/exception_throw_unwind_2:OK
#114/5   exceptions/exception_throw_default:OK
#114/6   exceptions/exception_throw_default_value:OK
#114/7   exceptions/exception_tail_call:OK
#114/8   exceptions/exception_ext:OK
#114/9   exceptions/exception_ext_mod_cb_runtime:OK
#114/10  exceptions/exception_throw_subprog:OK
#114/11  exceptions/exception_assert_nz_gfunc:OK
#114/12  exceptions/exception_assert_zero_gfunc:OK
#114/13  exceptions/exception_assert_neg_gfunc:OK
#114/14  exceptions/exception_assert_pos_gfunc:OK
#114/15  exceptions/exception_assert_negeq_gfunc:OK
#114/16  exceptions/exception_assert_poseq_gfunc:OK
#114/17  exceptions/exception_assert_nz_gfunc_with:OK
#114/18  exceptions/exception_assert_zero_gfunc_with:OK
#114/19  exceptions/exception_assert_neg_gfunc_with:OK
#114/20  exceptions/exception_assert_pos_gfunc_with:OK
#114/21  exceptions/exception_assert_negeq_gfunc_with:OK
#114/22  exceptions/exception_assert_poseq_gfunc_with:OK
#114/23  exceptions/exception_bad_assert_nz_gfunc:OK
#114/24  exceptions/exception_bad_assert_zero_gfunc:OK
#114/25  exceptions/exception_bad_assert_neg_gfunc:OK
#114/26  exceptions/exception_bad_assert_pos_gfunc:OK
#114/27  exceptions/exception_bad_assert_negeq_gfunc:OK
#114/28  exceptions/exception_bad_assert_poseq_gfunc:OK
#114/29  exceptions/exception_bad_assert_nz_gfunc_with:OK
#114/30  exceptions/exception_bad_assert_zero_gfunc_with:OK
#114/31  exceptions/exception_bad_assert_neg_gfunc_with:OK
#114/32  exceptions/exception_bad_assert_pos_gfunc_with:OK
#114/33  exceptions/exception_bad_assert_negeq_gfunc_with:OK
#114/34  exceptions/exception_bad_assert_poseq_gfunc_with:OK
#114/35  exceptions/exception_assert_range:OK
#114/36  exceptions/exception_assert_range_with:OK
#114/37  exceptions/exception_bad_assert_range:OK
#114/38  exceptions/exception_bad_assert_range_with:OK
#114/39  exceptions/exception_throw_from_void_global:OK
#114/40  exceptions/non-throwing fentry -> exception_cb:OK
#114/41  exceptions/throwing fentry -> exception_cb:OK
#114/42  exceptions/non-throwing fexit -> exception_cb:OK
#114/43  exceptions/throwing fexit -> exception_cb:OK
#114/44  exceptions/throwing extension (with custom cb) -> exception_cb:OK
#114/45  exceptions/throwing extension -> global func in exception_cb:OK
#114/46  exceptions/exception_ext_mod_cb_runtime:OK
#114/47  exceptions/throwing extension (with custom cb) -> global func in exception_cb:OK
#114/48  exceptions/exception_ext:OK
#114/49  exceptions/non-throwing fentry -> non-throwing subprog:OK
#114/50  exceptions/throwing fentry -> non-throwing subprog:OK
#114/51  exceptions/non-throwing fentry -> throwing subprog:OK
#114/52  exceptions/throwing fentry -> throwing subprog:OK
#114/53  exceptions/non-throwing fexit -> non-throwing subprog:OK
#114/54  exceptions/throwing fexit -> non-throwing subprog:OK
#114/55  exceptions/non-throwing fexit -> throwing subprog:OK
#114/56  exceptions/throwing fexit -> throwing subprog:OK
#114/57  exceptions/non-throwing fmod_ret -> non-throwing subprog:OK
#114/58  exceptions/non-throwing fmod_ret -> non-throwing global subprog:OK
#114/59  exceptions/non-throwing extension -> non-throwing subprog:OK
#114/60  exceptions/non-throwing extension -> throwing subprog:OK
#114/61  exceptions/non-throwing extension -> non-throwing subprog:OK
#114/62  exceptions/non-throwing extension -> throwing global subprog:OK
#114/63  exceptions/throwing extension -> throwing global subprog:OK
#114/64  exceptions/throwing extension -> non-throwing global subprog:OK
#114/65  exceptions/non-throwing extension -> main subprog:OK
#114/66  exceptions/throwing extension -> main subprog:OK
#114/67  exceptions/reject_exception_cb_type_1:OK
#114/68  exceptions/reject_exception_cb_type_2:OK
#114/69  exceptions/reject_exception_cb_type_3:OK
#114/70  exceptions/reject_exception_cb_type_4:OK
#114/71  exceptions/reject_exception_cb_type_5:OK
#114/72  exceptions/reject_async_callback_throw:OK
#114/73  exceptions/reject_with_lock:OK
#114/74  exceptions/reject_subprog_with_lock:OK
#114/75  exceptions/reject_with_rcu_read_lock:OK
#114/76  exceptions/reject_subprog_with_rcu_read_lock:OK
#114/77  exceptions/reject_with_rbtree_add_throw:OK
#114/78  exceptions/reject_with_reference:OK
#114/79  exceptions/reject_global_subprog_throw_with_reference:OK
#114/80  exceptions/reject_with_cb_reference:OK
#114/81  exceptions/reject_with_cb:OK
#114/82  exceptions/reject_with_subprog_reference:OK
#114/83  exceptions/reject_throwing_exception_cb:OK
#114/84  exceptions/reject_exception_cb_call_global_func:OK
#114/85  exceptions/reject_exception_cb_call_static_func:OK
#114/86  exceptions/reject_multiple_exception_cb:OK
#114/87  exceptions/reject_exception_throw_cb:OK
#114/88  exceptions/reject_exception_throw_cb_diff:OK
#114/89  exceptions/reject_subprog_rcu_lock_throw:OK
#114/90  exceptions/reject_subprog_throw_preempt_lock:OK
#114/91  exceptions/reject_subprog_throw_irq_lock:OK
#114/92  exceptions/reject_set_exception_cb_bad_ret1:OK
#114/93  exceptions/reject_set_exception_cb_bad_ret2:OK
#114/94  exceptions/reject_out_of_range_global_throw:OK
#114/95  exceptions/check_assert_eq_int_min:OK
#114/96  exceptions/check_assert_eq_int_max:OK
#114/97  exceptions/check_assert_eq_zero:OK
#114/98  exceptions/check_assert_eq_llong_min:OK
#114/99  exceptions/check_assert_eq_llong_max:OK
#114/100 exceptions/check_assert_lt_pos:OK
#114/101 exceptions/check_assert_lt_zero:OK
#114/102 exceptions/check_assert_lt_neg:OK
#114/103 exceptions/check_assert_le_pos:OK
#114/104 exceptions/check_assert_le_zero:OK
#114/105 exceptions/check_assert_le_neg:OK
#114/106 exceptions/check_assert_gt_pos:OK
#114/107 exceptions/check_assert_gt_zero:OK
#114/108 exceptions/check_assert_gt_neg:OK
#114/109 exceptions/check_assert_ge_pos:OK
#114/110 exceptions/check_assert_ge_zero:OK
#114/111 exceptions/check_assert_ge_neg:OK
#114/112 exceptions/check_assert_range_s64:OK
#114/113 exceptions/check_assert_range_u64:OK
#114/114 exceptions/check_assert_single_range_s64:OK
#114/115 exceptions/check_assert_single_range_u64:OK
#114/116 exceptions/check_assert_generic:OK
#114/117 exceptions/check_assert_with_return:OK
#114     exceptions:OK
Summary: 1/117 PASSED, 0 SKIPPED, 0 FAILED

Since all exception tests pass now, the denylist file has been removed as it is empty now.

Changes in v1->v2:
v1: https://lore.kernel.org/bpf/[email protected]/
- resolve the ftrace trampoline address with ftrace_graph_ret_addr() inside arch_bpf_stack_walk()
- Remove duplicated code in bpf_jit_build_prologue()

Changes in v2->v3:
v2: https://lore.kernel.org/bpf/[email protected]/
- extend walk_stackframe to use sp/fp similar to how arm64 does it. (Björn Töpel)
- refactor prologue/epilogue into helpers. (Björn Töpel)
- fix comments to make them netdev style.
- save/restore the tail-call counter in the exception prologue/epilogue
- exception_tail_call now passes since bpf2bpf+tailcall mixing landed [3], so drop the (now-empty) DENYLIST.riscv64
- (!aux->exception_cb) has been simplified to only one condition (Pu Lehui)

[1]: https://riscv-non-isa.github.io/riscv-elf-psabi-doc/#_frame_pointer_convention
[2]: https://riscv-non-isa.github.io/riscv-elf-psabi-doc/#_integer_register_convention
[3]: https://lore.kernel.org/bpf/[email protected]

Varun R Mallya (6):
riscv: Use arch_stack_walk() for the kernel callchain riscv, stacktrace: Pass sp and fp to the walk_stackframe() callback riscv, stacktrace: Implement arch_bpf_stack_walk() for BPF riscv, bpf: Split prologue and epilogue into helper functions riscv, bpf: Add support for BPF exceptions
riscv, bpf: Remove BPF exceptions from denylist

 arch/riscv/include/asm/stacktrace.h          |   5 +-
 arch/riscv/kernel/perf_callchain.c           |   3 +-
 arch/riscv/kernel/stacktrace.c               |  64 ++++-
 arch/riscv/net/bpf_jit_comp64.c              | 254 ++++++++++++++-----
 tools/testing/selftests/bpf/DENYLIST.riscv64 |   2 -
 5 files changed, 253 insertions(+), 75 deletions(-)
 delete mode 100644 tools/testing/selftests/bpf/DENYLIST.riscv64
Signature · 1 line
2.55.0
#1 lore raw

PATCH v3 1/6 riscv: Use arch_stack_walk() for the kernel callchain

Replace walk_stackframe() with generic arch_stack_walk() which already wraps walk_stackframe() and is the generic mechanism used in archs like arm64 to build the kernel callchain.

Also decouples perf from the internal walk_stackframe unwinder which is modified in later commits to accept a different function signature, which warranted this change.

---
 arch/riscv/kernel/perf_callchain.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/riscv/kernel/perf_callchain.c +2 -1
index b465bc9eb870..93a12d53aa26 100644
@@ -2,6 +2,7 @@
 /* Copyright (C) 2019 Hangzhou C-SKY Microsystems co.,ltd. */
 
 #include <linux/perf_event.h>
+#include <linux/stacktrace.h>
 #include <linux/uaccess.h>
 
 #include <asm/stacktrace.h>
@@ -44,5 +45,5 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry,
 		return;
 	}
 
-	walk_stackframe(NULL, regs, fill_callchain, entry);
+	arch_stack_walk(fill_callchain, entry, current, regs);
 }
Signature · 1 line
2.55.0
#2 lore raw

PATCH v3 2/6 riscv, stacktrace: Pass sp and fp to the walk_stackframe() callback

In reply to Varun R Mallya

Extend the walk_stackframe() consume callback so it receives the stack pointer and frame pointer of each frame, in addition to the pc. The extra state is needed by the BPF exceptions machinery. bpf_throw() unwinds the kernel stack looking for the BPF program that acts as the exception boundary and needs (pc, stack and frame pointers) of that frame to invoke the exception callback in its stack context.

This mirrors what arm64 does, where kunwind_stack_walk() hands the full unwind state to its consume functions and arch_stack_walk() adapts the generic stack_trace_consume_fn via a small wrapper (kunwind_consume_entry_data), so that a single unwinder serves the generic stacktrace API and BPF alike.

The pc-only callbacks print_trace_address() and save_wchan() simply gain ignored arguments, and arch_stack_walk() now adapts the generic stack_trace_consume_fn with a wrapper struct. In the !CONFIG_FRAME_POINTER variant there is no frame pointer to report, so pass the scan position as sp and 0 as fp.

---
 arch/riscv/include/asm/stacktrace.h |  5 +++-
 arch/riscv/kernel/stacktrace.c      | 39 +++++++++++++++++++++++------
 2 files changed, 35 insertions(+), 9 deletions(-)
arch/riscv/include/asm/stacktrace.h +4 -1
index b1495a7e06ce..fd1ff04a040e 100644
@@ -11,8 +11,11 @@ struct stackframe {
 	unsigned long ra;
 };
 
+typedef bool (*walk_stackframe_fn)(void *arg, unsigned long pc,
+				   unsigned long sp, unsigned long fp);
+
 extern void notrace walk_stackframe(struct task_struct *task, struct pt_regs *regs,
-				    bool (*fn)(void *, unsigned long), void *arg);
+				    walk_stackframe_fn fn, void *arg);
 extern void dump_backtrace(struct pt_regs *regs, struct task_struct *task,
 			   const char *loglvl);
 
arch/riscv/kernel/stacktrace.c +31 -8
index c7555447149b..30b369dd2815 100644
@@ -46,7 +46,7 @@ static inline int fp_is_valid(unsigned long fp, unsigned long sp)
 }
 
 void notrace walk_stackframe(struct task_struct *task, struct pt_regs *regs,
-			     bool (*fn)(void *, unsigned long), void *arg)
+			     walk_stackframe_fn fn, void *arg)
 {
 	unsigned long fp, sp, pc;
 	int graph_idx = 0;
@@ -71,7 +71,9 @@ void notrace walk_stackframe(struct task_struct *task, struct pt_regs *regs,
 	for (;;) {
 		struct stackframe *frame;
 
-		if (unlikely(!__kernel_text_address(pc) || (level++ >= 0 && !fn(arg, pc))))
+		/* pc belongs to the function whose frame pointer is fp */
+		if (unlikely(!__kernel_text_address(pc) ||
+			     (level++ >= 0 && !fn(arg, pc, sp, fp))))
 			break;
 
 		if (unlikely(!fp_is_valid(fp, sp)))
@@ -91,7 +93,7 @@ void notrace walk_stackframe(struct task_struct *task, struct pt_regs *regs,
 						   &frame->ra);
 			if (pc >= (unsigned long)handle_exception &&
 			    pc < (unsigned long)&ret_from_exception_end) {
-				if (unlikely(!fn(arg, pc)))
+				if (unlikely(!fn(arg, pc, sp, fp)))
 					break;
 
 				pc = ((struct pt_regs *)sp)->epc;
@@ -105,7 +107,7 @@ void notrace walk_stackframe(struct task_struct *task, struct pt_regs *regs,
 #else /* !CONFIG_FRAME_POINTER */
 
 void notrace walk_stackframe(struct task_struct *task,
-	struct pt_regs *regs, bool (*fn)(void *, unsigned long), void *arg)
+	struct pt_regs *regs, walk_stackframe_fn fn, void *arg)
 {
 	unsigned long sp, pc;
 	unsigned long *ksp;
@@ -127,7 +129,8 @@ void notrace walk_stackframe(struct task_struct *task,
 
 	ksp = (unsigned long *)sp;
 	while (!kstack_end(ksp)) {
-		if (__kernel_text_address(pc) && unlikely(!fn(arg, pc)))
+		if (__kernel_text_address(pc) &&
+		    unlikely(!fn(arg, pc, (unsigned long)ksp, 0)))
 			break;
 		pc = READ_ONCE_NOCHECK(*ksp++);
 	}
@@ -135,7 +138,8 @@ void notrace walk_stackframe(struct task_struct *task,
 
 #endif /* CONFIG_FRAME_POINTER */
 
-static bool print_trace_address(void *arg, unsigned long pc)
+static bool print_trace_address(void *arg, unsigned long pc,
+				unsigned long sp, unsigned long fp)
 {
 	const char *loglvl = arg;
 
@@ -155,7 +159,8 @@ void show_stack(struct task_struct *task, unsigned long *sp, const char *loglvl)
 	dump_backtrace(NULL, task, loglvl);
 }
 
-static bool save_wchan(void *arg, unsigned long pc)
+static bool save_wchan(void *arg, unsigned long pc,
+		       unsigned long sp, unsigned long fp)
 {
 	if (!in_sched_functions(pc)) {
 		unsigned long *p = arg;
@@ -176,10 +181,28 @@ unsigned long __get_wchan(struct task_struct *task)
 	return pc;
 }
 
+struct walk_stackframe_consume_entry_data {
+	stack_trace_consume_fn consume_entry;
+	void *cookie;
+};
+
+static bool walk_stackframe_consume_entry(void *arg, unsigned long pc,
+					  unsigned long sp, unsigned long fp)
+{
+	struct walk_stackframe_consume_entry_data *data = arg;
+
+	return data->consume_entry(data->cookie, pc);
+}
+
 noinline noinstr void arch_stack_walk(stack_trace_consume_fn consume_entry, void *cookie,
 		     struct task_struct *task, struct pt_regs *regs)
 {
-	walk_stackframe(task, regs, consume_entry, cookie);
+	struct walk_stackframe_consume_entry_data data = {
+		.consume_entry = consume_entry,
+		.cookie = cookie,
+	};
+
+	walk_stackframe(task, regs, walk_stackframe_consume_entry, &data);
 }
 
 /*
Signature · 1 line
2.55.0
#3 lore raw

Re: [PATCH bpf-next v3 2/6] riscv, stacktrace: Pass sp and fp to the walk_stackframe() callback

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [High] The newly added wrapper callback walk_stackframe_consume_entry lacks the notrace (and noinstr) attribute, leading to ftrace instrumentation escaping the noinstr context.
- [High] The stack pointer (sp) passed to the callback for interrupted functions is stale; it points to the exception frame (pt_regs) rather than the actual stack pointer of the interrupted function.

Pre-existing issues:
- [High] The exception unwinding logic reads from pt_regs without verifying that the large structure fits entirely within the remaining stack bounds, causing an out-of-bounds read. --

--- Patch [2]: [PATCH bpf-next v3 2/6] riscv, stacktrace: Pass sp and fp to the walk_stackframe() callback --- commit 546502e11288943b2cf30f9608406a872a42ffb2
Author: Varun R Mallya <[email protected]>

riscv, stacktrace: Pass sp and fp to the walk_stackframe() callback

This commit updates RISC-V's walk_stackframe() and related callbacks to pass the stack and frame pointers along with the PC. This enables the BPF exceptions machinery to retrieve the required context for unwinding the stack. It introduces walk_stackframe_consume_entry() as a wrapper to adapt the generic stack trace API.

Quoted 4 lines
arch/riscv/kernel/stacktrace.c +0 -0
index c7555447149bf..30b369dd28151 100644

[ ... ]

Quoted 10 lines
arch/riscv/kernel/stacktrace.c +1 -1
@@ -91,7 +93,7 @@ void notrace walk_stackframe(struct task_struct *task, struct pt_regs *regs,
 						   &frame->ra);
 			if (pc >= (unsigned long)handle_exception &&
 			    pc < (unsigned long)&ret_from_exception_end) {
-				if (unlikely(!fn(arg, pc)))
+				if (unlikely(!fn(arg, pc, sp, fp)))
 					break;
 
 				pc = ((struct pt_regs *)sp)->epc;

fp = ((struct pt_regs *)sp)->s0;

[Severity: High]
This is a pre-existing issue, but this exception unwinding logic reads from pt_regs without verifying that the large structure fits entirely within the remaining stack bounds.

If unwinding a concurrently running task, an inconsistent read can result in sp being close to the top of the stack bounds. Because s0 is at a significant offset within pt_regs, could this blind cast and dereference read past the top of the stack into the unmapped guard page and cause a panic?

[Severity: High]
When updating the state for the interrupted frame here, pc and fp are updated, but sp is not. Should sp also be updated to the interrupted function's stack pointer (e.g., sp = ((struct pt_regs *)sp)->sp)?

Because this patch specifically introduces sp and fp to the callback for BPF exceptions, passing the exception frame pointer instead of the interrupted function's sp might corrupt the stack when BPF attempts to restore state.

[ ... ]

Quoted 18 lines
arch/riscv/kernel/stacktrace.c +13 -0
@@ -176,10 +181,28 @@ unsigned long __get_wchan(struct task_struct *task)
 	return pc;
 }
 
+struct walk_stackframe_consume_entry_data {
+	stack_trace_consume_fn consume_entry;
+	void *cookie;
+};
+
+static bool walk_stackframe_consume_entry(void *arg, unsigned long pc,
+					  unsigned long sp, unsigned long fp)
+{
+	struct walk_stackframe_consume_entry_data *data = arg;
+
+	return data->consume_entry(data->cookie, pc);
+}
+
 noinline noinstr void arch_stack_walk(stack_trace_consume_fn consume_entry, void *cookie,

arch/riscv/kernel/stacktrace.c unsigned long __get_wchan(struct task_struct *task) · line 206

Quoted 10 lines
arch/riscv/kernel/stacktrace.c +6 -1
 		     struct task_struct *task, struct pt_regs *regs)
 {
-	walk_stackframe(task, regs, consume_entry, cookie);
+	struct walk_stackframe_consume_entry_data data = {
+		.consume_entry = consume_entry,
+		.cookie = cookie,
+	};
+
+	walk_stackframe(task, regs, walk_stackframe_consume_entry, &data);
 }

[Severity: High]
Does walk_stackframe_consume_entry() need a notrace annotation?

It is invoked directly from the noinstr function arch_stack_walk(). Without notrace, this allows ftrace instrumentation to escape the noinstr context, which can lead to infinite recursion or crashes if triggered from highly restricted contexts like NMIs or exception entries.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#4 lore raw

PATCH v3 3/6 riscv, stacktrace: Implement arch_bpf_stack_walk() for BPF

In reply to Varun R Mallya

This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the exception callback with the stack of the main program.
This is a wrapper around walk_stackframe(), which now passes (pc, sp and fp) to it's consume callback.

---
 arch/riscv/kernel/stacktrace.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
arch/riscv/kernel/stacktrace.c +25 -0
index 30b369dd2815..c0f2393e5163 100644
@@ -5,6 +5,7 @@
  */
 
 #include <linux/export.h>
+#include <linux/filter.h>
 #include <linux/kallsyms.h>
 #include <linux/sched.h>
 #include <linux/sched/debug.h>
@@ -104,6 +105,30 @@ void notrace walk_stackframe(struct task_struct *task, struct pt_regs *regs,
 	}
 }
 
+struct bpf_unwind_consume_entry_data {
+	bool (*consume_entry)(void *cookie, u64 ip, u64 sp, u64 fp);
+	void *cookie;
+};
+
+static bool bpf_unwind_consume_entry(void *arg, unsigned long pc,
+				     unsigned long sp, unsigned long fp)
+{
+	struct bpf_unwind_consume_entry_data *data = arg;
+
+	return data->consume_entry(data->cookie, pc, sp, fp);
+}
+
+void notrace arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp),
+				 void *cookie)
+{
+	struct bpf_unwind_consume_entry_data data = {
+		.consume_entry = consume_fn,
+		.cookie = cookie,
+	};
+
+	walk_stackframe(current, NULL, bpf_unwind_consume_entry, &data);
+}
+
 #else /* !CONFIG_FRAME_POINTER */
 
 void notrace walk_stackframe(struct task_struct *task,
Signature · 1 line
2.55.0
#5 lore raw

Re: [PATCH bpf-next v3 3/6] riscv, stacktrace: Implement arch_bpf_stack_walk() for BPF

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] arch_bpf_stack_walk is incorrectly placed inside the #ifdef CONFIG_FRAME_POINTER block, silently…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] arch_bpf_stack_walk is incorrectly placed inside the #ifdef CONFIG_FRAME_POINTER block, silently breaking BPF stack walking features when frame pointers are disabled. --

--- Patch [3]: [PATCH bpf-next v3 3/6] riscv, stacktrace: Implement arch_bpf_stack_walk() for BPF --- commit 0c8984f224c945ac7ba31a2cde4b91bf4740e71c
Author: Varun R Mallya <[email protected]>

riscv, stacktrace: Implement arch_bpf_stack_walk() for BPF

This patch implements arch_bpf_stack_walk() for RISC-V by wrapping walk_stackframe(). It is intended to be used by bpf_throw() for unwinding, passing the required state to the consume callback.

Quoted 4 lines
arch/riscv/kernel/stacktrace.c +0 -0
index 30b369dd28151..c0f2393e5163c 100644

[ ... ]

Quoted 19 lines
arch/riscv/kernel/stacktrace.c +15 -0
@@ -104,6 +105,30 @@ void notrace walk_stackframe(struct task_struct *task, struct pt_regs *regs,
 	}
 }
 
+struct bpf_unwind_consume_entry_data {
+	bool (*consume_entry)(void *cookie, u64 ip, u64 sp, u64 fp);
+	void *cookie;
+};
+
+static bool bpf_unwind_consume_entry(void *arg, unsigned long pc,
+				     unsigned long sp, unsigned long fp)
+{
+	struct bpf_unwind_consume_entry_data *data = arg;
+
+	return data->consume_entry(data->cookie, pc, sp, fp);
+}
+
+void notrace arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp),
+				 void *cookie)

arch/riscv/kernel/stacktrace.c void notrace walk_stackframe(struct task_struct *task, struct pt_regs *regs, · line 132

Quoted 10 lines
arch/riscv/kernel/stacktrace.c +9 -0
+{
+	struct bpf_unwind_consume_entry_data data = {
+		.consume_entry = consume_fn,
+		.cookie = cookie,
+	};
+
+	walk_stackframe(current, NULL, bpf_unwind_consume_entry, &data);
+}
+
 #else /* !CONFIG_FRAME_POINTER */

[Severity: Medium]
Will arch_bpf_stack_walk() be excluded when CONFIG_FRAME_POINTER is disabled?

It looks like this function is placed inside the #ifdef CONFIG_FRAME_POINTER block. Since walk_stackframe() has a valid implementation for !CONFIG_FRAME_POINTER that does stack-scanning, shouldn't arch_bpf_stack_walk() be defined outside the #ifdef block so BPF stack walking continues to work when frame pointers are disabled?

If this is omitted, the kernel will fall back to the weak empty stub in kernel/bpf/core.c, which causes bpf_prog_find_from_stack() to return NULL and could make BPF diagnostic features silently fail.

Quoted 2 lines
arch/riscv/kernel/stacktrace.c +0 -0
 void notrace walk_stackframe(struct task_struct *task,
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#6 lore raw

PATCH v3 4/6 riscv, bpf: Split prologue and epilogue into helper functions

In reply to Varun R Mallya

Split bpf_jit_build_prologue() and __build_epilogue() into helpers. normal_stack_adjust() computes the frame size for the callee-saved registers a program actually clobbers, emit_normal_prologue() allocates the frame and spills those registers, and emit_normal_restore() reloads them.

The kcfi preamble, fentry nops and tail-call-counter setup are hoisted before the frame-size computation and they do not depend on it and the emitted instruction sequence is unchanged.

No functional change.

---
 arch/riscv/net/bpf_jit_comp64.c | 152 ++++++++++++++++++--------------
 1 file changed, 87 insertions(+), 65 deletions(-)
arch/riscv/net/bpf_jit_comp64.c +87 -65
index ad089a9a4ea9..b804382075c4 100644
@@ -198,9 +198,84 @@ static void emit_imm(u8 rd, s64 val, struct rv_jit_context *ctx)
 		emit_addi(rd, rd, lower, ctx);
 }
 
-static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx)
+/* Stack space for the callee-saved registers a normal program spills,
+ * excluding the BPF stack itself.
+ */
+static int normal_stack_adjust(struct rv_jit_context *ctx)
 {
-	int stack_adjust = ctx->stack_size, store_offset = stack_adjust - 8;
+	int stack_adjust = 0;
+
+	if (seen_reg(RV_REG_RA, ctx))
+		stack_adjust += 8;
+	stack_adjust += 8; /* RV_REG_FP */
+	if (seen_reg(RV_REG_S1, ctx))
+		stack_adjust += 8;
+	if (seen_reg(RV_REG_S2, ctx))
+		stack_adjust += 8;
+	if (seen_reg(RV_REG_S3, ctx))
+		stack_adjust += 8;
+	if (seen_reg(RV_REG_S4, ctx))
+		stack_adjust += 8;
+	if (seen_reg(RV_REG_S5, ctx))
+		stack_adjust += 8;
+	if (ctx->arena_vm_start)
+		stack_adjust += 8;
+	stack_adjust += 8; /* RV_REG_TCC */
+
+	return round_up(stack_adjust, STACK_ALIGN);
+}
+
+/* Allocate the frame and save the callee-saved registers the program
+ * actually clobbers, then point FP at the frame top.
+ */
+static void emit_normal_prologue(struct rv_jit_context *ctx, int stack_adjust)
+{
+	int store_offset = stack_adjust - 8;
+	/* tailcall starts here, emit insn before it must be fixed */
+
+	emit_addi(RV_REG_SP, RV_REG_SP, -stack_adjust, ctx);
+
+	if (seen_reg(RV_REG_RA, ctx)) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_RA, ctx);
+		store_offset -= 8;
+	}
+	emit_sd(RV_REG_SP, store_offset, RV_REG_FP, ctx);
+	store_offset -= 8;
+	if (seen_reg(RV_REG_S1, ctx)) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_S1, ctx);
+		store_offset -= 8;
+	}
+	if (seen_reg(RV_REG_S2, ctx)) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_S2, ctx);
+		store_offset -= 8;
+	}
+	if (seen_reg(RV_REG_S3, ctx)) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_S3, ctx);
+		store_offset -= 8;
+	}
+	if (seen_reg(RV_REG_S4, ctx)) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_S4, ctx);
+		store_offset -= 8;
+	}
+	if (seen_reg(RV_REG_S5, ctx)) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_S5, ctx);
+		store_offset -= 8;
+	}
+	if (ctx->arena_vm_start) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_ARENA, ctx);
+		store_offset -= 8;
+	}
+
+	/* store TCC from RV_REG_TCC to stack */
+	emit_sd(RV_REG_SP, store_offset, RV_REG_TCC, ctx);
+	ctx->tcc_offset = store_offset;
+
+	emit_addi(RV_REG_FP, RV_REG_SP, stack_adjust, ctx);
+}
+
+static void emit_normal_restore(struct rv_jit_context *ctx, int stack_adjust)
+{
+	int store_offset = stack_adjust - 8;
 
 	if (seen_reg(RV_REG_RA, ctx)) {
 		emit_ld(RV_REG_RA, store_offset, RV_REG_SP, ctx);
@@ -235,6 +310,13 @@ static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx)
 
 	/* restore TCC from stack to RV_REG_TCC */
 	emit_ld(RV_REG_TCC, ctx->tcc_offset, RV_REG_SP, ctx);
+}
+
+static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx)
+{
+	int stack_adjust = ctx->stack_size;
+
+	emit_normal_restore(ctx, stack_adjust);
 
 	emit_addi(RV_REG_SP, RV_REG_SP, stack_adjust, ctx);
 	/* Set return value. */
@@ -2006,34 +2088,12 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 
 void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
 {
-	int i, stack_adjust = 0, store_offset, bpf_stack_adjust;
+	int i, stack_adjust, bpf_stack_adjust;
 
 	bpf_stack_adjust = round_up(ctx->prog->aux->stack_depth, STACK_ALIGN);
 	if (bpf_stack_adjust)
 		mark_fp(ctx);
 
-	if (seen_reg(RV_REG_RA, ctx))
-		stack_adjust += 8;
-	stack_adjust += 8; /* RV_REG_FP */
-	if (seen_reg(RV_REG_S1, ctx))
-		stack_adjust += 8;
-	if (seen_reg(RV_REG_S2, ctx))
-		stack_adjust += 8;
-	if (seen_reg(RV_REG_S3, ctx))
-		stack_adjust += 8;
-	if (seen_reg(RV_REG_S4, ctx))
-		stack_adjust += 8;
-	if (seen_reg(RV_REG_S5, ctx))
-		stack_adjust += 8;
-	if (ctx->arena_vm_start)
-		stack_adjust += 8;
-	stack_adjust += 8; /* RV_REG_TCC */
-
-	stack_adjust = round_up(stack_adjust, STACK_ALIGN);
-	stack_adjust += bpf_stack_adjust;
-
-	store_offset = stack_adjust - 8;
-
 	/* emit kcfi type preamble immediately before the  first insn */
 	emit_kcfi(is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash, ctx);
 
@@ -2046,46 +2106,8 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
 	if (!is_subprog)
 		emit(rv_addi(RV_REG_TCC, RV_REG_ZERO, MAX_TAIL_CALL_CNT), ctx);
 
-	/* tailcall starts here, emit insn before it must be fixed */
-
-	emit_addi(RV_REG_SP, RV_REG_SP, -stack_adjust, ctx);
-
-	if (seen_reg(RV_REG_RA, ctx)) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_RA, ctx);
-		store_offset -= 8;
-	}
-	emit_sd(RV_REG_SP, store_offset, RV_REG_FP, ctx);
-	store_offset -= 8;
-	if (seen_reg(RV_REG_S1, ctx)) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_S1, ctx);
-		store_offset -= 8;
-	}
-	if (seen_reg(RV_REG_S2, ctx)) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_S2, ctx);
-		store_offset -= 8;
-	}
-	if (seen_reg(RV_REG_S3, ctx)) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_S3, ctx);
-		store_offset -= 8;
-	}
-	if (seen_reg(RV_REG_S4, ctx)) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_S4, ctx);
-		store_offset -= 8;
-	}
-	if (seen_reg(RV_REG_S5, ctx)) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_S5, ctx);
-		store_offset -= 8;
-	}
-	if (ctx->arena_vm_start) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_ARENA, ctx);
-		store_offset -= 8;
-	}
-
-	/* store TCC from RV_REG_TCC to stack */
-	emit_sd(RV_REG_SP, store_offset, RV_REG_TCC, ctx);
-	ctx->tcc_offset = store_offset;
-
-	emit_addi(RV_REG_FP, RV_REG_SP, stack_adjust, ctx);
+	stack_adjust = normal_stack_adjust(ctx) + bpf_stack_adjust;
+	emit_normal_prologue(ctx, stack_adjust);
 
 	if (bpf_stack_adjust)
 		emit_addi(RV_REG_S5, RV_REG_SP, bpf_stack_adjust, ctx);
Signature · 1 line
2.55.0
maintainer
#7 lore raw

Re: [PATCH bpf-next v3 4/6] riscv, bpf: Split prologue and epilogue into helper functions

Hi Varun,

looks good to me, but some real small nits.

Quoted 22 lines · Varun R Mallya

Split bpf_jit_build_prologue() and __build_epilogue() into helpers. normal_stack_adjust() computes the frame size for the callee-saved registers a program actually clobbers, emit_normal_prologue() allocates the frame and spills those registers, and emit_normal_restore() reloads them.

The kcfi preamble, fentry nops and tail-call-counter setup are hoisted before the frame-size computation and they do not depend on it and the emitted instruction sequence is unchanged.

No functional change.

---
  arch/riscv/net/bpf_jit_comp64.c | 152 ++++++++++++++++++--------------
  1 file changed, 87 insertions(+), 65 deletions(-)
arch/riscv/net/bpf_jit_comp64.c +0 -0
index ad089a9a4ea9..b804382075c4 100644

arch/riscv/net/bpf_jit_comp64.c static void emit_imm(u8 rd, s64 val, struct rv_jit_context *ctx) · line 200

Quoted 3 lines
arch/riscv/net/bpf_jit_comp64.c +0 -0
@@ -198,9 +198,84 @@ static void emit_imm(u8 rd, s64 val, struct rv_jit_context *ctx)
  		emit_addi(rd, rd, lower, ctx);
  }
 

pls use the following comment format:

/*
* xxx
*/

or

/* xxx */

arch/riscv/net/bpf_jit_comp64.c static void emit_imm(u8 rd, s64 val, struct rv_jit_context *ctx) · line 201

Quoted 2 lines
arch/riscv/net/bpf_jit_comp64.c +1 -1
-static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx)
+/* Stack space for the callee-saved registers a normal program spills,

how about, the registers based strictly on what is seen in the bpf prog.

Quoted 21 lines
arch/riscv/net/bpf_jit_comp64.c +19 -1
+ * excluding the BPF stack itself.
+ */
+static int normal_stack_adjust(struct rv_jit_context *ctx)
  {
-	int stack_adjust = ctx->stack_size, store_offset = stack_adjust - 8;
+	int stack_adjust = 0;
+
+	if (seen_reg(RV_REG_RA, ctx))
+		stack_adjust += 8;
+	stack_adjust += 8; /* RV_REG_FP */
+	if (seen_reg(RV_REG_S1, ctx))
+		stack_adjust += 8;
+	if (seen_reg(RV_REG_S2, ctx))
+		stack_adjust += 8;
+	if (seen_reg(RV_REG_S3, ctx))
+		stack_adjust += 8;
+	if (seen_reg(RV_REG_S4, ctx))
+		stack_adjust += 8;
+	if (seen_reg(RV_REG_S5, ctx))
+		stack_adjust += 8;
+	if (ctx->arena_vm_start)

arch/riscv/net/bpf_jit_comp64.c static void emit_imm(u8 rd, s64 val, struct rv_jit_context *ctx) · line 231

Quoted 10 lines
arch/riscv/net/bpf_jit_comp64.c +10 -0
+		stack_adjust += 8;
+	stack_adjust += 8; /* RV_REG_TCC */
+
+	return round_up(stack_adjust, STACK_ALIGN);
+}
+
+/* Allocate the frame and save the callee-saved registers the program
+ * actually clobbers, then point FP at the frame top.
+ */
+static void emit_normal_prologue(struct rv_jit_context *ctx, int stack_adjust)

prefer to let ctx as last param, and the following pls make a change.

arch/riscv/net/bpf_jit_comp64.c static void emit_imm(u8 rd, s64 val, struct rv_jit_context *ctx) · line 234

Quoted 3 lines
arch/riscv/net/bpf_jit_comp64.c +3 -0
+{
+	int store_offset = stack_adjust - 8;
+	/* tailcall starts here, emit insn before it must be fixed */

remove this comment.

Quoted 97 lines
arch/riscv/net/bpf_jit_comp64.c +51 -23
+
+	emit_addi(RV_REG_SP, RV_REG_SP, -stack_adjust, ctx);
+
+	if (seen_reg(RV_REG_RA, ctx)) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_RA, ctx);
+		store_offset -= 8;
+	}
+	emit_sd(RV_REG_SP, store_offset, RV_REG_FP, ctx);
+	store_offset -= 8;
+	if (seen_reg(RV_REG_S1, ctx)) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_S1, ctx);
+		store_offset -= 8;
+	}
+	if (seen_reg(RV_REG_S2, ctx)) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_S2, ctx);
+		store_offset -= 8;
+	}
+	if (seen_reg(RV_REG_S3, ctx)) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_S3, ctx);
+		store_offset -= 8;
+	}
+	if (seen_reg(RV_REG_S4, ctx)) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_S4, ctx);
+		store_offset -= 8;
+	}
+	if (seen_reg(RV_REG_S5, ctx)) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_S5, ctx);
+		store_offset -= 8;
+	}
+	if (ctx->arena_vm_start) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_ARENA, ctx);
+		store_offset -= 8;
+	}
+
+	/* store TCC from RV_REG_TCC to stack */
+	emit_sd(RV_REG_SP, store_offset, RV_REG_TCC, ctx);
+	ctx->tcc_offset = store_offset;
+
+	emit_addi(RV_REG_FP, RV_REG_SP, stack_adjust, ctx);
+}
+
+static void emit_normal_restore(struct rv_jit_context *ctx, int stack_adjust) > +{
+	int store_offset = stack_adjust - 8;
 
  	if (seen_reg(RV_REG_RA, ctx)) {
  		emit_ld(RV_REG_RA, store_offset, RV_REG_SP, ctx);
@@ -235,6 +310,13 @@ static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx)
 
  	/* restore TCC from stack to RV_REG_TCC */
  	emit_ld(RV_REG_TCC, ctx->tcc_offset, RV_REG_SP, ctx);
+}
+
+static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx)
+{
+	int stack_adjust = ctx->stack_size;
+
+	emit_normal_restore(ctx, stack_adjust);
 
  	emit_addi(RV_REG_SP, RV_REG_SP, stack_adjust, ctx);
  	/* Set return value. */
@@ -2006,34 +2088,12 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 
  void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
  {
-	int i, stack_adjust = 0, store_offset, bpf_stack_adjust;
+	int i, stack_adjust, bpf_stack_adjust;
 
  	bpf_stack_adjust = round_up(ctx->prog->aux->stack_depth, STACK_ALIGN);
  	if (bpf_stack_adjust)
  		mark_fp(ctx);
 
-	if (seen_reg(RV_REG_RA, ctx))
-		stack_adjust += 8;
-	stack_adjust += 8; /* RV_REG_FP */
-	if (seen_reg(RV_REG_S1, ctx))
-		stack_adjust += 8;
-	if (seen_reg(RV_REG_S2, ctx))
-		stack_adjust += 8;
-	if (seen_reg(RV_REG_S3, ctx))
-		stack_adjust += 8;
-	if (seen_reg(RV_REG_S4, ctx))
-		stack_adjust += 8;
-	if (seen_reg(RV_REG_S5, ctx))
-		stack_adjust += 8;
-	if (ctx->arena_vm_start)
-		stack_adjust += 8;
-	stack_adjust += 8; /* RV_REG_TCC */
-
-	stack_adjust = round_up(stack_adjust, STACK_ALIGN);
-	stack_adjust += bpf_stack_adjust;
-
-	store_offset = stack_adjust - 8;
-
  	/* emit kcfi type preamble immediately before the  first insn */
  	emit_kcfi(is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash, ctx);
 

arch/riscv/net/bpf_jit_comp64.c void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog) · line 2049

Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +0 -1
@@ -2046,46 +2106,8 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
  	if (!is_subprog)
  		emit(rv_addi(RV_REG_TCC, RV_REG_ZERO, MAX_TAIL_CALL_CNT), ctx);
 
-	/* tailcall starts here, emit insn before it must be fixed */

keep this comment here.

Quoted 44 lines
arch/riscv/net/bpf_jit_comp64.c +2 -39
-
-	emit_addi(RV_REG_SP, RV_REG_SP, -stack_adjust, ctx);
-
-	if (seen_reg(RV_REG_RA, ctx)) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_RA, ctx);
-		store_offset -= 8;
-	}
-	emit_sd(RV_REG_SP, store_offset, RV_REG_FP, ctx);
-	store_offset -= 8;
-	if (seen_reg(RV_REG_S1, ctx)) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_S1, ctx);
-		store_offset -= 8;
-	}
-	if (seen_reg(RV_REG_S2, ctx)) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_S2, ctx);
-		store_offset -= 8;
-	}
-	if (seen_reg(RV_REG_S3, ctx)) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_S3, ctx);
-		store_offset -= 8;
-	}
-	if (seen_reg(RV_REG_S4, ctx)) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_S4, ctx);
-		store_offset -= 8;
-	}
-	if (seen_reg(RV_REG_S5, ctx)) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_S5, ctx);
-		store_offset -= 8;
-	}
-	if (ctx->arena_vm_start) {
-		emit_sd(RV_REG_SP, store_offset, RV_REG_ARENA, ctx);
-		store_offset -= 8;
-	}
-
-	/* store TCC from RV_REG_TCC to stack */
-	emit_sd(RV_REG_SP, store_offset, RV_REG_TCC, ctx);
-	ctx->tcc_offset = store_offset;
-
-	emit_addi(RV_REG_FP, RV_REG_SP, stack_adjust, ctx);
+	stack_adjust = normal_stack_adjust(ctx) + bpf_stack_adjust;
+	emit_normal_prologue(ctx, stack_adjust);
 
  	if (bpf_stack_adjust)
  		emit_addi(RV_REG_S5, RV_REG_SP, bpf_stack_adjust, ctx);
#8 lore raw

PATCH v3 5/6 riscv, bpf: Add support for BPF exceptions

In reply to Varun R Mallya

Add the JIT support required for BPF exceptions on riscv64.

Two kinds of programs need special prologue/epilogue handling:

- A program acting as an exception boundary must save the full set of riscv callee-saved GP registers (ra, s0-s11), not just the ones it clobbers, so that the exception callback can restore the state that was live at the boundary. ra and fp are stored first so the saved ra/fp pair forms a valid stackframe record for the unwinder.

- exception_cb does not allocate its own frame. It reuses the boundary program's frame, whose frame pointer is passed in a2, by setting SP = FP - stack_adjust. This lines the epilogue's loads up with the registers the boundary saved, so both paths restore the same order.

Both cases are implemented as prologue/epilogue helpers next to the normal-path ones, selected with a plain if/else in bpf_jit_build_prologue() and __build_epilogue().

---
 arch/riscv/net/bpf_jit_comp64.c | 112 +++++++++++++++++++++++++++++++-
 1 file changed, 109 insertions(+), 3 deletions(-)
arch/riscv/net/bpf_jit_comp64.c +109 -3
index b804382075c4..e0c2f73bc646 100644
@@ -56,6 +56,29 @@ static const int pt_regmap[] = {
 	[RV_REG_T0] = offsetof(struct pt_regs, t0),
 };
 
+/* Full set of RISC-V callee-saved GP registers (ra, s0-s11) saved by a program
+ * acting as an exception boundary, in the order they are stored on the stack.
+ * RA and FP come first so the saved ra/fp pair forms a valid stackframe record
+ * at [FP-8]/[FP-16] for the unwinder. The exception callback reuses the
+ * boundary program's frame and restores this same set in its epilogue, so both
+ * paths must agree on the contents and ordering of this list.
+ */
+static const int rv_exception_csave_regs[] = {
+	RV_REG_RA,
+	RV_REG_FP,
+	RV_REG_S1,
+	RV_REG_S2,
+	RV_REG_S3,
+	RV_REG_S4,
+	RV_REG_S5,
+	RV_REG_S6,
+	RV_REG_S7,
+	RV_REG_S8,
+	RV_REG_S9,
+	RV_REG_S10,
+	RV_REG_S11,
+};
+
 enum {
 	RV_CTX_F_SEEN_CALL =		RV_REG_RA,
 	RV_CTX_F_SEEN_S1 =		RV_REG_S1,
@@ -312,11 +335,67 @@ static void emit_normal_restore(struct rv_jit_context *ctx, int stack_adjust)
 	emit_ld(RV_REG_TCC, ctx->tcc_offset, RV_REG_SP, ctx);
 }
 
+static bool is_exception_prog(const struct bpf_prog *prog)
+{
+	return prog->aux->exception_boundary || prog->aux->exception_cb;
+}
+
+static void emit_exception_boundary_prologue(struct rv_jit_context *ctx,
+					     int stack_adjust)
+{
+	int store_offset = stack_adjust - 8;
+	int i;
+
+	emit_addi(RV_REG_SP, RV_REG_SP, -stack_adjust, ctx);
+	for (i = 0; i < ARRAY_SIZE(rv_exception_csave_regs); i++) {
+		emit_sd(RV_REG_SP, store_offset, rv_exception_csave_regs[i], ctx);
+		store_offset -= 8;
+	}
+
+	/* store TCC from RV_REG_TCC to stack */
+	emit_sd(RV_REG_SP, store_offset, RV_REG_TCC, ctx);
+	ctx->tcc_offset = store_offset;
+
+	emit_addi(RV_REG_FP, RV_REG_SP, stack_adjust, ctx);
+}
+
+/* reuses the boundary program's frame, whose frame pointer
+ * is passed in a2 (since it's the third argument from bpf_throw()).
+ * Setting SP = FP - stack_adjust lines the epilogue's loads up with
+ * the registers the boundary saved.
+ */
+static void emit_exception_cb_prologue(struct rv_jit_context *ctx,
+				       int stack_adjust)
+{
+	emit_mv(RV_REG_FP, RV_REG_A2, ctx);
+	emit_addi(RV_REG_SP, RV_REG_FP, -stack_adjust, ctx);
+	ctx->tcc_offset = stack_adjust - 8 - ARRAY_SIZE(rv_exception_csave_regs) * 8;
+}
+
+/* Used by both the boundary program and the exception callback alike, since the
+ * callback runs on the boundary's frame.
+ */
+static void emit_exception_restore(struct rv_jit_context *ctx, int stack_adjust)
+{
+	int store_offset = stack_adjust - 8;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(rv_exception_csave_regs); i++) {
+		emit_ld(rv_exception_csave_regs[i], store_offset, RV_REG_SP, ctx);
+		store_offset -= 8;
+	}
+	/* restore TCC from stack to RV_REG_TCC */
+	emit_ld(RV_REG_TCC, ctx->tcc_offset, RV_REG_SP, ctx);
+}
+
 static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx)
 {
 	int stack_adjust = ctx->stack_size;
 
-	emit_normal_restore(ctx, stack_adjust);
+	if (is_exception_prog(ctx->prog))
+		emit_exception_restore(ctx, stack_adjust);
+	else
+		emit_normal_restore(ctx, stack_adjust);
 
 	emit_addi(RV_REG_SP, RV_REG_SP, stack_adjust, ctx);
 	/* Set return value. */
@@ -2086,6 +2165,15 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 	return 0;
 }
 
+static int exception_stack_adjust(void)
+{
+	int stack_adjust = 0;
+
+	stack_adjust += 8; /* RV_REG_TCC */
+	stack_adjust += ARRAY_SIZE(rv_exception_csave_regs) * 8;
+	return round_up(stack_adjust, STACK_ALIGN);
+}
+
 void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
 {
 	int i, stack_adjust, bpf_stack_adjust;
@@ -2106,8 +2194,17 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
 	if (!is_subprog)
 		emit(rv_addi(RV_REG_TCC, RV_REG_ZERO, MAX_TAIL_CALL_CNT), ctx);
 
-	stack_adjust = normal_stack_adjust(ctx) + bpf_stack_adjust;
-	emit_normal_prologue(ctx, stack_adjust);
+	if (is_exception_prog(ctx->prog)) {
+		stack_adjust = exception_stack_adjust() + bpf_stack_adjust;
+
+		if (ctx->prog->aux->exception_cb)
+			emit_exception_cb_prologue(ctx, stack_adjust);
+		else
+			emit_exception_boundary_prologue(ctx, stack_adjust);
+	} else {
+		stack_adjust = normal_stack_adjust(ctx) + bpf_stack_adjust;
+		emit_normal_prologue(ctx, stack_adjust);
+	}
 
 	if (bpf_stack_adjust)
 		emit_addi(RV_REG_S5, RV_REG_SP, bpf_stack_adjust, ctx);
@@ -2183,3 +2280,12 @@ bool bpf_jit_supports_subprog_tailcalls(void)
 {
 	return true;
 }
+
+bool bpf_jit_supports_exceptions(void)
+{
+	/* bpf_throw() unwinds by walking the frame-pointer chain from inside
+	 * the kernel back into the BPF frames (see arch_bpf_stack_walk()), so
+	 * exceptions require the frame-pointer unwinder to be enabled.
+	 */
+	return IS_ENABLED(CONFIG_FRAME_POINTER);
+}
Signature · 1 line
2.55.0
#9 lore raw

Re: [PATCH bpf-next v3 5/6] riscv, bpf: Add support for BPF exceptions

This isn't a bug, but Bjorn Topel requested netdev-style multi-line comments in v2. Should the opening /* be on the same line as the first text instead of on its own line?

arch/riscv/net/bpf_jit_comp64.c static const int pt_regmap[] = { · line 60

Quoted 10 lines
arch/riscv/net/bpf_jit_comp64.c +2 -0
index b804382075c40..e0c2f73bc646a 100644
@@ -56,6 +56,29 @@ static const int pt_regmap[] = {
 	[RV_REG_T0] = offsetof(struct pt_regs, t0),
 };
 
+/* Full set of RISC-V callee-saved GP registers (ra, s0-s11) saved by a program
+ * acting as an exception boundary, in the order they are stored on the stack.
     ^^

This isn't a bug, but Bjorn Topel requested netdev-style multi-line comments in v2. Should the opening /* be on the same line as the first text instead of on its own line?

Quoted 20 lines
arch/riscv/net/bpf_jit_comp64.c +20 -0
+ * RA and FP come first so the saved ra/fp pair forms a valid stackframe record
+ * at [FP-8]/[FP-16] for the unwinder. The exception callback reuses the
+ * boundary program's frame and restores this same set in its epilogue, so both
+ * paths must agree on the contents and ordering of this list.
+ */
+static const int rv_exception_csave_regs[] = {
+	RV_REG_RA,
+	RV_REG_FP,
+	RV_REG_S1,
+	RV_REG_S2,
+	RV_REG_S3,
+	RV_REG_S4,
+	RV_REG_S5,
+	RV_REG_S6,
+	RV_REG_S7,
+	RV_REG_S8,
+	RV_REG_S9,
+	RV_REG_S10,
+	RV_REG_S11,
+};

[ ... ]

Quoted 20 lines
arch/riscv/net/bpf_jit_comp64.c +16 -0
@@ -312,11 +335,67 @@ static void emit_normal_restore(struct rv_jit_context *ctx, int stack_adjust)
 	emit_ld(RV_REG_TCC, ctx->tcc_offset, RV_REG_SP, ctx);
 }
 
+static bool is_exception_prog(const struct bpf_prog *prog)
+{
+	return prog->aux->exception_boundary || prog->aux->exception_cb;
+}
+
+static void emit_exception_boundary_prologue(struct rv_jit_context *ctx,
+					     int stack_adjust)
+{
+	int store_offset = stack_adjust - 8;
+	int i;
+
+	emit_addi(RV_REG_SP, RV_REG_SP, -stack_adjust, ctx);
+	for (i = 0; i < ARRAY_SIZE(rv_exception_csave_regs); i++) {
+		emit_sd(RV_REG_SP, store_offset, rv_exception_csave_regs[i], ctx);
+		store_offset -= 8;
+	}

arch/riscv/net/bpf_jit_comp64.c static void emit_normal_restore(struct rv_jit_context *ctx, int stack_adjust) · line 363

Quoted 10 lines
arch/riscv/net/bpf_jit_comp64.c +10 -0
+
+	/* store TCC from RV_REG_TCC to stack */
+	emit_sd(RV_REG_SP, store_offset, RV_REG_TCC, ctx);
+	ctx->tcc_offset = store_offset;
+
+	emit_addi(RV_REG_FP, RV_REG_SP, stack_adjust, ctx);
+}
+
+/* reuses the boundary program's frame, whose frame pointer
+ * is passed in a2 (since it's the third argument from bpf_throw()).
     ^^

Same comment style here - should this also follow netdev style?

arch/riscv/net/bpf_jit_comp64.c static void emit_normal_restore(struct rv_jit_context *ctx, int stack_adjust) · line 376

Quoted 13 lines
arch/riscv/net/bpf_jit_comp64.c +13 -0
+ * Setting SP = FP - stack_adjust lines the epilogue's loads up with
+ * the registers the boundary saved.
+ */
+static void emit_exception_cb_prologue(struct rv_jit_context *ctx,
+				       int stack_adjust)
+{
+	emit_mv(RV_REG_FP, RV_REG_A2, ctx);
+	emit_addi(RV_REG_SP, RV_REG_FP, -stack_adjust, ctx);
+	ctx->tcc_offset = stack_adjust - 8 - ARRAY_SIZE(rv_exception_csave_regs) * 8;
+}
+
+/* Used by both the boundary program and the exception callback alike, since the
+ * callback runs on the boundary's frame.
     ^^

And here as well.

Quoted 26 lines
arch/riscv/net/bpf_jit_comp64.c +18 -1
+ */
+static void emit_exception_restore(struct rv_jit_context *ctx, int stack_adjust)
+{
+	int store_offset = stack_adjust - 8;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(rv_exception_csave_regs); i++) {
+		emit_ld(rv_exception_csave_regs[i], store_offset, RV_REG_SP, ctx);
+		store_offset -= 8;
+	}
+	/* restore TCC from stack to RV_REG_TCC */
+	emit_ld(RV_REG_TCC, ctx->tcc_offset, RV_REG_SP, ctx);
+}
+
 static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx)
 {
 	int stack_adjust = ctx->stack_size;
 
-	emit_normal_restore(ctx, stack_adjust);
+	if (is_exception_prog(ctx->prog))
+		emit_exception_restore(ctx, stack_adjust);
+	else
+		emit_normal_restore(ctx, stack_adjust);
 
 	emit_addi(RV_REG_SP, RV_REG_SP, stack_adjust, ctx);
 	/* Set return value. */

[ ... ]

arch/riscv/net/bpf_jit_comp64.c bool bpf_jit_supports_subprog_tailcalls(void) · line 2287

Quoted 9 lines
arch/riscv/net/bpf_jit_comp64.c +5 -0
@@ -2183,3 +2280,12 @@ bool bpf_jit_supports_subprog_tailcalls(void)
 {
 	return true;
 }
+
+bool bpf_jit_supports_exceptions(void)
+{
+	/* bpf_throw() unwinds by walking the frame-pointer chain from inside
+	 * the kernel back into the BPF frames (see arch_bpf_stack_walk()), so
         ^^

And one more multi-line comment here with the same style.

arch/riscv/net/bpf_jit_comp64.c bool bpf_jit_supports_subprog_tailcalls(void) · line 2291

Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +4 -0
+	 * exceptions require the frame-pointer unwinder to be enabled.
+	 */
+	return IS_ENABLED(CONFIG_FRAME_POINTER);
+}

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/29929750005

#10 lore raw

PATCH v3 6/6 riscv, bpf: Remove BPF exceptions from denylist

In reply to Varun R Mallya

This patch removes BPF exceptions from riscv64 denylist on BPF selftests since support for exceptions has been added now. Also, since the file is empty, delete the file.

---
 tools/testing/selftests/bpf/DENYLIST.riscv64 | 2 --
 1 file changed, 2 deletions(-)
 delete mode 100644 tools/testing/selftests/bpf/DENYLIST.riscv64
tools/testing/selftests/bpf/DENYLIST.riscv64 deleted +0 -2
deleted file mode 100644
index ca1beae7fe8f..000000000000
@@ -1,2 +0,0 @@
-# riscv64 deny list for BPF CI and local vmtest
-exceptions					# JIT does not support exceptions
Signature · 1 line
2.55.0
#11 lore raw

Re: [PATCH bpf-next v3 6/6] riscv, bpf: Remove BPF exceptions from denylist

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Deleting `DENYLIST.riscv64` breaks the explicit testing instructions provided in `README.rst`. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Deleting DENYLIST.riscv64 breaks the explicit testing instructions provided in README.rst. --

--- Patch [6]: [PATCH bpf-next v3 6/6] riscv, bpf: Remove BPF exceptions from denylist --- commit e62702d45ea21775bb10258f8620e8e53e10eee4
Author: Varun R Mallya <[email protected]>

riscv, bpf: Remove BPF exceptions from denylist

This patch removes BPF exceptions from the riscv64 denylist in BPF selftests and deletes the file entirely since it is now empty.

tools/testing/selftests/bpf/DENYLIST.riscv64

Quoted 5 lines
tools/testing/selftests/bpf/DENYLIST.riscv64 deleted +0 -0
deleted file mode 100644
index ca1beae7fe8f1..0000000000000

[Severity: Low]
Will deleting this file break the manual testing instructions provided in tools/testing/selftests/bpf/README.rst?

The documentation explicitly uses cat on this file to assemble the vmtest.sh command:

tools/testing/selftests/bpf/README.rst:

    tools/testing/selftests/bpf/vmtest.sh \
    -l <path of local rootfs image> -- \
    ./test_progs -d \
        "$(cat tools/testing/selftests/bpf/DENYLIST.riscv64 \
            | cut -d'#' -f1 \
            ...

Since running this example command will now result in a shell error due to the missing file, should tools/testing/selftests/bpf/README.rst be updated in this commit to remove or adjust the reference?

Quoted 3 lines
tools/testing/selftests/bpf/DENYLIST.riscv64 deleted +0 -2
@@ -1,2 +0,0 @@
-# riscv64 deny list for BPF CI and local vmtest
-exceptions					# JIT does not support exceptions
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
maintainer
#12 lore raw

Re: [PATCH bpf-next v3 6/6] riscv, bpf: Remove BPF exceptions from denylist

In reply to Varun R Mallya
Quoted 16 lines · Varun R Mallya

This patch removes BPF exceptions from riscv64 denylist on BPF selftests since support for exceptions has been added now. Also, since the file is empty, delete the file.

---
  tools/testing/selftests/bpf/DENYLIST.riscv64 | 2 --
  1 file changed, 2 deletions(-)

delete mode 100644 tools/testing/selftests/bpf/DENYLIST.riscv64

tools/testing/selftests/bpf/DENYLIST.riscv64 deleted +0 -0
deleted file mode 100644
index ca1beae7fe8f..000000000000

tools/testing/selftests/bpf/DENYLIST.riscv64 line 1

Quoted 1 line
tools/testing/selftests/bpf/DENYLIST.riscv64 deleted +0 -1
@@ -1,2 +0,0 @@
-# riscv64 deny list for BPF CI and local vmtest

let's keep this file with the above comment.

Quoted 1 line
tools/testing/selftests/bpf/DENYLIST.riscv64 deleted +0 -1
-exceptions					# JIT does not support exceptions
maintainer
#13 lore raw
In reply to Varun R Mallya
Quoted 15 lines · Varun R Mallya

This patchset aims to add BPF exceptions supports for riscv64 by implementing the arch_bpf_stack_walk function and then updating the prologue and epilogue for BPF JIT on riscv. Also, bpf_jit_supports_exceptions() returns true now so that the verifier does not reject programs containing BPF exceptions on riscv64.

On riscv the unwinder used by arch_bpf_stack_walk() is the frame-pointer unwinder, so exception support is gated on CONFIG_FRAME_POINTER.

In the prologue and epilogue of the RISC-V JIT, I saved the return address and then the frame-pointer according to [1]. Also, according to [2], s0 to s11 are callee saved registers, which is why a new array (rv_exception_csave_regs) has been created to save these registers which contains all the required registers along with the frame pointer as well as return address.

...

Quoted 9 lines

Since all exception tests pass now, the denylist file has been removed as it is empty now.

Changes in v1->v2:
v1: https://lore.kernel.org/bpf/[email protected]/
- resolve the ftrace trampoline address with ftrace_graph_ret_addr() inside arch_bpf_stack_walk()
- Remove duplicated code in bpf_jit_build_prologue()

Quoted 9 lines

Changes in v2->v3:
v2: https://lore.kernel.org/bpf/[email protected]/
- extend walk_stackframe to use sp/fp similar to how arm64 does it. (Björn Töpel)
- refactor prologue/epilogue into helpers. (Björn Töpel)
- fix comments to make them netdev style.
- save/restore the tail-call counter in the exception prologue/epilogue
- exception_tail_call now passes since bpf2bpf+tailcall mixing landed [3], so drop the (now-empty) DENYLIST.riscv64
- (!aux->exception_cb) has been simplified to only one condition (Pu Lehui)

Varun!

Thanks for the rework. The overall structure is much better now, and the main v2 comments around reusing the stack walker and splitting the prologue/epilogue have been addressed!

I tried to make sense of Sashiko's reports as well...

Sashiko seems correct about sp being stale after crossing an exception frame. We restore pc and fp from pt_regs, so sp should be restored from there as well. Would you agree?

The full pt_regs should also be validated before dereferencing it...?

I *think* the instrumentation concern also looks valid. The new stack-walk adapter callbacks should not be instrumentable.

I don't think the !CONFIG_FRAME_POINTER report is an exception-support bug. Exceptions require a valid boundary frame pointer, so the current gating looks correct.

Sashiko is also correct that deleting DENYLIST.riscv64 breaks the documented selftest command. Please update the README or retain an empty file.

Minor wording issue: ra is not callee-saved. Please say “RA and the callee-saved registers s0-s11.”

Thanks,
Björn