Skip to contentSkip to conversation

riscv: bpf: Add BPF private stack support

PATCH In review Chen Pei 2 participants 5 messages Nov 6 – Nov 7

Subsystems (6)
Kselftest BPF RISC-V BPF selftests +2 more subsystems
lore
Share & tools
lore

Chen Pei

author
#0 lore raw

v3 cover riscv: bpf: Add BPF private stack support

This series adds private-stack support to the RV64 BPF JIT.

The verifier rejects a struct_ops program that requests a private stack with -EACCES when the JIT does not report bpf_jit_supports_private_stack(), and sched_ext requests one for its dispatch callback, so no sched_ext scheduler can load on riscv64 today.

Allocation, the guard value and the lifetime handling follow arm64, x86-64 and powerpc64; the 16 byte two-copy guard layout follows arm64 and powerpc64, since x86-64 uses an 8 byte guard with a single copy. What is specific to riscv64 is where the pointers end up:

- Every BPF stack access goes through BPF_REG_FP (S5), so pointing S5 at the top of the per-CPU area and leaving SP on the kernel stack is all the JIT has to do. S5 is callee-saved and each subprog derives its own, so bpf2bpf calls need no extra save and restore.
- The outgoing stack argument area stays in the kernel frame: callers write it relative to SP while callees read it relative to RV_REG_FP. Unlike x86-64, no register here is shared with the private-stack frame pointer, so no verifier-side exclusion is needed and stack arguments keep working together with a private stack. arm64 already does the same, and patch 3 tests that combination.
- The per-CPU lookup is emitted after the tail call entry point. A program using a private stack can still be entered as a tail call target, at RV_TAILCALL_OFFSET, so it has to set up its own S5 there; and since that offset is a constant added to the target's bpf_func, everything in front of it must be the same size in every program.

The base address is materialized with absolute moves, as on arm64, x86-64 and powerpc64, which reach the per-CPU offset through gs:, TPIDR or r13; riscv64 has no such register, so the address of __per_cpu_offset is loaded too. Patch 1 has the detail, including why emit_imm() is a choice there and why the prologue length stays fixed across passes.

RV32 does not declare support and keeps the existing behaviour; the prologue signature is unchanged, so bpf_jit_comp32.c needs no edit.

Changes since v2:
- Patch 1: PRIV_STACK_GUARD_VAL moved to bpf_jit.h.
- Patch 3: The test redone as two asm programs in progs/verifier_stack_arg.c.

Changes since v1:
- Patch 1 commit message: guard layout credited to arm64 and powerpc64, value alone to x86-64; __per_cpu_offset described as a choice.
- Patch 3: test_private_stack_runtime() renamed to serial_test_private_stack_runtime(), and the stack argument skip prints a reason.

Testing
=======
QEMU virt rv64, an 8 vCPU SMP kernel and a 1 vCPU UP kernel, both built on the base commit below, source tree clean at the head of this series, selftests built from these patches. There are no private-stack related failures, and the private-stack suites logged no BUG, WARNING, unknown opcode or guard corruption. On UP, kfunc_call/subprog, kfunc_call/subprog_lskel and scx_simple are not run: all three fail on the unpatched base as well.

- verifier_private_stack, struct_ops_private_stack.
- tailcalls, stack_arg, subprogs, fentry_fexit, kfunc_call, percpu_alloc, async_stack_depth, arena_atomics, arena_ldsx.
- verifier_stack_arg, stack_arg_precision.
- may_goto_priv_stack, may_goto_far, verifier_may_goto_1, verifier_may_goto_2.
- scx_simple.
- sched_ext runner sweep on SMP: 40 cases, 39 pass. enable_cmask fails because its ops.enable() takes a struct scx_cmask __arena pointer and riscv64 does not implement bpf_jit_supports_arena_args().
- patch 3's test again on x86-64 and arm64, so all three architectures it is gated on have run it.

Chen Pei (3):
riscv: bpf: Add BPF private stack support selftests/bpf: Enable private stack tests for riscv64 selftests/bpf: Test a private stack together with stack arguments

 arch/riscv/net/bpf_jit.h                      |  2 +
 arch/riscv/net/bpf_jit_comp64.c               | 42 +++++++++-
 arch/riscv/net/bpf_jit_core.c                 | 82 +++++++++++++++++--
 .../bpf/prog_tests/struct_ops_private_stack.c |  3 +-
 .../bpf/progs/verifier_private_stack.c        | 41 +++++++++-
 .../selftests/bpf/progs/verifier_stack_arg.c  | 49 +++++++++++
 6 files changed, 205 insertions(+), 14 deletions(-)

base-commit: a3c8813409f13779fec225468aba7d4067a66468

Signature · 1 line
2.50.1
author
#1 lore raw

PATCH v3 1/3 riscv: bpf: Add BPF private stack support

A program on a private stack keeps its BPF frames in a per-CPU area of its own rather than on the kernel stack of whichever context entered it. bpf_enable_priv_stack() grants that mode only once the JIT reports bpf_jit_supports_private_stack(), which riscv64 did not: programs there always ran on the kernel stack, and a struct_ops program that requires one was rejected with -EACCES.

Allocate a per-CPU area of the verifier calculated depth plus a 16 byte guard region at each end, and point BPF_REG_FP (S5) at the top of it while leaving SP on the kernel stack. Every BPF stack access already goes through S5, and S5 is callee-saved with each subprog deriving its own, so nothing else in the JIT has to know about the private stack.

The outgoing stack argument area stays in the kernel frame: callers write it relative to SP, callees relative to RV_REG_FP, which the prologue sets to the caller's SP.

The per-CPU lookup is emitted after the tail call entry point: a program using a private stack can still be entered as a tail call target, and RV_TAILCALL_OFFSET is a constant added to the target's bpf_func, so what is in front of it must be the same size in every program.

The base is materialized with absolute moves, as on arm64, x86-64 and
powerpc64: the vmalloc'd per-CPU area is beyond an auipc pair from the JIT region, and riscv64 has no register holding the per-CPU offset, so the address of __per_cpu_offset is loaded too. emit_imm() is a choice there, not a requirement; emit_addr() as used by insn_is_mov_percpu_addr() would need an extra_pass argument and an error return in bpf_jit_build_prologue(), touching bpf_jit_comp32.c too. Both constants are fixed before the offset convergence loop, so the prologue length is stable across passes.

The guard value follows arm64, x86-64 and powerpc64, and the 16 byte two-copy region at each end follows arm64 and powerpc64, since x86-64 uses an 8 byte guard with a single copy. A jited program has them checked when it is freed, reporting the first corrupted CPU only; failure paths release the area unchecked since the image never ran, which also clears priv_stack_ptr for the later bpf_jit_free().

RV32 does not declare support and keeps the existing kernel stack behaviour.

---
 arch/riscv/net/bpf_jit.h        |  2 +
 arch/riscv/net/bpf_jit_comp64.c | 42 +++++++++++++++--
 arch/riscv/net/bpf_jit_core.c   | 82 +++++++++++++++++++++++++++++----
 3 files changed, 114 insertions(+), 12 deletions(-)
arch/riscv/net/bpf_jit.h +2 -0
index 21a190a6a6d9..dbe1ac8cd0de 100644
@@ -14,6 +14,8 @@
 
 #define RV_MAX_REG_ARGS		8
 #define RV_EXTRA_STK_ARGS	(RV_MAX_REG_ARGS - MAX_BPF_FUNC_REG_ARGS)
+#define PRIV_STACK_GUARD_SZ	16
+#define PRIV_STACK_GUARD_VAL	0xEB9F12345678eb9fULL
 
 /* verify runtime detection extension status */
 #define rv_ext_enabled(ext) \
arch/riscv/net/bpf_jit_comp64.c +39 -3

61 lines not on this page · raw message

arch/riscv/net/bpf_jit_core.c +73 -9

134 lines not on this page · raw message

Signature · 1 line
2.50.1
author
#2 lore raw

PATCH v3 2/3 selftests/bpf: Enable private stack tests for riscv64

In reply to Chen Pei

The RV64 JIT now supports the private stack. Allow the existing verifier and struct_ops private-stack tests to run on riscv64 and add the expected RV64 JITed instruction patterns.

---
 .../bpf/prog_tests/struct_ops_private_stack.c |  3 +-
 .../bpf/progs/verifier_private_stack.c        | 41 ++++++++++++++++++-
 2 files changed, 42 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c +2 -1
index 2b3ec2b79091..103179ff8733 100644
@@ -6,7 +6,8 @@
 #include "struct_ops_private_stack_recur.skel.h"
 #include "struct_ops_private_stack_large.skel.h"
 
-#if defined(__x86_64__) || defined(__aarch64__) || defined(__powerpc64__)
+#if defined(__x86_64__) || defined(__aarch64__) || defined(__powerpc64__) || \
+	(defined(__riscv) && __riscv_xlen == 64)
 static void test_private_stack(void)
 {
 	struct struct_ops_private_stack *skel;
tools/testing/selftests/bpf/progs/verifier_private_stack.c +40 -1

83 lines not on this page · raw message

Signature · 1 line
2.50.1
author
#3 lore raw

PATCH v3 3/3 selftests/bpf: Test a private stack together with stack arguments

In reply to Chen Pei

Add two asm programs to verifier_stack_arg.c, inside the block that is already gated on x86, arm64 and riscv64. A raw_tp program with a 64 byte frame calls a ten-argument subprog with a 64 byte frame of its own, and BPF_PROG_TEST_RUN runs the pair without attaching it. Args 9 and 10 stay in the kernel frame and arg 10 points into the caller's frame, so the two have to agree on where the outgoing area is. The callee returns what the caller stored plus arg 9, so a wrong area shows up as a wrong retval rather than as a crash.

---
 .../selftests/bpf/progs/verifier_stack_arg.c  | 49 +++++++++++++++++++
 1 file changed, 49 insertions(+)
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +49 -0
index 146cdf81bbc4..e8108e768b89 100644
@@ -975,6 +975,55 @@ __naked void stack_arg_precision_fallback(void)
 	);
 }
 
+/* Writes both ends of its frame, then reads args 9, 10 and the frame of the caller. */
+static __naked __noinline __used
+long priv_stack_10args(long a, long b, long c, long d, long e, long f, long g, long h,
+		       long i, long *j)
+{
+	asm volatile (
+		"r1 = 0;"
+		"*(u64 *)(r10 - 8) = r1;"
+		"*(u64 *)(r10 - 64) = r1;"
+		"r0 = *(u64 *)(r11 + 32);"
+		"r1 = *(u64 *)(r11 + 40);"
+		"r1 = *(u64 *)(r1 + 0);"
+		"r0 += r1;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * A frame of 64 bytes in a raw_tp prog goes to the private stack.
+ * arm64 and riscv64 pass args 9 and 10 on the kernel stack. On x86-64
+ * both frames stay there as well.
+ */
+SEC("raw_tp")
+__description("stack_arg: private stack in the caller and in the callee")
+__success __retval(890)
+__naked void stack_arg_priv_stack(void)
+{
+	asm volatile (
+		"r1 = 800;"
+		"*(u64 *)(r10 - 64) = r1;"
+		"r1 = r10;"
+		"r1 += -64;"
+		"*(u64 *)(r11 - 40) = r1;"
+		"*(u64 *)(r11 - 32) = 90;"
+		"*(u64 *)(r11 - 24) = 8;"
+		"*(u64 *)(r11 - 16) = 7;"
+		"*(u64 *)(r11 - 8) = 6;"
+		"r1 = 1;"
+		"r2 = 2;"
+		"r3 = 3;"
+		"r4 = 4;"
+		"r5 = 5;"
+		"call priv_stack_10args;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
 #else
 
 /* No BTF tells about this store. The insn is what JIT has to deal with. */
Signature · 1 line
2.50.1
maintainer Reviewed-by on v3 3/3
#4 lore raw

Re: [PATCH v3 bpf-next 3/3] selftests/bpf: Test a private stack together with stack arguments

Quoted 65 lines · Chen Pei

Add two asm programs to verifier_stack_arg.c, inside the block that is already gated on x86, arm64 and riscv64. A raw_tp program with a 64 byte frame calls a ten-argument subprog with a 64 byte frame of its own, and BPF_PROG_TEST_RUN runs the pair without attaching it. Args 9 and 10 stay in the kernel frame and arg 10 points into the caller's frame, so the two have to agree on where the outgoing area is. The callee returns what the caller stored plus arg 9, so a wrong area shows up as a wrong retval rather than as a crash.

---
  .../selftests/bpf/progs/verifier_stack_arg.c  | 49 +++++++++++++++++++
  1 file changed, 49 insertions(+)
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +42 -0
index 146cdf81bbc4..e8108e768b89 100644
@@ -975,6 +975,55 @@ __naked void stack_arg_precision_fallback(void)
  	);
  }
 
+/* Writes both ends of its frame, then reads args 9, 10 and the frame of the caller. */
+static __naked __noinline __used
+long priv_stack_10args(long a, long b, long c, long d, long e, long f, long g, long h,
+		       long i, long *j)
+{
+	asm volatile (
+		"r1 = 0;"
+		"*(u64 *)(r10 - 8) = r1;"
+		"*(u64 *)(r10 - 64) = r1;"
+		"r0 = *(u64 *)(r11 + 32);"
+		"r1 = *(u64 *)(r11 + 40);"
+		"r1 = *(u64 *)(r1 + 0);"
+		"r0 += r1;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+/*
+ * A frame of 64 bytes in a raw_tp prog goes to the private stack.
+ * arm64 and riscv64 pass args 9 and 10 on the kernel stack. On x86-64
+ * both frames stay there as well.
+ */
+SEC("raw_tp")
+__description("stack_arg: private stack in the caller and in the callee")
+__success __retval(890)
+__naked void stack_arg_priv_stack(void)
+{
+	asm volatile (
+		"r1 = 800;"
+		"*(u64 *)(r10 - 64) = r1;"
+		"r1 = r10;"
+		"r1 += -64;"
+		"*(u64 *)(r11 - 40) = r1;"
+		"*(u64 *)(r11 - 32) = 90;"
+		"*(u64 *)(r11 - 24) = 8;"
+		"*(u64 *)(r11 - 16) = 7;"
+		"*(u64 *)(r11 - 8) = 6;"
+		"r1 = 1;"
+		"r2 = 2;"
+		"r3 = 3;"
+		"r4 = 4;"

tools/testing/selftests/bpf/progs/verifier_stack_arg.c __naked void stack_arg_precision_fallback(void) · line 1029

Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +7 -0
+		"r5 = 5;"
+		"call priv_stack_10args;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
  #else
 
  /* No BTF tells about this store. The insn is what JIT has to deal with. */