Skip to contentSkip to conversation

riscv: bpf: Add BPF private stack support

PATCH Superseded Chen Pei 5 participants 11 messages Nov 4 – Nov 6

Subsystems (6)
Kselftest BPF RISC-V BPF selftests +2 more subsystems

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Chen Pei

author
#0 lore raw

v2 cover riscv: bpf: Add BPF private stack support

This series adds private-stack support to the RV64 BPF JIT.

The verifier rejects a struct_ops program that requests a private stack with -EACCES when the JIT does not report bpf_jit_supports_private_stack(), and sched_ext requests one for its dispatch callback, so no sched_ext scheduler can load on riscv64 today.

Allocation, the guard value and the lifetime handling follow arm64, x86-64 and powerpc64; the 16-byte two-copy guard layout follows arm64 and powerpc64, since x86-64 uses an 8-byte guard with a single copy. What is specific to riscv64 is where the pointers end up:

- Every BPF stack access goes through BPF_REG_FP (S5), so pointing S5 at the top of the per-CPU area and leaving SP on the kernel stack is all the JIT has to do. S5 is callee-saved and each subprog derives its own, so bpf2bpf calls need no extra save and restore.
- The outgoing stack-argument area stays in the kernel frame: callers write it relative to SP while callees read it relative to RV_REG_FP. Unlike x86-64, no register here is shared with the private-stack frame pointer, so no verifier-side exclusion is needed and stack arguments keep working together with a private stack. arm64 already does the same, and patch 3 measures the three ranges at run time and pins them in the disassembly.
- The per-CPU lookup is emitted after the tail-call entry point. A program using a private stack can still be entered as a tail-call target, at RV_TAILCALL_OFFSET, so it has to set up its own S5 there; and since that offset is a constant added to the target's bpf_func, everything in front of it must be the same size in every program.

The base address is materialized with absolute moves, as on arm64, x86-64 and powerpc64. Those three reach the per-CPU offset through gs:, TPIDR or r13 and only have to move the base into a register; riscv64 has no such register, so the address of __per_cpu_offset is loaded here too. A PC-relative pair is not an option for the base itself: the dynamic per-CPU area is vmalloc'd, further from the JIT region than auipc reaches, which is why the whole setup is ten instructions on the kernel tested here. __per_cpu_offset is a kernel-image symbol and on that kernel lands in the sign-extended 32-bit range, so emit_imm() renders it in two instructions through its 32-bit fast path - what an emit_addr() pair would cost, and the form insn_is_mov_percpu_addr() already uses; unifying the two needs an extra_pass argument and an error return in bpf_jit_build_prologue(), which also touches bpf_jit_comp32.c, so it is left as is. Both constants are fixed before the offset-convergence loop, so the prologue length is stable across passes.

RV32 does not declare support and keeps the existing behaviour; the prologue signature is unchanged, so bpf_jit_comp32.c needs no edit.

Changes since v1:
- Patch 1 commit message: guard layout credited to arm64 and powerpc64, value alone to x86-64; __per_cpu_offset described as a choice.
- Patch 3: test_private_stack_runtime() renamed to serial_test_private_stack_runtime(), and the stack-argument skip prints a reason.

Testing
=======
QEMU virt rv64, an 8 vCPU SMP kernel and a 1 vCPU UP kernel, both built on the base commit below, source tree clean at the head of this series, selftests built from these patches. There are no private-stack related failures, and no run logged a BUG, WARNING, panic, unknown opcode or guard corruption. On UP, kfunc_call/subprog, kfunc_call/subprog_lskel and scx_simple are not run: all three fail the same way on the unpatched base.

- verifier_private_stack, struct_ops_private_stack, private_stack_runtime.
- tailcalls, stack_arg, subprogs, fentry_fexit, kfunc_call, percpu_alloc, async_stack_depth, arena_atomics, arena_ldsx.
- verifier_stack_arg, stack_arg_precision.
- may_goto_priv_stack, may_goto_far, verifier_may_goto_1, verifier_may_goto_2.
- scx_simple.
- sched_ext runner sweep on SMP: 40 cases, 39 pass. enable_cmask fails because its ops.enable() takes a struct scx_cmask __arena pointer and riscv64 does not implement bpf_jit_supports_arena_args().

Chen Pei (3):
riscv: bpf: Add BPF private stack support selftests/bpf: Enable private stack tests for riscv64 selftests/bpf: Add a runtime test for private stack and stack
arguments

 arch/riscv/net/bpf_jit.h                      |   1 +
 arch/riscv/net/bpf_jit_comp64.c               |  42 +++++-
 arch/riscv/net/bpf_jit_core.c                 |  84 +++++++++--
 .../selftests/bpf/private_stack_runtime.h     |  12 ++
 .../bpf/prog_tests/private_stack_runtime.c    |  84 +++++++++++
 .../bpf/prog_tests/struct_ops_private_stack.c |   3 +-
 .../bpf/progs/private_stack_runtime.c         | 140 ++++++++++++++++++
 .../bpf/progs/verifier_private_stack.c        |  41 ++++-
 8 files changed, 393 insertions(+), 14 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/private_stack_runtime.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c
 create mode 100644 tools/testing/selftests/bpf/progs/private_stack_runtime.c

base-commit: 8fb43ffc3327d12d57d3fbe6f4820b5a522de0d1

Signature · 1 line
2.50.1
author
#1 lore raw

PATCH v2 1/3 riscv: bpf: Add BPF private stack support

A program on a private stack keeps its BPF frames in a per-CPU area of its own rather than on the kernel stack of whichever context entered it. bpf_enable_priv_stack() grants that mode only once the JIT reports bpf_jit_supports_private_stack(), which riscv64 did not: programs there always ran on the kernel stack, and a struct_ops program that requires one was rejected with -EACCES.

Allocate a per-CPU area of the verifier calculated depth plus a 16 byte guard region at each end, and point BPF_REG_FP (S5) at the top of it while leaving SP on the kernel stack. Every BPF stack access already goes through S5, and S5 is callee-saved with each subprog deriving its own, so nothing else in the JIT has to know about the private stack.

The outgoing stack argument area stays in the kernel frame: callers write it relative to SP, callees relative to RV_REG_FP, which the prologue sets to the caller's SP.

The per-CPU lookup is emitted after the tail call entry point: a program using a private stack can still be entered as a tail call target, and RV_TAILCALL_OFFSET is a constant added to the target's bpf_func, so what is in front of it must be the same size in every program.

The base is materialized with absolute moves, as on arm64, x86-64 and
powerpc64: the vmalloc'd per-CPU area is beyond an auipc pair from the JIT region, and riscv64 has no register holding the per-CPU offset, so the address of __per_cpu_offset is loaded too. emit_imm() is a choice there, not a requirement; emit_addr() as used by insn_is_mov_percpu_addr() would need an extra_pass argument and an error return in bpf_jit_build_prologue(), touching bpf_jit_comp32.c too. Both constants are fixed before the offset convergence loop, so the prologue length is stable across passes.

The guard value follows arm64, x86-64 and powerpc64, and the 16 byte two-copy region at each end follows arm64 and powerpc64, since x86-64 uses an 8 byte guard with a single copy. A jited program has them checked when it is freed, reporting the first corrupted CPU only; failure paths release the area unchecked since the image never ran, which also clears priv_stack_ptr for the later bpf_jit_free().

RV32 does not declare support and keeps the existing kernel stack behaviour.

---
 arch/riscv/net/bpf_jit.h        |  1 +
 arch/riscv/net/bpf_jit_comp64.c | 42 +++++++++++++++--
 arch/riscv/net/bpf_jit_core.c   | 84 +++++++++++++++++++++++++++++----
 3 files changed, 115 insertions(+), 12 deletions(-)
arch/riscv/net/bpf_jit.h +1 -0
index 21a190a6a6d9..efab2b6803e4 100644
@@ -14,6 +14,7 @@
 
 #define RV_MAX_REG_ARGS		8
 #define RV_EXTRA_STK_ARGS	(RV_MAX_REG_ARGS - MAX_BPF_FUNC_REG_ARGS)
+#define PRIV_STACK_GUARD_SZ	16
 
 /* verify runtime detection extension status */
 #define rv_ext_enabled(ext) \
arch/riscv/net/bpf_jit_comp64.c +39 -3

61 lines not on this page · raw message

arch/riscv/net/bpf_jit_core.c +75 -9

136 lines not on this page · raw message

Signature · 1 line
2.50.1
maintainer Reviewed-by on v2 1/3
#2 lore raw

Hi Pei,

Quoted 142 lines · Chen Pei

A program on a private stack keeps its BPF frames in a per-CPU area of its own rather than on the kernel stack of whichever context entered it. bpf_enable_priv_stack() grants that mode only once the JIT reports bpf_jit_supports_private_stack(), which riscv64 did not: programs there always ran on the kernel stack, and a struct_ops program that requires one was rejected with -EACCES.

Allocate a per-CPU area of the verifier calculated depth plus a 16 byte guard region at each end, and point BPF_REG_FP (S5) at the top of it while leaving SP on the kernel stack. Every BPF stack access already goes through S5, and S5 is callee-saved with each subprog deriving its own, so nothing else in the JIT has to know about the private stack.

The outgoing stack argument area stays in the kernel frame: callers write it relative to SP, callees relative to RV_REG_FP, which the prologue sets to the caller's SP.

The per-CPU lookup is emitted after the tail call entry point: a program using a private stack can still be entered as a tail call target, and RV_TAILCALL_OFFSET is a constant added to the target's bpf_func, so what is in front of it must be the same size in every program.

The base is materialized with absolute moves, as on arm64, x86-64 and
powerpc64: the vmalloc'd per-CPU area is beyond an auipc pair from the JIT region, and riscv64 has no register holding the per-CPU offset, so the address of __per_cpu_offset is loaded too. emit_imm() is a choice there, not a requirement; emit_addr() as used by insn_is_mov_percpu_addr() would need an extra_pass argument and an error return in bpf_jit_build_prologue(), touching bpf_jit_comp32.c too. Both constants are fixed before the offset convergence loop, so the prologue length is stable across passes.

The guard value follows arm64, x86-64 and powerpc64, and the 16 byte two-copy region at each end follows arm64 and powerpc64, since x86-64 uses an 8 byte guard with a single copy. A jited program has them checked when it is freed, reporting the first corrupted CPU only; failure paths release the area unchecked since the image never ran, which also clears priv_stack_ptr for the later bpf_jit_free().

RV32 does not declare support and keeps the existing kernel stack behaviour.

---
  arch/riscv/net/bpf_jit.h        |  1 +
  arch/riscv/net/bpf_jit_comp64.c | 42 +++++++++++++++--
  arch/riscv/net/bpf_jit_core.c   | 84 +++++++++++++++++++++++++++++----
  3 files changed, 115 insertions(+), 12 deletions(-)
arch/riscv/net/bpf_jit.h +1 -0
index 21a190a6a6d9..efab2b6803e4 100644
@@ -14,6 +14,7 @@
 
  #define RV_MAX_REG_ARGS		8
  #define RV_EXTRA_STK_ARGS	(RV_MAX_REG_ARGS - MAX_BPF_FUNC_REG_ARGS)
+#define PRIV_STACK_GUARD_SZ	16
 
  /* verify runtime detection extension status */
  #define rv_ext_enabled(ext) \
arch/riscv/net/bpf_jit_comp64.c +39 -3
index 371405598faa..a7a7d2f9da85 100644
@@ -2134,12 +2134,34 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
  	return 0;
  }
 
+static void emit_priv_stack_frame(struct rv_jit_context *ctx)
+{
+	u64 fp = (__force u64)ctx->prog->aux->priv_stack_ptr + PRIV_STACK_GUARD_SZ;
+
+	fp += round_up(ctx->prog->aux->stack_depth, STACK_ALIGN);
+	emit_imm(RV_REG_S5, fp, ctx);
+#ifdef CONFIG_SMP
+	emit_lw(RV_REG_T1, offsetof(struct thread_info, cpu), RV_REG_TP, ctx);
+	emit_imm(RV_REG_T2, (u64)__per_cpu_offset, ctx);
+	emit_sh3add(RV_REG_T1, RV_REG_T1, RV_REG_T2, ctx);
+	emit_ld(RV_REG_T1, 0, RV_REG_T1, ctx);
+	emit_add(RV_REG_S5, RV_REG_S5, RV_REG_T1, ctx);
+#endif
+}
+
  void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
  {
+	bool priv_stack = ctx->prog->aux->priv_stack_ptr;
  	int i, stack_adjust = 0, store_offset, bpf_stack_adjust = ctx->stack_arg_sz;
 
-	bpf_stack_adjust += round_up(ctx->prog->aux->stack_depth, STACK_ALIGN);
-	if (bpf_stack_adjust)
+	/*
+	 * With a private stack only the outgoing stack argument area is left in
+	 * this frame. It has to stay: the caller writes it relative to SP while
+	 * the callee reads it relative to RV_REG_FP, the caller's SP set below.
+	 */
+	if (!priv_stack)
+		bpf_stack_adjust += round_up(ctx->prog->aux->stack_depth, STACK_ALIGN);
+	if (bpf_stack_adjust || priv_stack)
  		mark_fp(ctx);
 
  	if (seen_reg(RV_REG_RA, ctx))
@@ -2217,7 +2239,16 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
 
  	emit_addi(RV_REG_FP, RV_REG_SP, stack_adjust, ctx);
 
-	if (bpf_stack_adjust)
+	/*
+	 * Point BPF_REG_FP at the top of this CPU's private stack. A program
+	 * with one can still be entered as a tail call target, at
+	 * RV_TAILCALL_OFFSET, so this has to come after that entry point; the
+	 * offset is a constant added to the target's bpf_func, so everything in
+	 * front of it must also be the same size in every program.
+	 */
+	if (priv_stack)
+		emit_priv_stack_frame(ctx);
+	else if (bpf_stack_adjust)
  		emit_addi(RV_REG_S5, RV_REG_SP, bpf_stack_adjust, ctx);
 
  	ctx->stack_size = stack_adjust;
@@ -2320,3 +2351,8 @@ bool bpf_jit_supports_hmul(void)
  {
  	return true;
  }
+
+bool bpf_jit_supports_private_stack(void)
+{
+	return true;
+}
arch/riscv/net/bpf_jit_core.c +1 -0
index f2497ec7d378..f2021088e447 100644
@@ -9,6 +9,7 @@
  #include <linux/bpf.h>
  #include <linux/filter.h>
  #include <linux/memory.h>
+#include <linux/percpu.h>
  #include <asm/text-patching.h>
  #include <asm/cfi.h>
  #include "bpf_jit.h"

arch/riscv/net/bpf_jit_core.c bool bpf_jit_needs_zext(void) · line 48

Quoted 4 lines
arch/riscv/net/bpf_jit_core.c +1 -0
@@ -44,6 +45,62 @@ bool bpf_jit_needs_zext(void)
  	return true;
  }
 
+#define PRIV_STACK_GUARD_VAL	0xEB9F12345678eb9fULL

a nit, let move it with the PRIV_STACK_GUARD_SZ macro.

others, looks good.

Quoted 95 lines
arch/riscv/net/bpf_jit_core.c +62 -4
+
+static u32 priv_stack_size(struct bpf_prog *prog)
+{
+	return round_up(prog->aux->stack_depth, STACK_ALIGN) +
+	       2 * PRIV_STACK_GUARD_SZ;
+}
+
+static int alloc_priv_stack(struct bpf_prog *prog)
+{
+	u32 size = priv_stack_size(prog);
+	u32 end = (size - PRIV_STACK_GUARD_SZ) / sizeof(u64);
+	void __percpu *ptr;
+	int cpu, i;
+
+	ptr = __alloc_percpu_gfp(size, STACK_ALIGN, GFP_KERNEL);
+	if (!ptr)
+		return -ENOMEM;
+
+	for_each_possible_cpu(cpu) {
+		u64 *stack = per_cpu_ptr(ptr, cpu);
+
+		for (i = 0; i < PRIV_STACK_GUARD_SZ / sizeof(u64); i++) {
+			stack[i] = PRIV_STACK_GUARD_VAL;
+			stack[end + i] = PRIV_STACK_GUARD_VAL;
+		}
+	}
+	prog->aux->priv_stack_ptr = ptr;
+	return 0;
+}
+
+static void priv_stack_check_guard(struct bpf_prog *prog)
+{
+	u32 end = (priv_stack_size(prog) - PRIV_STACK_GUARD_SZ) / sizeof(u64);
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		u64 *stack = per_cpu_ptr(prog->aux->priv_stack_ptr, cpu);
+
+		if (stack[0] != PRIV_STACK_GUARD_VAL ||
+		    stack[1] != PRIV_STACK_GUARD_VAL ||
+		    stack[end] != PRIV_STACK_GUARD_VAL ||
+		    stack[end + 1] != PRIV_STACK_GUARD_VAL) {
+			pr_err("bpf-jit: private stack overflow/underflow guard of prog %s corrupted on CPU %d\n",
+			       bpf_jit_get_prog_name(prog), cpu);
+			return;
+		}
+	}
+}
+
+static void free_priv_stack(struct bpf_prog *prog)
+{
+	free_percpu(prog->aux->priv_stack_ptr);
+	prog->aux->priv_stack_ptr = NULL;
+}
+
  struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
  {
  	unsigned int prog_size = 0, extable_size = 0;
@@ -84,12 +141,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
  			    round_up((stack_arg_cnt - RV_EXTRA_STK_ARGS) * 8, STACK_ALIGN) : 0;
 
  	ctx->prog = prog;
+	if (prog->aux->jits_use_priv_stack && alloc_priv_stack(prog))
+		goto out_free_hdr;
+
  	ctx->offset = kvzalloc_objs(int, prog->len);
  	if (!ctx->offset)
-		goto out_offset;
+		goto out_free_hdr;
 
  	if (build_body(ctx, extra_pass, NULL))
-		goto out_offset;
+		goto out_free_hdr;
 
  	for (i = 0; i < prog->len; i++) {
  		prev_ninsns += 32;
@@ -104,7 +164,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
  		ctx->prologue_len = ctx->ninsns;
 
  		if (build_body(ctx, extra_pass, ctx->offset))
-			goto out_offset;
+			goto out_free_hdr;
 
  		ctx->epilogue_offset = ctx->ninsns;
  		bpf_jit_build_epilogue(ctx);
@@ -124,7 +184,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
  							  bpf_fill_ill_insns,
  							  bpf_prog_was_classic(prog));
  			if (!jit_data->ro_header)
-				goto out_offset;
+				goto out_free_hdr;
 
  			/*
  			 * Use the image(RW) for writing the JITed instructions. But also save

arch/riscv/net/bpf_jit_core.c struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · line 253

Quoted 8 lines
arch/riscv/net/bpf_jit_core.c +1 -4
@@ -190,10 +250,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
  			ctx->offset[i] = ctx->offset[i - 1];
  		ctx->offset[0] = ninsns_rvoff(ctx->prologue_len);
  		bpf_prog_update_insn_ptrs(prog, ctx->offset, jit_data->ro_image);
-out_offset:
-		kvfree(ctx->offset);
-		kfree(jit_data);
-		prog->aux->jit_data = NULL;
+		goto out_free_ctx;

happy to see this, this place was indeed very confusing before.

Quoted 28 lines
arch/riscv/net/bpf_jit_core.c +10 -1
  	}
 
  	return prog;
@@ -209,7 +266,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
  				   sizeof(jit_data->header->size));
  		bpf_jit_binary_pack_free(jit_data->ro_header, jit_data->header);
  	}
-	goto out_offset;
+	free_priv_stack(prog);
+out_free_ctx:
+	kvfree(ctx->offset);
+	kfree(jit_data);
+	prog->aux->jit_data = NULL;
+	return prog;
  }
 
  u64 bpf_jit_alloc_exec_limit(void)
@@ -261,6 +323,10 @@ void bpf_jit_free(struct bpf_prog *prog)
  		prog->bpf_func = (void *)prog->bpf_func - cfi_get_offset();
  		hdr = bpf_jit_binary_pack_hdr(prog);
  		bpf_jit_binary_pack_free(hdr, NULL);
+		if (prog->aux->priv_stack_ptr) {
+			priv_stack_check_guard(prog);
+			free_priv_stack(prog);
+		}
  		WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(prog));
  	}
 
author
#3 lore raw

Hi Lehui,

Thanks for the review.

Quoted 1 line

a nit, let move it with the PRIV_STACK_GUARD_SZ macro.

Will fix it in the next version.

Best regards,
Chen

author
#4 lore raw

PATCH v2 2/3 selftests/bpf: Enable private stack tests for riscv64

In reply to Chen Pei

The RV64 JIT now supports the private stack. Allow the existing verifier and struct_ops private-stack tests to run on riscv64 and add the expected RV64 JITed instruction patterns.

---
 .../bpf/prog_tests/struct_ops_private_stack.c |  3 +-
 .../bpf/progs/verifier_private_stack.c        | 41 ++++++++++++++++++-
 2 files changed, 42 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c +2 -1
index 2b3ec2b79091..103179ff8733 100644
@@ -6,7 +6,8 @@
 #include "struct_ops_private_stack_recur.skel.h"
 #include "struct_ops_private_stack_large.skel.h"
 
-#if defined(__x86_64__) || defined(__aarch64__) || defined(__powerpc64__)
+#if defined(__x86_64__) || defined(__aarch64__) || defined(__powerpc64__) || \
+	(defined(__riscv) && __riscv_xlen == 64)
 static void test_private_stack(void)
 {
 	struct struct_ops_private_stack *skel;
tools/testing/selftests/bpf/progs/verifier_private_stack.c +40 -1

83 lines not on this page · raw message

Signature · 1 line
2.50.1
maintainer Reviewed-by on v2 2/3
#5 lore raw

Re: [PATCH v2 bpf-next 2/3] selftests/bpf: Enable private stack tests for riscv64

Quoted 108 lines · Chen Pei

The RV64 JIT now supports the private stack. Allow the existing verifier and struct_ops private-stack tests to run on riscv64 and add the expected RV64 JITed instruction patterns.

---
  .../bpf/prog_tests/struct_ops_private_stack.c |  3 +-
  .../bpf/progs/verifier_private_stack.c        | 41 ++++++++++++++++++-
  2 files changed, 42 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c +2 -1
index 2b3ec2b79091..103179ff8733 100644
@@ -6,7 +6,8 @@
  #include "struct_ops_private_stack_recur.skel.h"
  #include "struct_ops_private_stack_large.skel.h"
 
-#if defined(__x86_64__) || defined(__aarch64__) || defined(__powerpc64__)
+#if defined(__x86_64__) || defined(__aarch64__) || defined(__powerpc64__) || \
+	(defined(__riscv) && __riscv_xlen == 64)
  static void test_private_stack(void)
  {
  	struct struct_ops_private_stack *skel;
tools/testing/selftests/bpf/progs/verifier_private_stack.c +36 -1
index bbc09f516d92..4df06a82dd04 100644
@@ -8,7 +8,8 @@
  /* From include/linux/filter.h */
  #define MAX_BPF_STACK    512
 
-#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
+#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || \
+	(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)
 
  struct elem {
  	struct bpf_timer t;
@@ -42,6 +43,14 @@ __jited("	mov	x0, #0x2a")
  __jited("	str	x0, [x27]")
  __jited("...")
  __jited("	ldp	x25, x27, [sp], {{.*}}")
+__arch_riscv64
+__jited("	addi	sp, sp, -0x20")
+__jited("...")
+__jited("	sd	s5, 0x10(sp)")
+__jited("...")
+__jited("	sd	a0, -0x100(s5)")
+__jited("...")
+__jited("	ld	s5, 0x10(sp)")
  __naked void private_stack_single_prog(void)
  {
  	asm volatile ("			\
@@ -60,6 +69,10 @@ __jited("	subq	$0x8, %rsp")
  __arch_arm64
  __jited("	mov	x25, sp")
  __jited("	sub	sp, sp, #0x10")
+__arch_riscv64
+__jited("	addi	sp, sp, -0x30")
+__jited("...")
+__jited("	addi	s5, sp, 0x10")
  __naked void no_private_stack_nested(void)
  {
  	asm volatile ("			\
@@ -113,6 +126,14 @@ __jited("	str	x0, [x27]")
  __jited("	bl	{{.*}}")
  __jited("...")
  __jited("	ldp	x25, x27, [sp], {{.*}}")
+__arch_riscv64
+__jited("	addi	sp, sp, -0x20")
+__jited("...")
+__jited("	sd	s5, 0x8(sp)")
+__jited("...")
+__jited("	sd	a0, -0x200(s5)")
+__jited("...")
+__jited("	ld	s5, 0x8(sp)")
  __naked void private_stack_nested_1(void)
  {
  	asm volatile ("				\
@@ -183,6 +204,16 @@ __jited("	bl	0x{{.*}}")
  __jited("	mov	x8, x0")
  __jited("	mov	x8, #0x0")
  __jited("	ldp	x25, x27, [sp], {{.*}}")
+__arch_riscv64
+__jited("func #1")
+__jited("...")
+__jited("	addi	sp, sp, -0x20")
+__jited("...")
+__jited("	sd	s5, 0x8(sp)")
+__jited("...")
+__jited("	sd	a0, -0x200(s5)")
+__jited("...")
+__jited("	ld	s5, 0x8(sp)")
  __naked void private_stack_callback(void)
  {
  	asm volatile ("			\
@@ -412,6 +443,10 @@ __arch_arm64
  __jited("	mrs	x10, TPIDR_EL{{[0-1]}}")
  __jited("	add	x27, x27, x10")
  __jited("	add	x25, x27, {{.*}}")
+__arch_riscv64
+__jited("	addi	sp, sp, -0x20")
+__jited("...")
+__jited("	sd	s5, 0x8(sp)")
  int private_stack_async_callback_1(void)
  {
  	struct bpf_timer *arr_timer;

tools/testing/selftests/bpf/progs/verifier_private_stack.c __arch_x86_64 · line 485

Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_private_stack.c +4 -0
@@ -441,6 +476,10 @@ __arch_x86_64
  __jited("	subq	$0x100, %rsp")
  __arch_arm64
  __jited("	sub	sp, sp, #0x100")
+__arch_riscv64
+__jited("	addi	sp, sp, -0x120")
+__jited("...")
+__jited("	addi	s5, sp, 0x100")
  int private_stack_async_callback_2(void)
  {
  	struct bpf_timer *arr_timer;

Apart from not yet supporting exception handling, everything else is fine.

author
#6 lore raw

PATCH v2 3/3 selftests/bpf: Add a runtime test for private stack and stack arguments

In reply to Chen Pei

No existing test reaches the combination. bpf_enable_priv_stack() grants an adaptive private stack only to kprobe, tracepoint, perf_event and raw_tracepoint programs, and to tracing, lsm and struct_ops ones that request it or recurse. The stack-argument tests run from tc, socket, syscall and fentry/fexit, all left at NO_PRIV_STACK; the private-stack tests call subprograms of up to three arguments, which stay in registers. Neither pairs a private stack with outgoing stack arguments, where the caller writes the argument area relative to SP and the callee reads it relative to the frame pointer.

Add a runtime test for that combination. A static ten-argument callee writes through a pointer into the caller frame and keeps a 128-byte volatile local array, so its own depth crosses the 64-byte threshold at which the verifier grants a subprog the adaptive private stack. The same workload runs from a raw tracepoint, which gets a private stack, and from tc, which does not; the test pins itself to each allowed CPU in turn and checks the CPU the program reported.

A runtime result cannot show where the three stack areas landed, so the riscv64 __jited patterns pin the layout: the caller's kernel frame is 48 bytes against the control group's 176 and the callee's 64 against 208, the 128 and 144 byte differences being exactly the BPF stack that moved to the per-CPU area. Both groups write the two outgoing arguments at 0x0(sp) and 0x8(sp) and the callee reads them back at 0x0(s0) and 0x8(s0), so the outgoing area, the saved registers and the BPF stack occupy disjoint ranges. None of the patterns assert the per-CPU lookup, which is only emitted on CONFIG_SMP, so one set matches SMP and UP alike.

The zero-depth struct_ops program is a control case for the kernel stack path, not a private-stack allocation: the verifier grants the adaptive private stack only once a subprog's stack depth, rounded up to the JIT's 16-byte granularity, reaches 64 bytes.

---

Notes:
Reachability of a private stack together with outgoing stack arguments, per architecture, for review context only:

- x86-64 cannot reach it. The verifier denies the private stack to subprograms with stack arguments there because R9 doubles as the private stack frame pointer and argument 6, so both halves of this test would be kernel stack runs.
- powerpc64 has no stack arguments at all, as it does not report bpf_jit_supports_stack_args(), so it cannot reach it either.
- arm64 reports both bpf_jit_supports_private_stack() and bpf_jit_supports_stack_args(), so the combination is reachable there today and has no test for it. Widening the gate to arm64 needs arm64 __jited patterns and an arm64 machine to validate them, so I would post that separately once the maintainers agree this runtime test is worth keeping.

 .../selftests/bpf/private_stack_runtime.h     |  12 ++
 .../bpf/prog_tests/private_stack_runtime.c    |  84 +++++++++++
 .../bpf/progs/private_stack_runtime.c         | 140 ++++++++++++++++++
 3 files changed, 236 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/private_stack_runtime.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c
 create mode 100644 tools/testing/selftests/bpf/progs/private_stack_runtime.c
tools/testing/selftests/bpf/private_stack_runtime.h new file +12 -0
new file mode 100644
index 000000000000..8ed49ff66a6e
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __PRIVATE_STACK_RUNTIME_H
+#define __PRIVATE_STACK_RUNTIME_H
+
+struct private_stack_result {
+	long value;
+	long first;
+	long last;
+	__u32 cpu;
+};
+
+#endif
tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c new file +84 -0

84 lines not on this page · raw message

tools/testing/selftests/bpf/progs/private_stack_runtime.c new file +140 -0

140 lines not on this page · raw message

Signature · 1 line
2.50.1
#7 lore raw

Re: [PATCH v2 bpf-next 3/3] selftests/bpf: Add a runtime test for private stack and stack arguments

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] selftests/bpf: broken compiler skip in serial_test_private_stack_runtime() --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] selftests/bpf: broken compiler skip in serial_test_private_stack_runtime() --

commit d5748c85c54dbc46700395b1926da29c28fcdef7
Author: Chen Pei <[email protected]>

selftests/bpf: Add a runtime test for private stack and stack arguments

This commit adds a runtime and JIT disassembly selftest on 64-bit RISC-V for BPF programs that combine an adaptive private stack with outgoing stack arguments.

Quoted 34 lines
tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c new file +28 -0
new file mode 100644
index 0000000000000..918c41d6b179c
@@ -0,0 +1,84 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include <network_helpers.h>
+#include "private_stack_runtime.h"
+#include "private_stack_runtime.skel.h"
+
+void serial_test_private_stack_runtime(void)
+{
+#if defined(__riscv) && __riscv_xlen == 64
+	LIBBPF_OPTS(bpf_test_run_opts, opts,
+		    .data_in = &pkt_v4,
+		    .data_size_in = sizeof(pkt_v4),
+	);
+	struct private_stack_runtime *skel;
+	struct bpf_link *trace = NULL, *ops = NULL;
+	cpu_set_t saved, single;
+	int cpu, err, tested = 0;
+
+	/*
+	 * Two independent halves. RUN_TESTS() validates the __jited patterns
+	 * through the test loader, which builds its own object from the ELF
+	 * bytes. The half below loads a separate skeleton to check the runtime
+	 * values and the CPU the program reported, which disassembly patterns
+	 * cannot express.
+	 */
+	RUN_TESTS(private_stack_runtime);
+
+	if (!ASSERT_OK(sched_getaffinity(0, sizeof(saved), &saved), "getaffinity"))

tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c line 38

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c new file +10 -0
+		return;
+
+	skel = private_stack_runtime__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+	if (!skel->rodata->has_stack_args) {
+		printf("%s:SKIP: clang lacks __BPF_FEATURE_STACK_ARGUMENT\n", __func__);
+		test__skip();
+		goto out;
+	}

[Severity: Medium]
In tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c, serial_test_private_stack_runtime() calls RUN_TESTS(private_stack_runtime) before opening the skeleton and checking skel->rodata->has_stack_args.

When the BPF compiler does not define __BPF_FEATURE_STACK_ARGUMENT (such as bpf-gcc or an older Clang), tools/testing/selftests/bpf/progs/private_stack_runtime.c compiles run_args() as an empty stub and omits the __arch_riscv64 and __jited() annotations on private_args and kernel_args without adding a __skip() annotation:

tools/testing/selftests/bpf/progs/private_stack_runtime.c:

    #else
    const volatile bool has_stack_args = false;

    static __always_inline void run_args(struct private_stack_result *result)
    {
    }
    #endif

    SEC("raw_tp/sys_enter")
    #ifdef __BPF_FEATURE_STACK_ARGUMENT
    __arch_riscv64
    ...
    __jited("	ld	{{.*}}, 0x8(s0)")
    #endif

    int private_args(void *ctx)

When RUN_TESTS(private_stack_runtime) runs in that configuration, parse_test_spec() and run_subtest() treat private_args, kernel_args, and zero_stack as three valid subtests with no __jited() expectations, load all three, and record 3 passing subtests before resetting env.subtest_state to NULL.

Execution then reaches the !skel->rodata->has_stack_args check and calls test__skip() outside any subtest, incrementing test_state->skip_cnt to 1 while test_state->subtest_num is 3. Because skipped_cnt != subtests_cnt in print_test_result(), doesn't this run three vacuous subtests and report the test as OK (SKIP: 1/3) instead of skipping it when the compiler lacks __BPF_FEATURE_STACK_ARGUMENT? Could the has_stack_args check be performed before RUN_TESTS(private_stack_runtime), or should the programs be annotated with __skip() when __BPF_FEATURE_STACK_ARGUMENT is undefined?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#8 lore raw

Re: [PATCH v2 bpf-next 3/3] selftests/bpf: Add a runtime test for private stack and stack arguments

In reply to Chen Pei
TL;DR: Commit message only: it may be inaccurate in saying fentry/fexit tests are left at NO_PRIV_STACK, since tracing programs that recurse are eligible for a private stack; the review suggests…
Quoted 1 line

selftests/bpf: Add a runtime test for private stack and stack arguments

TL;DR: Commit message only: it may be inaccurate in saying fentry/fexit tests are left at NO_PRIV_STACK, since tracing programs that recurse are eligible for a private stack; the review suggests rewording.

Quoted 31 lines

selftests/bpf: Add a runtime test for private stack and stack arguments

No existing test reaches the combination. bpf_enable_priv_stack() grants an adaptive private stack only to kprobe, tracepoint, perf_event and raw_tracepoint programs, and to tracing, lsm and struct_ops ones that request it or recurse. The stack-argument tests run from tc, socket, syscall and fentry/fexit, all left at NO_PRIV_STACK; the private-stack tests call subprograms of up to three arguments, which stay in registers. Neither pairs a private stack with outgoing stack arguments, where the caller writes the argument area relative to SP and the callee reads it relative to the frame pointer.

Add a runtime test for that combination. A static ten-argument callee writes through a pointer into the caller frame and keeps a 128-byte volatile local array, so its own depth crosses the 64-byte threshold at which the verifier grants a subprog the adaptive private stack. The same workload runs from a raw tracepoint, which gets a private stack, and from tc, which does not; the test pins itself to each allowed CPU in turn and checks the CPU the program reported.

A runtime result cannot show where the three stack areas landed, so the riscv64 __jited patterns pin the layout: the caller's kernel frame is 48 bytes against the control group's 176 and the callee's 64 against 208, the 128 and 144 byte differences being exactly the BPF stack that moved to the per-CPU area. Both groups write the two outgoing arguments at 0x0(sp) and 0x8(sp) and the callee reads them back at 0x0(s0) and 0x8(s0), so the outgoing area, the saved registers and the BPF stack occupy disjoint ranges. None of the patterns assert the per-CPU lookup, which is only emitted on CONFIG_SMP, so one set matches SMP and UP alike.

Quoted 6 lines

The zero-depth struct_ops program is a control case for the kernel stack path, not a private-stack allocation: the verifier grants the adaptive private stack only once a subprog's stack depth, rounded up to the JIT's 16-byte granularity, reaches 64 bytes.

The commit message says the stack-argument tests run from "tc, socket, syscall and fentry/fexit, all left at NO_PRIV_STACK". Is that accurate for fentry/fexit?

Those are BPF_PROG_TYPE_TRACING programs, and bpf_enable_priv_stack() does:

    case BPF_PROG_TYPE_TRACING:
    case BPF_PROG_TYPE_LSM:
    case BPF_PROG_TYPE_STRUCT_OPS:
        if (prog->aux->priv_stack_requested || bpf_prog_check_recur(prog))
            return PRIV_STACK_ADAPTIVE;

bpf_prog_check_recur() returns true for TRACING unless expected_attach_type is BPF_TRACE_ITER, so the fentry/fexit tracers in progs/tracing_stack_arg_tailcall_tracer.c and progs/tracing_struct_int128_tracer.c get PRIV_STACK_ADAPTIVE at the program level. That is the "tracing ... ones that ... recurse" case named in the preceding sentence.

The real reasons those tracers never pair a private stack with stack arguments seem to be different. check_max_stack_depth_subprog() only gives a subprog PRIV_STACK_ADAPTIVE when its rounded depth reaches BPF_PRIV_STACK_MIN_SIZE (64), and these programs only copy a few ctx[] slots into globals. They also make no outgoing stack-argument calls; the stack arguments belong to the traced syscall-program target.

Could the sentence say that the fentry/fexit tracers are eligible for a private stack, but stay below the 64-byte threshold and make no stack-argument calls?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/41813628482

#9 lore raw

Re: [PATCH v2 bpf-next 3/3] selftests/bpf: Add a runtime test for private stack and stack arguments

In reply to Chen Pei
Quoted 4 lines · Chen Pei
 .../selftests/bpf/private_stack_runtime.h     |  12 ++
 .../bpf/prog_tests/private_stack_runtime.c    |  84 +++++++++++
 .../bpf/progs/private_stack_runtime.c         | 140 ++++++++++++++++++
 3 files changed, 236 insertions(+)

bpf ci is right in the other email.

pw-bot: cr

Also, the test runs on riscv64 only. BPF CI runs arm64 and not riscv64. arm64 has the same combination and no test for it.
The run time part has nothing specific to riscv64.
The __jited patterns pin the frame sizes that clang picks for C code. zero_stack doesn't use the private stack.
It's the only reason for serial_.

Two asm progs in verifier_stack_arg.c are enough. BPF_PROG_TEST_RUN runs a raw_tp prog. No need to attach it. The test runs on x86-64, arm64 and riscv64.
On x86-64 both frames stay on the kernel stack.

Like (not even compile tested):

tools/testing/selftests/bpf/progs/verifier_stack_arg.c +47 -0
index abf319c..9423048 100644
@@ -667,6 +667,53 @@ __naked void stack_arg_precision_fallback(void)
 	);
 }
 
+/* Writes both ends of its frame, then reads args 9, 10 and the frame of the caller. */
+__naked __noinline __used
+static long priv_stack_10args(long a, long b, long c, long d, long e, long f, long g, long h,
+			      long i, long *j)
+{
+	asm volatile (
+		"r1 = 0;"
+		"*(u64 *)(r10 - 8) = r1;"
+		"*(u64 *)(r10 - 64) = r1;"
+		"r0 = *(u64 *)(r11 + 32);"
+		"r1 = *(u64 *)(r11 + 40);"
+		"r1 = *(u64 *)(r1 + 0);"
+		"r0 += r1;"
+		"exit;"
+	);
+}
+
+/*
+ * A frame of 64 bytes in a raw_tp prog goes to the private stack.
+ * arm64 and riscv64 pass args 9 and 10 on the kernel stack.
+ */
+SEC("raw_tp")
+__description("stack_arg: private stack in the caller and in the callee")
+__success __retval(890)
+__naked void stack_arg_priv_stack(void)
+{
+	asm volatile (
+		"r1 = 800;"
+		"*(u64 *)(r10 - 64) = r1;"
+		"r1 = r10;"
+		"r1 += -64;"
+		"*(u64 *)(r11 - 40) = r1;"
+		"*(u64 *)(r11 - 32) = 90;"
+		"*(u64 *)(r11 - 24) = 8;"
+		"*(u64 *)(r11 - 16) = 7;"
+		"*(u64 *)(r11 - 8) = 6;"
+		"r1 = 1;"
+		"r2 = 2;"
+		"r3 = 3;"
+		"r4 = 4;"
+		"r5 = 5;"
+		"call priv_stack_10args;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
 #else
 
 SEC("socket")
author
#10 lore raw

Re: [PATCH v2 bpf-next 3/3] selftests/bpf: Add a runtime test for private stack and stack arguments

Hi Alexei,

Thanks for the review and for writing out the two programs.

Quoted 1 line

bpf ci is right in the other email.

Okay, will fix it in the next version.

Quoted 11 lines

Also, the test runs on riscv64 only. BPF CI runs arm64 and not riscv64. arm64 has the same combination and no test for it.
The run time part has nothing specific to riscv64.
The __jited patterns pin the frame sizes that clang picks for C code. zero_stack doesn't use the private stack.
It's the only reason for serial_.

Two asm progs in verifier_stack_arg.c are enough. BPF_PROG_TEST_RUN runs a raw_tp prog. No need to attach it. The test runs on x86-64, arm64 and riscv64.
On x86-64 both frames stay on the kernel stack.

Agreed, will redo the test this way in the next version.

Best regards,
Chen