Skip to contentSkip to conversation

riscv: bpf: Add BPF private stack support

PATCH Superseded Chen Pei 3 participants 9 messages Oct 15

Subsystems (6)
Kselftest BPF RISC-V BPF selftests +2 more subsystems

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Chen Pei

author
#0 lore raw

cover riscv: bpf: Add BPF private stack support

This series adds private-stack support to the RV64 BPF JIT.

The verifier rejects a struct_ops program that requests a private stack with -EACCES when the JIT does not report bpf_jit_supports_private_stack(), and sched_ext requests one for its dispatch callback, so no sched_ext scheduler can load on riscv64 today.

Allocation, the 16-byte guard regions at both ends, the guard value and the lifetime handling follow arm64, x86-64 and powerpc64. What is specific to riscv64 is where the pointers end up:

- Every BPF stack access goes through BPF_REG_FP (S5), so pointing S5 at the top of the per-CPU area and leaving SP on the kernel stack is all the JIT has to do. S5 is callee-saved and each subprog derives its own, so bpf2bpf calls need no extra save and restore.
- The outgoing stack-argument area stays in the kernel frame: callers write it relative to SP while callees read it relative to RV_REG_FP. Unlike x86-64, no register here is shared with the private-stack frame pointer, so no verifier-side exclusion is needed and stack arguments keep working together with a private stack. arm64 already does the same, and patch 3 measures the three ranges at run time and pins them in the disassembly.
- The per-CPU lookup is emitted after the tail-call entry point. A program using a private stack can still be entered as a tail-call target, at RV_TAILCALL_OFFSET, so it has to set up its own S5 there; and since that offset is a constant added to the target's bpf_func, everything in front of it must be the same size in every program.

The base address is materialized with absolute moves, as on arm64, x86-64 and powerpc64. Those three reach the per-CPU offset through gs:, TPIDR or r13 and only have to move the base into a register; riscv64 has no such register, so the address of __per_cpu_offset is loaded here too. A PC-relative pair is not an option for the base itself: the dynamic per-CPU area is vmalloc'd, further from the JIT region than auipc reaches, which is why the whole setup is ten instructions on the sv39 kernel tested here. __per_cpu_offset is a kernel-image symbol and could use the emit_addr() form that insn_is_mov_percpu_addr() already uses, at the same two instructions; unifying the two needs an extra_pass argument and an error return in bpf_jit_build_prologue(), which also touches bpf_jit_comp32.c, so it is left as is. Both constants are fixed before the offset-convergence loop, so the prologue length is stable across passes.

RV32 does not declare support and keeps the existing behaviour; the prologue signature is unchanged, so bpf_jit_comp32.c needs no edit.

Testing
=======
QEMU virt rv64, an 8 vCPU SMP kernel and a 1 vCPU UP kernel, source tree clean at the head of this series, selftests built from these patches, no private-stack related failures. On UP, kfunc_call/subprog, kfunc_call/subprog_lskel and scx_simple are not run: all three fail the same way on the unpatched base.

- verifier_private_stack, struct_ops_private_stack, private_stack_runtime
- tailcalls, stack_arg, subprogs, fentry_fexit, kfunc_call, percpu_alloc, async_stack_depth, arena_atomics, arena_ldsx
- may_goto_priv_stack, may_goto_far, verifier_may_goto_1, verifier_may_goto_2
- scx_simple on SMP

Chen Pei (3):
riscv: bpf: Add BPF private stack support selftests/bpf: Enable private stack tests for riscv64 selftests/bpf: Add a runtime test for private stack and stack
arguments

 arch/riscv/net/bpf_jit.h                      |   1 +
 arch/riscv/net/bpf_jit_comp64.c               |  42 +++++-
 arch/riscv/net/bpf_jit_core.c                 |  84 +++++++++--
 .../selftests/bpf/private_stack_runtime.h     |  12 ++
 .../bpf/prog_tests/private_stack_runtime.c    |  83 +++++++++++
 .../bpf/prog_tests/struct_ops_private_stack.c |   3 +-
 .../bpf/progs/private_stack_runtime.c         | 140 ++++++++++++++++++
 .../bpf/progs/verifier_private_stack.c        |  41 ++++-
 8 files changed, 392 insertions(+), 14 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/private_stack_runtime.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c
 create mode 100644 tools/testing/selftests/bpf/progs/private_stack_runtime.c
Signature · 1 line
2.50.1
author
#1 lore raw

PATCH 1/3 riscv: bpf: Add BPF private stack support

A program on a private stack keeps its BPF frames in a per-CPU area of its own rather than on the kernel stack of whichever context entered it. bpf_enable_priv_stack() grants that mode only once the JIT reports bpf_jit_supports_private_stack(), which riscv64 did not: programs there always ran on the kernel stack, and a struct_ops program that requires one was rejected with -EACCES.

Allocate a per-CPU area of the verifier calculated depth plus a 16 byte guard region at each end, and point BPF_REG_FP (S5) at the top of it while leaving SP on the kernel stack. Every BPF stack access already goes through S5, and S5 is callee-saved with each subprog deriving its own, so nothing else in the JIT has to know about the private stack.

The outgoing stack argument area stays in the kernel frame: callers write it relative to SP, callees relative to RV_REG_FP, which the prologue sets to the caller's SP.

The per-CPU lookup is emitted after the tail call entry point: a program using a private stack can still be entered as a tail call target, and RV_TAILCALL_OFFSET is a constant added to the target's bpf_func, so what is in front of it must be the same size in every program.

The base is materialized with absolute moves, as on arm64, x86-64 and
powerpc64: the dynamic per-CPU area is vmalloc'd, beyond a PC-relative auipc pair from the JIT region, and riscv64 has no segment or register holding the per-CPU offset, so __per_cpu_offset needs materializing the same way. Both constants are fixed before the offset convergence loop, so the prologue length is stable across passes.

The guard layout and value follow arm64, x86-64 and powerpc64: a 16 byte region at each end holding two copies of the pattern. A jited program has them checked when it is freed, reporting the first corrupted CPU only; failure paths release the area unchecked since the image never ran, which also clears priv_stack_ptr for the later bpf_jit_free().

RV32 does not declare support and keeps the existing kernel stack behaviour.

---
 arch/riscv/net/bpf_jit.h        |  1 +
 arch/riscv/net/bpf_jit_comp64.c | 42 +++++++++++++++--
 arch/riscv/net/bpf_jit_core.c   | 84 +++++++++++++++++++++++++++++----
 3 files changed, 115 insertions(+), 12 deletions(-)
arch/riscv/net/bpf_jit.h +1 -0
index 5c4f53060c91..6d42083829f7 100644
@@ -14,6 +14,7 @@
 
 #define RV_MAX_REG_ARGS		8
 #define RV_EXTRA_STK_ARGS	(RV_MAX_REG_ARGS - MAX_BPF_FUNC_REG_ARGS)
+#define PRIV_STACK_GUARD_SZ	16
 
 /* verify runtime detection extension status */
 #define rv_ext_enabled(ext) \
arch/riscv/net/bpf_jit_comp64.c +39 -3

61 lines not on this page · raw message

arch/riscv/net/bpf_jit_core.c +75 -9

136 lines not on this page · raw message

Signature · 1 line
2.50.1
#2 lore raw
TL;DR: Commit message only: it says the guard layout follows x86-64, but x86-64 uses an 8 byte single-copy guard, and it may overstate why __per_cpu_offset needs absolute moves.
Quoted 1 line

riscv: bpf: Add BPF private stack support

TL;DR: Commit message only: it says the guard layout follows x86-64, but x86-64 uses an 8 byte single-copy guard, and it may overstate why __per_cpu_offset needs absolute moves.

Quoted 1 line

riscv: bpf: Add BPF private stack support

[ ... ]

Quoted 7 lines

The base is materialized with absolute moves, as on arm64, x86-64 and
powerpc64: the dynamic per-CPU area is vmalloc'd, beyond a PC-relative auipc pair from the JIT region, and riscv64 has no segment or register holding the per-CPU offset, so __per_cpu_offset needs materializing the same way. Both constants are fixed before the offset convergence loop, so the prologue length is stable across passes.

Quoted 6 lines

The guard layout and value follow arm64, x86-64 and powerpc64: a 16 byte region at each end holding two copies of the pattern. A jited program has them checked when it is freed, reporting the first corrupted CPU only; failure paths release the area unchecked since the image never ran, which also clears priv_stack_ptr for the later bpf_jit_free().

Is the statement that the guard layout follows x86-64 accurate?

arm64 and powerpc64 use a 16 byte guard with two copies of the pattern at each end, but x86-64 differs:

arch/x86/net/bpf_jit_comp.c:

    #define PRIV_STACK_GUARD_SZ    8
    ...
    stack_ptr[0] = PRIV_STACK_GUARD_VAL;
    stack_ptr[underflow_idx] = PRIV_STACK_GUARD_VAL;

It uses an 8 byte guard with a single copy at each end, and it rounds the stack depth to 8 rather than 16. Only the guard value (0xEB9F12345678eb9f) is shared by all three. Could the changelog say that the 16 byte, two-copy layout follows arm64 and powerpc64, and that only the guard value follows x86-64?

Separately, the changelog says __per_cpu_offset "needs materializing the same way", meaning with absolute moves. The existing BPF_MOV64_PERCPU_REG handler in bpf_jit_emit_insn() already loads the same symbol PC-relatively:

    /* Load address of __per_cpu_offset array in T2 */
    emit_addr(RV_REG_T2, (u64)&__per_cpu_offset, extra_pass, ctx);

Is emit_imm() for __per_cpu_offset in emit_priv_stack_frame() required, or a choice made so that the prologue length stays fixed across passes? The changelog could say which.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/38671177164

author
#3 lore raw

PATCH 2/3 selftests/bpf: Enable private stack tests for riscv64

In reply to Chen Pei

The RV64 JIT now supports the private stack. Allow the existing verifier and struct_ops private-stack tests to run on riscv64 and add the expected RV64 JITed instruction patterns.

---
 .../bpf/prog_tests/struct_ops_private_stack.c |  3 +-
 .../bpf/progs/verifier_private_stack.c        | 41 ++++++++++++++++++-
 2 files changed, 42 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c +2 -1
index 2b3ec2b79091..103179ff8733 100644
@@ -6,7 +6,8 @@
 #include "struct_ops_private_stack_recur.skel.h"
 #include "struct_ops_private_stack_large.skel.h"
 
-#if defined(__x86_64__) || defined(__aarch64__) || defined(__powerpc64__)
+#if defined(__x86_64__) || defined(__aarch64__) || defined(__powerpc64__) || \
+	(defined(__riscv) && __riscv_xlen == 64)
 static void test_private_stack(void)
 {
 	struct struct_ops_private_stack *skel;
tools/testing/selftests/bpf/progs/verifier_private_stack.c +40 -1

83 lines not on this page · raw message

Signature · 1 line
2.50.1
#4 lore raw

Re: [PATCH bpf-next 2/3] selftests/bpf: Enable private stack tests for riscv64

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] bpf: Missing enablement for private stack test in verifier_large_stack.c for riscv64 --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] bpf: Missing enablement for private stack test in verifier_large_stack.c for riscv64 --

commit 6110b152bad2869b21761f9abd7de948a19afdbf
Author: Chen Pei <[email protected]>

selftests/bpf: Enable private stack tests for riscv64

This commit enables the verifier and struct_ops private-stack tests to run on riscv64 by adding the expected RV64 JITed instruction patterns.

Quoted 5 lines

selftests/bpf: Enable private stack tests for riscv64

The RV64 JIT now supports the private stack. Allow the existing verifier and struct_ops private-stack tests to run on riscv64 and add the expected RV64 JITed instruction patterns.

[Severity: Medium]
Was the private_stack_two_frames test in verifier_large_stack.c missed in this patch?

The commit enables private-stack tests in verifier_private_stack.c and struct_ops_private_stack.c, but verifier_large_stack.c contains a test that explicitly validates the verifier log messages for a 2048-byte large stack (private stack) setup, which is currently restricted to x86_64 and arm64:

tools/testing/selftests/bpf/progs/verifier_large_stack.c:
__arch_x86_64
__arch_arm64
__success __log_level(4)
__msg("stack depth max 2048")
__msg("subprog 0 (private_stack_two_frames) main {{.*}} stack 2048") __msg("subprog 1 (priv_stack_frame_2048) static {{.*}} stack 2048") __naked void private_stack_two_frames(void)

Since the RV64 JIT now supports the private stack, should __arch_riscv64 be added to this test as well so that the riscv64 architecture receives this large stack test coverage?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author
#5 lore raw

Re: [PATCH bpf-next 2/3] selftests/bpf: Enable private stack tests for riscv64

This one I am not taking: what gates the test is large stack support, a JIT capability separate from the private stack this series adds.

Quoted 7 lines

[Severity: Medium]
Was the private_stack_two_frames test in verifier_large_stack.c missed in this patch?

Since the RV64 JIT now supports the private stack, should __arch_riscv64 be added to this test as well so that the riscv64 architecture receives this large stack test coverage?

The test builds a 2048 byte frame. Above MAX_BPF_STACK that is only accepted when bpf_prog_stack_limit() returns MAX_BPF_STACK_JIT, which requires bpf_jit_supports_large_stack() in the JIT as well as bpf_jit_supports_subprog_tailcalls(); the former exists only for x86-64 and arm64. riscv64 rejects the program on the depth with or without a private stack, which is why it skips behind is_large_stack_supported(). Adding __arch_riscv64 would turn that skip into a failure.

The other comments on this set are valid and are taken in v2.

Best regards,
Chen

author
#6 lore raw

PATCH 3/3 selftests/bpf: Add a runtime test for private stack and stack arguments

In reply to Chen Pei

The stack-argument tests all run from SEC("tc"), which bpf_enable_priv_stack() leaves at NO_PRIV_STACK, and the private-stack tests only call subprograms of up to three arguments, which stay in registers. Nothing exercises a private stack together with outgoing stack arguments, where the caller writes the argument area relative to SP and the callee reads it relative to the frame pointer.

Add a runtime test for that combination. A static ten-argument callee writes through a pointer into the caller frame and keeps a 128-byte volatile local array, so its own depth crosses the 64-byte threshold at which the verifier grants a subprog the adaptive private stack. The same workload runs from a raw tracepoint, which gets a private stack, and from tc, which does not; the test pins itself to each allowed CPU in turn and checks the CPU the program reported.

A runtime result cannot show where the three stack areas landed, so the riscv64 __jited patterns pin the layout: the caller's kernel frame is 48 bytes against the control group's 176 and the callee's 64 against 208, the 128 and 144 byte differences being exactly the BPF stack that moved to the per-CPU area. Both groups write the two outgoing arguments at 0x0(sp) and 0x8(sp) and the callee reads them back at 0x0(s0) and 0x8(s0), so the outgoing area, the saved registers and the BPF stack occupy disjoint ranges. None of the patterns assert the per-CPU lookup, which is only emitted on CONFIG_SMP, so one set matches SMP and UP alike.

The zero-depth struct_ops program is a control case for the kernel stack path, not a private-stack allocation: the verifier grants the adaptive private stack only once a subprog's stack depth, rounded up to the JIT's 16-byte granularity, reaches 64 bytes.

---

Notes:
Reachability of a private stack together with outgoing stack arguments, per architecture, for review context only:

- x86-64 cannot reach it. The verifier denies the private stack to subprograms with stack arguments there because R9 doubles as the private stack frame pointer and argument 6, so both halves of this test would be kernel stack runs.
- powerpc64 has no stack arguments at all, as it does not report bpf_jit_supports_stack_args(), so it cannot reach it either.
- arm64 reports both bpf_jit_supports_private_stack() and bpf_jit_supports_stack_args(), so the combination is reachable there today and has no test for it. Widening the gate to arm64 needs arm64 __jited patterns and an arm64 machine to validate them, so I would post that separately once the maintainers agree this runtime test is worth keeping.

 .../selftests/bpf/private_stack_runtime.h     |  12 ++
 .../bpf/prog_tests/private_stack_runtime.c    |  83 +++++++++++
 .../bpf/progs/private_stack_runtime.c         | 140 ++++++++++++++++++
 3 files changed, 235 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/private_stack_runtime.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c
 create mode 100644 tools/testing/selftests/bpf/progs/private_stack_runtime.c
tools/testing/selftests/bpf/private_stack_runtime.h new file +12 -0
new file mode 100644
index 000000000000..8ed49ff66a6e
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __PRIVATE_STACK_RUNTIME_H
+#define __PRIVATE_STACK_RUNTIME_H
+
+struct private_stack_result {
+	long value;
+	long first;
+	long last;
+	__u32 cpu;
+};
+
+#endif
tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c new file +83 -0

83 lines not on this page · raw message

tools/testing/selftests/bpf/progs/private_stack_runtime.c new file +140 -0

140 lines not on this page · raw message

Signature · 1 line
2.50.1
#7 lore raw

Re: [PATCH bpf-next 3/3] selftests/bpf: Add a runtime test for private stack and stack arguments

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] bpf: test flakiness due to parallel struct_ops attachment --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] bpf: test flakiness due to parallel struct_ops attachment
--

commit 30217e3723f8427977b857ce4b668a913d8e6da3
Author: Chen Pei <[email protected]>
selftests/bpf: Add a runtime test for private stack and stack arguments

This commit adds a runtime test for combining private stacks with outgoing stack arguments. It validates the structural layout via disassembly patterns and checks runtime execution context across CPUs. It also includes a struct_ops test to act as a kernel stack control case.

Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c new file +0 -0
new file mode 100644
index 0000000000000..3af24ca8009f4

tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c line 8

Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c new file +8 -0
@@ -0,0 +1,83 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include <network_helpers.h>
+#include "private_stack_runtime.h"
+#include "private_stack_runtime.skel.h"
+
+void test_private_stack_runtime(void)
+{

[Severity: Medium]
Will this cause test flakiness in CI due to parallel execution?

The test function lacks the serial_ prefix, so it will run concurrently in the test_progs worker threads rather than in isolation.

[ ... ]

tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c line 72

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c new file +5 -0
+	ASSERT_GT(tested, 0, "tested_cpus");
+
+	ops = bpf_map__attach_struct_ops(skel->maps.zero_stack_ops);
+	if (!ASSERT_OK_PTR(ops, "attach_zero_stack"))
+		goto out;

[Severity: Medium]
Since zero_stack_ops targets the singleton bpf_testmod_ops3 in bpf_testmod, executing this concurrently with other non-serial tests that attach to the same struct_ops (such as test_struct_ops_private_stack) can cause random -EEXIST failures.

st_ops3_reg() in bpf_testmod.c enforces a single active attachment:

    if (st_ops3)
        ... err = -EEXIST

Could this function be renamed to serial_test_private_stack_runtime to avoid resource conflicts with other tests?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#8 lore raw

Re: [PATCH bpf-next 3/3] selftests/bpf: Add a runtime test for private stack and stack arguments

In reply to Chen Pei
TL;DR: Test issue only: the new private stack selftest calls test__skip() without printing a reason, so a SKIP doesn't show why. The review says this may be acceptable, since other tests do the same.
Quoted 1 line

selftests/bpf: Add a runtime test for private stack and stack arguments

TL;DR: Test issue only: the new private stack selftest calls test__skip() without printing a reason, so a SKIP doesn't show why. The review says this may be acceptable, since other tests do the same.

Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c new file +0 -0
new file mode 100644
index 000000000000..3af24ca8009f
@@ -0,0 +1,83 @@

[ ... ]

tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c line 37

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c new file +4 -0
+	if (!skel->rodata->has_stack_args) {
+		test__skip();
+		goto out;
+	}

A subsystem pattern flags this as potentially concerning: a selftest that skips without printing a reason. Both skip paths call test__skip() bare, this one when the compiler lacks __BPF_FEATURE_STACK_ARGUMENT (has_stack_args is false), and the other in the non-riscv64 branch:

    #else
        test__skip();
    #endif

Without a message, the test shows up as SKIP with no hint whether clang needs to be upgraded or the test is only built for riscv64. Would a printf() before test__skip(), like the "%s:SKIP: ..." message that test_loader's run_subtest() prints, make the cause visible?

This may be acceptable as is: test_progs does report the test as SKIP, and prog_tests/stack_arg.c and prog_tests/struct_ops_private_stack.c also call test__skip() without a message.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/38671177164