Skip to contentSkip to conversation

powerpc64/bpf: Implement JIT support for private stack

PATCH Superseded [email protected] 2 participants 3 messages Feb 25

Subsystems (5)
PowerPC Kselftest BPF BPF selftests +1 more subsystems

v3 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/2 powerpc64/bpf: Implement JIT support for private stack

From: Abhishek Dubey <[email protected]>

Provision the private stack as a per-CPU allocation during bpf_int_jit_compile(). Align the stack to 16 bytes and place guard regions at both ends to detect runtime stack overflow and underflow.

Round the private stack size up to the nearest 16-byte boundary. Make each guard region 16 bytes to preserve the required overall 16-byte alignment. When private stack is set, skip bpf stack size accounting in kernel stack.

There is no stack pointer in powerpc. Stack referencing during JIT is done using frame pointer. Frame pointer calculation goes like:

BPF frame pointer = Priv stack allocation start address +
Overflow guard +
Actual stack size defined by verifier

Memory layout:

High Addr          +--------------------------------------------------+
                   |                                                  |
                   | 16 bytes Underflow guard (0xEB9F12345678eb9fULL) |
                   |                                                  |
         BPF FP -> +--------------------------------------------------+
                   |                                                  |
                   | Private stack - determined by verifier           |
                   | 16-bytes aligned                                 |
                   |                                                  |
                   +--------------------------------------------------+
                   |                                                  |
Lower Addr         | 16 byte Overflow guard (0xEB9F12345678eb9fULL)   |
                   |                                                  |

Priv stack alloc ->+--------------------------------------------------+ start

Update BPF_REG_FP to point to the calculated offset within the allocated private stack buffer. Now, BPF stack usage reference in the allocated private stack.

The patch is rebase over fixes by Hari:
https://lore.kernel.org/bpf/[email protected]/

v1->v2:
Fix ci-bot warning for percpu pointer casting Minor refactoring
v2->v3:
Fix ci-bot bug targeting clobbered NVRs on stack rollback

[v2]: https://lore.kernel.org/bpf/[email protected]
[v1]: https://lore.kernel.org/bpf/[email protected]

---
 arch/powerpc/net/bpf_jit.h        |  5 ++
 arch/powerpc/net/bpf_jit_comp.c   | 87 +++++++++++++++++++++++++++++--
 arch/powerpc/net/bpf_jit_comp64.c | 29 +++++++++--
 3 files changed, 113 insertions(+), 8 deletions(-)
arch/powerpc/net/bpf_jit.h +5 -0
index 7354e1d72f79..eb0a400b5a98 100644
@@ -178,8 +178,13 @@ struct codegen_context {
 	bool is_subprog;
 	bool exception_boundary;
 	bool exception_cb;
+	void __percpu *priv_sp;
 };
 
+/* Memory size & magic-value to detect private stack overflow/underflow */
+#define PRIV_STACK_GUARD_SZ    16
+#define PRIV_STACK_GUARD_VAL   0xEB9F12345678eb9fULL
+
 #define bpf_to_ppc(r)	(ctx->b2p[r])
 
 #ifdef CONFIG_PPC32
arch/powerpc/net/bpf_jit_comp.c +82 -5

143 lines not on this page · raw message

arch/powerpc/net/bpf_jit_comp64.c +26 -3
index 640b84409687..65ab8b7a7f82 100644
@@ -183,6 +183,22 @@ void bpf_jit_realloc_regs(struct codegen_context *ctx)
 {
 }
 
+static void emit_fp_priv_stack(u32 *image, struct codegen_context *ctx)
+{
+	/* Load percpu data offset */
+	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R13,
+			offsetof(struct paca_struct, data_offset)));
+	PPC_LI64(bpf_to_ppc(BPF_REG_FP), (__force long)ctx->priv_sp);
+	/*
+	 * Load base percpu pointer of private stack allocation.
+	 * Runtime per-cpu address = (base + data_offset) + (guard + stack_size)
+	 */
+	EMIT(PPC_RAW_ADD(bpf_to_ppc(BPF_REG_FP),
+			bpf_to_ppc(TMP_REG_1), bpf_to_ppc(BPF_REG_FP)));
+	EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), bpf_to_ppc(BPF_REG_FP),
+			PRIV_STACK_GUARD_SZ + round_up(ctx->stack_size, 16)));
+}
+
 /*
  * For exception boundary & exception_cb progs:
  *     return increased size to accommodate additional NVRs.
@@ -307,9 +323,16 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 	 * Exception_cb not restricted from using stack area or arena.
 	 * Setup frame pointer to point to the bpf stack area
 	 */
-	if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
-		EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
-			STACK_FRAME_MIN_SIZE + ctx->stack_size));
+	if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP))) {
+		if (ctx->priv_sp) {
+			/* Set up fp in private stack */
+			emit_fp_priv_stack(image, ctx);
+		} else {
+			/* Setup frame pointer to point to the bpf stack area */
+			EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
+				STACK_FRAME_MIN_SIZE + ctx->stack_size));
+		}
+	}
 
 	if (ctx->arena_vm_start)
 		PPC_LI64(bpf_to_ppc(ARENA_VM_START), ctx->arena_vm_start);
Signature · 1 line
2.52.0
#1 lore raw

PATCH v3 2/2 selftests/bpf: Enable private stack tests for powerpc64

From: Abhishek Dubey <[email protected]>

With support of private stack, relevant tests must pass on powerpc64.

#./test_progs -t struct_ops_private_stack

#434/1   struct_ops_private_stack/private_stack:OK
#434/2   struct_ops_private_stack/private_stack_fail:OK
#434/3   struct_ops_private_stack/private_stack_recur:OK
#434     struct_ops_private_stack:OK

Summary: 1/3 PASSED, 0 SKIPPED, 0 FAILED

v2->v3:
Enable testcase using __powerpc64__ instead of __TARGET_ARCH_powerpc, to prevent it getting invoked on powerpc32

[v2]: https://lore.kernel.org/bpf/[email protected]

---
 .../bpf/prog_tests/struct_ops_private_stack.c | 29 +++++++++----------
 .../bpf/progs/struct_ops_private_stack.c      |  6 ----
 .../bpf/progs/struct_ops_private_stack_fail.c |  6 ----
 .../progs/struct_ops_private_stack_recur.c    |  6 ----
 4 files changed, 13 insertions(+), 34 deletions(-)
tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c +13 -16
index 4006879ca3fe..afad3522b772 100644
@@ -5,6 +5,7 @@
 #include "struct_ops_private_stack_fail.skel.h"
 #include "struct_ops_private_stack_recur.skel.h"
 
+#if defined(__x86_64__) || defined(__aarch64__) || defined(__powerpc64__)
 static void test_private_stack(void)
 {
 	struct struct_ops_private_stack *skel;
@@ -15,11 +16,6 @@ static void test_private_stack(void)
 	if (!ASSERT_OK_PTR(skel, "struct_ops_private_stack__open"))
 		return;
 
-	if (skel->data->skip) {
-		test__skip();
-		goto cleanup;
-	}
-
 	err = struct_ops_private_stack__load(skel);
 	if (!ASSERT_OK(err, "struct_ops_private_stack__load"))
 		goto cleanup;
@@ -48,11 +44,6 @@ static void test_private_stack_fail(void)
 	if (!ASSERT_OK_PTR(skel, "struct_ops_private_stack_fail__open"))
 		return;
 
-	if (skel->data->skip) {
-		test__skip();
-		goto cleanup;
-	}
-
 	err = struct_ops_private_stack_fail__load(skel);
 	if (!ASSERT_ERR(err, "struct_ops_private_stack_fail__load"))
 		goto cleanup;
@@ -72,11 +63,6 @@ static void test_private_stack_recur(void)
 	if (!ASSERT_OK_PTR(skel, "struct_ops_private_stack_recur__open"))
 		return;
 
-	if (skel->data->skip) {
-		test__skip();
-		goto cleanup;
-	}
-
 	err = struct_ops_private_stack_recur__load(skel);
 	if (!ASSERT_OK(err, "struct_ops_private_stack_recur__load"))
 		goto cleanup;
@@ -95,7 +81,7 @@ static void test_private_stack_recur(void)
 	struct_ops_private_stack_recur__destroy(skel);
 }
 
-void test_struct_ops_private_stack(void)
+static void __test_struct_ops_private_stack(void)
 {
 	if (test__start_subtest("private_stack"))
 		test_private_stack();
@@ -104,3 +90,14 @@ void test_struct_ops_private_stack(void)
 	if (test__start_subtest("private_stack_recur"))
 		test_private_stack_recur();
 }
+#else
+static void __test_struct_ops_private_stack(void)
+{
+	test__skip();
+}
+#endif
+
+void test_struct_ops_private_stack(void)
+{
+	__test_struct_ops_private_stack();
+}
tools/testing/selftests/bpf/progs/struct_ops_private_stack.c +0 -6
index dbe646013811..3cd0c1a55cbd 100644
@@ -7,12 +7,6 @@
 
 char _license[] SEC("license") = "GPL";
 
-#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
-bool skip __attribute((__section__(".data"))) = false;
-#else
-bool skip = true;
-#endif
-
 void bpf_testmod_ops3_call_test_2(void) __ksym;
 
 int val_i, val_j;
tools/testing/selftests/bpf/progs/struct_ops_private_stack_fail.c +0 -6
index 3d89ad7cbe2a..1442728f5604 100644
@@ -7,12 +7,6 @@
 
 char _license[] SEC("license") = "GPL";
 
-#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
-bool skip __attribute((__section__(".data"))) = false;
-#else
-bool skip = true;
-#endif
-
 void bpf_testmod_ops3_call_test_2(void) __ksym;
 
 int val_i, val_j;
tools/testing/selftests/bpf/progs/struct_ops_private_stack_recur.c +0 -6
index b1f6d7e5a8e5..faaa0f8d65a4 100644
@@ -7,12 +7,6 @@
 
 char _license[] SEC("license") = "GPL";
 
-#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
-bool skip __attribute((__section__(".data"))) = false;
-#else
-bool skip = true;
-#endif
-
 void bpf_testmod_ops3_call_test_1(void) __ksym;
 
 int val_i, val_j;
Signature · 1 line
2.52.0
#2 lore raw
In reply to [email protected]
This zeros ctx->stack_size before the JIT passes, which is correct for the kernel stack frame allocation (skipping BPF stack space on the kernel stack).
Quoted 3 lines
arch/powerpc/net/bpf_jit_comp.c +0 -0

[ ... ]

arch/powerpc/net/bpf_jit_comp.c line 269

Quoted 7 lines
arch/powerpc/net/bpf_jit_comp.c +3 -0
@@ -209,6 +264,9 @@
 	cgctx.is_subprog = bpf_is_subprog(fp);
 	cgctx.exception_boundary = fp->aux->exception_boundary;
 	cgctx.exception_cb = fp->aux->exception_cb;
+	cgctx.priv_sp = priv_stack_ptr;
+	if (priv_stack_ptr)
+		cgctx.stack_size = 0;

This zeros ctx->stack_size before the JIT passes, which is correct for the kernel stack frame allocation (skipping BPF stack space on the kernel stack).

However, emit_fp_priv_stack() also reads ctx->stack_size to compute the BPF frame pointer offset within the private stack allocation:

Quoted 3 lines
arch/powerpc/net/bpf_jit_comp64.c +0 -0

[ ... ]

Quoted 5 lines
arch/powerpc/net/bpf_jit_comp64.c +5 -0
+static void emit_fp_priv_stack(u32 *image, struct codegen_context *ctx)
+{
+	/* Load percpu data offset */
+	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R13,
+			offsetof(struct paca_struct, data_offset)));

arch/powerpc/net/bpf_jit_comp64.c void bpf_jit_realloc_regs(struct codegen_context *ctx) · line 200

Quoted 10 lines
arch/powerpc/net/bpf_jit_comp64.c +10 -0
+	PPC_LI64(bpf_to_ppc(BPF_REG_FP), (__force long)ctx->priv_sp);
+	/*
+	 * Load base percpu pointer of private stack allocation.
+	 * Runtime per-cpu address = (base + data_offset) + (guard + stack_size)
+	 */
+	EMIT(PPC_RAW_ADD(bpf_to_ppc(BPF_REG_FP),
+			bpf_to_ppc(TMP_REG_1), bpf_to_ppc(BPF_REG_FP)));
+	EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), bpf_to_ppc(BPF_REG_FP),
+			PRIV_STACK_GUARD_SZ + round_up(ctx->stack_size, 16)));
+}

Since ctx->stack_size was set to 0 earlier in bpf_int_jit_compile(), the ADDI immediate here becomes PRIV_STACK_GUARD_SZ + 0 = 16. This places BPF_REG_FP at the bottom of the usable stack area (just past the overflow guard), rather than at the top.

The commit message describes the intended layout:

BPF frame pointer = Priv stack allocation start address +
Overflow guard +
Actual stack size defined by verifier

But the "Actual stack size defined by verifier" term is lost because ctx->stack_size is already zero at this point.

BPF programs access the stack at FP-8, FP-16, etc. With FP at offset 16 from the allocation base, FP-8 lands inside the overflow guard and larger offsets go below the allocation entirely.

For comparison, x86 in do_jit() pre-computes the offset before zeroing:

    priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ +
                     round_up(stack_depth, 8);
    stack_depth = 0;

And arm64 in build_prologue() does not zero ctx->stack_size at all, instead using it directly:

    priv_stack_ptr = prog->aux->priv_stack_ptr + PRIV_STACK_GUARD_SZ;
    emit_percpu_ptr(priv_sp, priv_stack_ptr, ctx);
    emit(A64_ADD_I(1, fp, priv_sp, ctx->stack_size), ctx);

The v1 of this patch also had it correct, pre-computing in bpf_jit_build_prologue():

    priv_frame_ptr = (void*) ctx->priv_sp + PRIV_STACK_GUARD_SZ +
                     round_up(ctx->stack_size, 16);

It appears the "Minor refactoring" in v1->v2 moved the cgctx.stack_size = 0 earlier in the pipeline and removed the pre-computation step, but did not account for emit_fp_priv_stack() needing the original stack depth.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/22413899411

AI-authorship-score: low
AI-authorship-explanation: Code follows established powerpc JIT patterns with architecture-specific constructs; commit history references specific CI-bot fixes consistent with human iterative development.
issues-found: 1
issue-severity-score: high
issue-severity-explanation: BPF frame pointer is miscalculated due to using zeroed ctx->stack_size in emit_fp_priv_stack(), causing all BPF stack accesses to write outside the allocated private stack region, resulting in memory corruption.