Skip to contentSkip to conversation

powerpc64/bpf: Support tailcalls with subprogs & BPF exceptions

PATCH Superseded [email protected] 5 participants 27 messages Jan 14 – Jan 17 5 messages not in the archive

Subsystems (4)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v2 cover powerpc64/bpf: Support tailcalls with subprogs & BPF exceptions

From: Abhishek Dubey <[email protected]>

This patch series enables support for two BPF JIT features on powerpc64. The first three patches target support for tail calls with subprogram combinations. The first patch supports realignment of tail_call_cnt offset in stack frame. Implementation details are provided in the commit messages.

The last three patches add support for BPF exceptions. An architecture-specific stack walker is implemented to assist with stack walk during exceptions.

All selftests related to tailcalls and exceptions are passing:

# ./test_progs -t tailcalls
#442/1   tailcalls/tailcall_1:OK
#442/2   tailcalls/tailcall_2:OK
#442/3   tailcalls/tailcall_3:OK
#442/4   tailcalls/tailcall_4:OK
#442/5   tailcalls/tailcall_5:OK
#442/6   tailcalls/tailcall_6:OK
#442/7   tailcalls/tailcall_bpf2bpf_1:OK
#442/8   tailcalls/tailcall_bpf2bpf_2:OK
#442/9   tailcalls/tailcall_bpf2bpf_3:OK
#442/10  tailcalls/tailcall_bpf2bpf_4:OK
#442/11  tailcalls/tailcall_bpf2bpf_5:OK
#442/12  tailcalls/tailcall_bpf2bpf_6:OK
#442/13  tailcalls/tailcall_bpf2bpf_fentry:OK
#442/14  tailcalls/tailcall_bpf2bpf_fexit:OK
#442/15  tailcalls/tailcall_bpf2bpf_fentry_fexit:OK
#442/16  tailcalls/tailcall_bpf2bpf_fentry_entry:OK
#442/17  tailcalls/tailcall_poke:OK
#442/18  tailcalls/tailcall_bpf2bpf_hierarchy_1:OK
#442/19  tailcalls/tailcall_bpf2bpf_hierarchy_fentry:OK
#442/20  tailcalls/tailcall_bpf2bpf_hierarchy_fexit:OK
#442/21  tailcalls/tailcall_bpf2bpf_hierarchy_fentry_fexit:OK
#442/22  tailcalls/tailcall_bpf2bpf_hierarchy_fentry_entry:OK
#442/23  tailcalls/tailcall_bpf2bpf_hierarchy_2:OK
#442/24  tailcalls/tailcall_bpf2bpf_hierarchy_3:OK
#442/25  tailcalls/tailcall_freplace:OK
#442/26  tailcalls/tailcall_bpf2bpf_freplace:OK
#442/27  tailcalls/tailcall_failure:OK
#442/28  tailcalls/reject_tail_call_spin_lock:OK
#442/29  tailcalls/reject_tail_call_rcu_lock:OK
#442/30  tailcalls/reject_tail_call_preempt_lock:OK
#442/31  tailcalls/reject_tail_call_ref:OK
#442     tailcalls:OK
Summary: 1/31 PASSED, 0 SKIPPED, 0 FAILED
# ./test_progs -t exceptions
#105/1   exceptions/exception_throw_always_1:OK
#105/2   exceptions/exception_throw_always_2:OK
#105/3   exceptions/exception_throw_unwind_1:OK
#105/4   exceptions/exception_throw_unwind_2:OK
#105/5   exceptions/exception_throw_default:OK
#105/6   exceptions/exception_throw_default_value:OK
#105/7   exceptions/exception_tail_call:OK
#105/8   exceptions/exception_ext:OK
#105/9   exceptions/exception_ext_mod_cb_runtime:OK
#105/10  exceptions/exception_throw_subprog:OK
#105/11  exceptions/exception_assert_nz_gfunc:OK
#105/12  exceptions/exception_assert_zero_gfunc:OK
#105/13  exceptions/exception_assert_neg_gfunc:OK
#105/14  exceptions/exception_assert_pos_gfunc:OK
#105/15  exceptions/exception_assert_negeq_gfunc:OK
#105/16  exceptions/exception_assert_poseq_gfunc:OK
#105/17  exceptions/exception_assert_nz_gfunc_with:OK
#105/18  exceptions/exception_assert_zero_gfunc_with:OK
#105/19  exceptions/exception_assert_neg_gfunc_with:OK
#105/20  exceptions/exception_assert_pos_gfunc_with:OK
#105/21  exceptions/exception_assert_negeq_gfunc_with:OK
#105/22  exceptions/exception_assert_poseq_gfunc_with:OK
#105/23  exceptions/exception_bad_assert_nz_gfunc:OK
#105/24  exceptions/exception_bad_assert_zero_gfunc:OK
#105/25  exceptions/exception_bad_assert_neg_gfunc:OK
#105/26  exceptions/exception_bad_assert_pos_gfunc:OK
#105/27  exceptions/exception_bad_assert_negeq_gfunc:OK
#105/28  exceptions/exception_bad_assert_poseq_gfunc:OK
#105/29  exceptions/exception_bad_assert_nz_gfunc_with:OK
#105/30  exceptions/exception_bad_assert_zero_gfunc_with:OK
#105/31  exceptions/exception_bad_assert_neg_gfunc_with:OK
#105/32  exceptions/exception_bad_assert_pos_gfunc_with:OK
#105/33  exceptions/exception_bad_assert_negeq_gfunc_with:OK
#105/34  exceptions/exception_bad_assert_poseq_gfunc_with:OK
#105/35  exceptions/exception_assert_range:OK
#105/36  exceptions/exception_assert_range_with:OK
#105/37  exceptions/exception_bad_assert_range:OK
#105/38  exceptions/exception_bad_assert_range_with:OK
#105/39  exceptions/non-throwing fentry -> exception_cb:OK
#105/40  exceptions/throwing fentry -> exception_cb:OK
#105/41  exceptions/non-throwing fexit -> exception_cb:OK
#105/42  exceptions/throwing fexit -> exception_cb:OK
#105/43  exceptions/throwing extension (with custom cb) -> exception_cb:OK
#105/44  exceptions/throwing extension -> global func in exception_cb:OK
#105/45  exceptions/exception_ext_mod_cb_runtime:OK
#105/46  exceptions/throwing extension (with custom cb) -> global func in exception_cb:OK
#105/47  exceptions/exception_ext:OK
#105/48  exceptions/non-throwing fentry -> non-throwing subprog:OK
#105/49  exceptions/throwing fentry -> non-throwing subprog:OK
#105/50  exceptions/non-throwing fentry -> throwing subprog:OK
#105/51  exceptions/throwing fentry -> throwing subprog:OK
#105/52  exceptions/non-throwing fexit -> non-throwing subprog:OK
#105/53  exceptions/throwing fexit -> non-throwing subprog:OK
#105/54  exceptions/non-throwing fexit -> throwing subprog:OK
#105/55  exceptions/throwing fexit -> throwing subprog:OK
#105/56  exceptions/non-throwing fmod_ret -> non-throwing subprog:OK
#105/57  exceptions/non-throwing fmod_ret -> non-throwing global subprog:OK
#105/58  exceptions/non-throwing extension -> non-throwing subprog:OK
#105/59  exceptions/non-throwing extension -> throwing subprog:OK
#105/60  exceptions/non-throwing extension -> non-throwing subprog:OK
#105/61  exceptions/non-throwing extension -> throwing global subprog:OK
#105/62  exceptions/throwing extension -> throwing global subprog:OK
#105/63  exceptions/throwing extension -> non-throwing global subprog:OK
#105/64  exceptions/non-throwing extension -> main subprog:OK
#105/65  exceptions/throwing extension -> main subprog:OK
#105/66  exceptions/reject_exception_cb_type_1:OK
#105/67  exceptions/reject_exception_cb_type_2:OK
#105/68  exceptions/reject_exception_cb_type_3:OK
#105/69  exceptions/reject_exception_cb_type_4:OK
#105/70  exceptions/reject_async_callback_throw:OK
#105/71  exceptions/reject_with_lock:OK
#105/72  exceptions/reject_subprog_with_lock:OK
#105/73  exceptions/reject_with_rcu_read_lock:OK
#105/74  exceptions/reject_subprog_with_rcu_read_lock:OK
#105/75  exceptions/reject_with_rbtree_add_throw:OK
#105/76  exceptions/reject_with_reference:OK
#105/77  exceptions/reject_with_cb_reference:OK
#105/78  exceptions/reject_with_cb:OK
#105/79  exceptions/reject_with_subprog_reference:OK
#105/80  exceptions/reject_throwing_exception_cb:OK
#105/81  exceptions/reject_exception_cb_call_global_func:OK
#105/82  exceptions/reject_exception_cb_call_static_func:OK
#105/83  exceptions/reject_multiple_exception_cb:OK
#105/84  exceptions/reject_exception_throw_cb:OK
#105/85  exceptions/reject_exception_throw_cb_diff:OK
#105/86  exceptions/reject_set_exception_cb_bad_ret1:OK
#105/87  exceptions/reject_set_exception_cb_bad_ret2:OK
#105/88  exceptions/check_assert_eq_int_min:OK
#105/89  exceptions/check_assert_eq_int_max:OK
#105/90  exceptions/check_assert_eq_zero:OK
#105/91  exceptions/check_assert_eq_llong_min:OK
#105/92  exceptions/check_assert_eq_llong_max:OK
#105/93  exceptions/check_assert_lt_pos:OK
#105/94  exceptions/check_assert_lt_zero:OK
#105/95  exceptions/check_assert_lt_neg:OK
#105/96  exceptions/check_assert_le_pos:OK
#105/97  exceptions/check_assert_le_zero:OK
#105/98  exceptions/check_assert_le_neg:OK
#105/99  exceptions/check_assert_gt_pos:OK
#105/100 exceptions/check_assert_gt_zero:OK
#105/101 exceptions/check_assert_gt_neg:OK
#105/102 exceptions/check_assert_ge_pos:OK
#105/103 exceptions/check_assert_ge_zero:OK
#105/104 exceptions/check_assert_ge_neg:OK
#105/105 exceptions/check_assert_range_s64:OK
#105/106 exceptions/check_assert_range_u64:OK
#105/107 exceptions/check_assert_single_range_s64:OK
#105/108 exceptions/check_assert_single_range_u64:OK
#105/109 exceptions/check_assert_generic:OK
#105/110 exceptions/check_assert_with_return:OK
#105     exceptions:OK
Summary: 1/110 PASSED, 0 SKIPPED, 0 FAILED

[v1]: https://lore.kernel.org/all/[email protected]/

Changes v1->v2:
Move tail_call_cnt to offset 0 in stack frame Remove trampoline NVR remapping-patch3/6[v1]

Abhishek Dubey (6):
powerpc64/bpf: Move tail_call_cnt to bottom of stack frame powerpc64/bpf: Support tailcalls with subprogs powerpc64/bpf: Tailcall handling with trampolines powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT powerpc64/bpf: Support exceptions
powerpc64/bpf: Additional NVR handling for bpf_throw

 arch/powerpc/net/bpf_jit.h        |  18 ++-
 arch/powerpc/net/bpf_jit_comp.c   |  98 ++++++++----
 arch/powerpc/net/bpf_jit_comp64.c | 240 ++++++++++++++++++++++++------
 3 files changed, 280 insertions(+), 76 deletions(-)
Signature · 1 line
2.48.1
#1 lore raw

PATCH v2 1/6 powerpc64/bpf: Move tail_call_cnt to bottom of stack frame

From: Abhishek Dubey <[email protected]>

In the conventional stack frame, the position of tail_call_cnt is after the NVR save area (BPF_PPC_STACK_SAVE). Whereas, the offset of tail_call_cnt in the trampoline frame is after the stack alignment padding. BPF JIT logic could become complex when dealing with frame-sensitive offset calculation of tail_call_cnt. Having the same offset in both frames is the desired objective.

The trampoline frame does not have a BPF_PPC_STACK_SAVE area. Introducing it leads to under-utilization of extra memory meant only for the offset alignment of tail_call_cnt. Another challenge is the variable alignment padding sitting at the bottom of the trampoline frame, which requires additional handling to compute tail_call_cnt offset.

This patch addresses the above issues by moving tail_call_cnt to the bottom of the stack frame at offset 0 for both types of frames. This saves additional bytes required by BPF_PPC_STACK_SAVE in trampoline frame, and a common offset computation for tail_call_cnt serves both frames.

The changes in this patch are required by the third patch in the series, where the 'reference to tail_call_info' of the main frame is copied into the trampoline frame from the previous frame.

---
 arch/powerpc/net/bpf_jit.h        |  4 ++++
 arch/powerpc/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++-----------
 2 files changed, 24 insertions(+), 11 deletions(-)
arch/powerpc/net/bpf_jit.h +4 -0
index 8334cd667bba..45d419c0ee73 100644
@@ -72,6 +72,10 @@
 	} } while (0)
 
 #ifdef CONFIG_PPC64
+
+/* for tailcall counter */
+#define BPF_PPC_TAILCALL        8
+
 /* If dummy pass (!image), account for maximum possible instructions */
 #define PPC_LI64(d, i)		do {					      \
 	if (!image)							      \
arch/powerpc/net/bpf_jit_comp64.c +20 -11
index 1fe37128c876..39061cd742c1 100644
@@ -20,13 +20,15 @@
 #include "bpf_jit.h"
 
 /*
- * Stack layout:
+ * Stack layout 1:
+ * Layout when setting up our own stack frame.
+ * Note: r1 at bottom, component offsets positive wrt r1.
  * Ensure the top half (upto local_tmp_var) stays consistent
  * with our redzone usage.
  *
  *		[	prev sp		] <-------------
- *		[   nv gpr save area	] 6*8		|
  *		[    tail_call_cnt	] 8		|
+ *		[   nv gpr save area	] 6*8		|
  *		[    local_tmp_var	] 24		|
  * fp (r31) -->	[   ebpf stack space	] upto 512	|
  *		[     frame header	] 32/112	|
@@ -36,10 +38,12 @@
 /* for gpr non volatile registers BPG_REG_6 to 10 */
 #define BPF_PPC_STACK_SAVE	(6*8)
 /* for bpf JIT code internal usage */
-#define BPF_PPC_STACK_LOCALS	32
+#define BPF_PPC_STACK_LOCALS	24
 /* stack frame excluding BPF stack, ensure this is quadword aligned */
 #define BPF_PPC_STACKFRAME	(STACK_FRAME_MIN_SIZE + \
-				 BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE)
+				 BPF_PPC_STACK_LOCALS + \
+				 BPF_PPC_STACK_SAVE   + \
+				 BPF_PPC_TAILCALL)
 
 /* BPF register usage */
 #define TMP_REG_1	(MAX_BPF_JIT_REG + 0)
@@ -87,27 +91,32 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
 }
 
 /*
+ * Stack layout 2:
  * When not setting up our own stackframe, the redzone (288 bytes) usage is:
+ * Note: r1 from prev frame. Component offset negative wrt r1.
  *
  *		[	prev sp		] <-------------
  *		[	  ...       	] 		|
  * sp (r1) --->	[    stack pointer	] --------------
- *		[   nv gpr save area	] 6*8
  *		[    tail_call_cnt	] 8
+ *		[   nv gpr save area	] 6*8
  *		[    local_tmp_var	] 24
  *		[   unused red zone	] 224
  */
 static int bpf_jit_stack_local(struct codegen_context *ctx)
 {
-	if (bpf_has_stack_frame(ctx))
+	if (bpf_has_stack_frame(ctx)) {
+		/* Stack layout 1 */
 		return STACK_FRAME_MIN_SIZE + ctx->stack_size;
-	else
-		return -(BPF_PPC_STACK_SAVE + 32);
+	} else {
+		/* Stack layout 2 */
+		return -(BPF_PPC_TAILCALL + BPF_PPC_STACK_SAVE + BPF_PPC_STACK_LOCALS);
+	}
 }
 
 static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx)
 {
-	return bpf_jit_stack_local(ctx) + 24;
+	return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE;
 }
 
 static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
@@ -115,7 +124,7 @@ static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
 	if (reg >= BPF_PPC_NVR_MIN && reg < 32)
 		return (bpf_has_stack_frame(ctx) ?
 			(BPF_PPC_STACKFRAME + ctx->stack_size) : 0)
-				- (8 * (32 - reg));
+				- (8 * (32 - reg)) - BPF_PPC_TAILCALL;
 
 	pr_err("BPF JIT is asking about unknown registers");
 	BUG();
@@ -145,7 +154,7 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 	if (ctx->seen & SEEN_TAILCALL) {
 		EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0));
 		/* this goes in the redzone */
-		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_STACK_SAVE + 8)));
+		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
 	} else {
 		EMIT(PPC_RAW_NOP());
 		EMIT(PPC_RAW_NOP());
Signature · 1 line
2.48.1
#2 lore raw

Re: [PATCH v2 1/6] powerpc64/bpf: Move tail_call_cnt to bottom of stack frame

This new constant defines tail_call_cnt offset as 8 bytes from prev sp.
Quoted 4 lines
arch/powerpc/net/bpf_jit.h +0 -0
index 8334cd667bba..45d419c0ee73 100644

[ ... ]

arch/powerpc/net/bpf_jit.h line 77

Quoted 2 lines
arch/powerpc/net/bpf_jit.h +2 -0
+/* for tailcall counter */
+#define BPF_PPC_TAILCALL        8

This new constant defines tail_call_cnt offset as 8 bytes from prev sp.

Quoted 4 lines
arch/powerpc/net/bpf_jit_comp64.c +0 -0
index 1fe37128c876..39061cd742c1 100644

[ ... ]

Quoted 5 lines

static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx) {

arch/powerpc/net/bpf_jit_comp64.c +1 -1
-	return bpf_jit_stack_local(ctx) + 24;
+	return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE;

}

The tail_call_cnt offset changes from +24 to +72 (24 + 48) relative to bpf_jit_stack_local().

There appears to be an inconsistency with the trampoline code in bpf_jit_comp.c. The function bpf_trampoline_setup_tail_call_cnt() and bpf_trampoline_restore_tail_call_cnt() use a hardcoded offset:

    int tailcallcnt_offset = 7 * 8;  /* = 56 */

The comment says "See bpf_jit_stack_tailcallcnt()" but after this patch, bpf_jit_stack_tailcallcnt() returns an offset that corresponds to 8 bytes from prev sp (BPF_PPC_TAILCALL), not 56 bytes. When BPF_TRAMP_F_TAIL_CALL_CTX is set, will the trampoline read/write to the wrong stack location?

Note: This appears to be fixed later in the same series by commit 6889ec3d3e10 ("powerpc64/bpf: Tailcall handling with trampolines") which updates the trampoline code to use BPF_PPC_TAILCALL instead of the hardcoded 7 * 8.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/20993216190

#3 lore raw

Re: [PATCH v2 1/6] powerpc64/bpf: Move tail_call_cnt to bottom of stack frame

In reply to [email protected]

Le 14/01/2026 à 12:44, [email protected] a écrit :

Quoted 38 lines

From: Abhishek Dubey <[email protected]>

In the conventional stack frame, the position of tail_call_cnt is after the NVR save area (BPF_PPC_STACK_SAVE). Whereas, the offset of tail_call_cnt in the trampoline frame is after the stack alignment padding. BPF JIT logic could become complex when dealing with frame-sensitive offset calculation of tail_call_cnt. Having the same offset in both frames is the desired objective.

The trampoline frame does not have a BPF_PPC_STACK_SAVE area. Introducing it leads to under-utilization of extra memory meant only for the offset alignment of tail_call_cnt. Another challenge is the variable alignment padding sitting at the bottom of the trampoline frame, which requires additional handling to compute tail_call_cnt offset.

This patch addresses the above issues by moving tail_call_cnt to the bottom of the stack frame at offset 0 for both types of frames. This saves additional bytes required by BPF_PPC_STACK_SAVE in trampoline frame, and a common offset computation for tail_call_cnt serves both frames.

The changes in this patch are required by the third patch in the series, where the 'reference to tail_call_info' of the main frame is copied into the trampoline frame from the previous frame.

---
  arch/powerpc/net/bpf_jit.h        |  4 ++++
  arch/powerpc/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++-----------
  2 files changed, 24 insertions(+), 11 deletions(-)
arch/powerpc/net/bpf_jit.h +0 -0
index 8334cd667bba..45d419c0ee73 100644

arch/powerpc/net/bpf_jit.h line 77

Quoted 6 lines
arch/powerpc/net/bpf_jit.h +3 -0
@@ -72,6 +72,10 @@
  	} } while (0)
 
  #ifdef CONFIG_PPC64
+
+/* for tailcall counter */
+#define BPF_PPC_TAILCALL        8

This needs to be defined outside of CONFIG_PPC64 ifdef because from patch 3 it is used in bpf_jit_comp.c which is also built on powerpc32.

Quoted 4 lines
arch/powerpc/net/bpf_jit.h +1 -0
+
  /* If dummy pass (!image), account for maximum possible instructions */
  #define PPC_LI64(d, i)		do {					      \
  	if (!image)							      \
maintainer
#4 lore raw

Re: [PATCH v2 1/6] powerpc64/bpf: Move tail_call_cnt to bottom of stack frame

In reply to [email protected]
Quoted 93 lines · On 14/01/26 5:14 pm, [email protected] wrote:

From: Abhishek Dubey <[email protected]>

In the conventional stack frame, the position of tail_call_cnt is after the NVR save area (BPF_PPC_STACK_SAVE). Whereas, the offset of tail_call_cnt in the trampoline frame is after the stack alignment padding. BPF JIT logic could become complex when dealing with frame-sensitive offset calculation of tail_call_cnt. Having the same offset in both frames is the desired objective.

The trampoline frame does not have a BPF_PPC_STACK_SAVE area. Introducing it leads to under-utilization of extra memory meant only for the offset alignment of tail_call_cnt. Another challenge is the variable alignment padding sitting at the bottom of the trampoline frame, which requires additional handling to compute tail_call_cnt offset.

This patch addresses the above issues by moving tail_call_cnt to the bottom of the stack frame at offset 0 for both types of frames. This saves additional bytes required by BPF_PPC_STACK_SAVE in trampoline frame, and a common offset computation for tail_call_cnt serves both frames.

The changes in this patch are required by the third patch in the series, where the 'reference to tail_call_info' of the main frame is copied into the trampoline frame from the previous frame.

---
  arch/powerpc/net/bpf_jit.h        |  4 ++++
  arch/powerpc/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++-----------
  2 files changed, 24 insertions(+), 11 deletions(-)
arch/powerpc/net/bpf_jit.h +4 -0
index 8334cd667bba..45d419c0ee73 100644
@@ -72,6 +72,10 @@
  	} } while (0)
 
  #ifdef CONFIG_PPC64
+
+/* for tailcall counter */
+#define BPF_PPC_TAILCALL        8
+
  /* If dummy pass (!image), account for maximum possible instructions */
  #define PPC_LI64(d, i)		do {					      \
  	if (!image)							      \
arch/powerpc/net/bpf_jit_comp64.c +10 -4
index 1fe37128c876..39061cd742c1 100644
@@ -20,13 +20,15 @@
  #include "bpf_jit.h"
 
  /*
- * Stack layout:
+ * Stack layout 1:
+ * Layout when setting up our own stack frame.
+ * Note: r1 at bottom, component offsets positive wrt r1.
   * Ensure the top half (upto local_tmp_var) stays consistent
   * with our redzone usage.
   *
   *		[	prev sp		] <-------------
- *		[   nv gpr save area	] 6*8		|
   *		[    tail_call_cnt	] 8		|
+ *		[   nv gpr save area	] 6*8		|
   *		[    local_tmp_var	] 24		|
   * fp (r31) -->	[   ebpf stack space	] upto 512	|
   *		[     frame header	] 32/112	|
@@ -36,10 +38,12 @@
  /* for gpr non volatile registers BPG_REG_6 to 10 */
  #define BPF_PPC_STACK_SAVE	(6*8)
  /* for bpf JIT code internal usage */
-#define BPF_PPC_STACK_LOCALS	32
+#define BPF_PPC_STACK_LOCALS	24
  /* stack frame excluding BPF stack, ensure this is quadword aligned */
  #define BPF_PPC_STACKFRAME	(STACK_FRAME_MIN_SIZE + \
-				 BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE)
+				 BPF_PPC_STACK_LOCALS + \
+				 BPF_PPC_STACK_SAVE   + \
+				 BPF_PPC_TAILCALL)
 
  /* BPF register usage */
  #define TMP_REG_1	(MAX_BPF_JIT_REG + 0)
@@ -87,27 +91,32 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
  }
 
 
  /*
+ * Stack layout 2:
   * When not setting up our own stackframe, the redzone (288 bytes) usage is:
+ * Note: r1 from prev frame. Component offset negative wrt r1.

arch/powerpc/net/bpf_jit_comp64.c static inline bool bpf_has_stack_frame(struct codegen_context *ctx) · line 106

Quoted 10 lines
arch/powerpc/net/bpf_jit_comp64.c +1 -1
   *
   *		[	prev sp		] <-------------
   *		[	  ...       	] 		|
   * sp (r1) --->	[    stack pointer	] --------------
- *		[   nv gpr save area	] 6*8
   *		[    tail_call_cnt	] 8
+ *		[   nv gpr save area	] 6*8
   *		[    local_tmp_var	] 24
   *		[   unused red zone	] 224
   */

Calling it stack layout 1 & 2 is inappropriate. The stack layout is essentially the same. It just goes to show things with reference to r1 when stack is setup explicitly vs when redzone is being used...

- Hari

A message that is not in the archive (sent off-list or not yet received): [email protected]

maintainer
#6 lore raw

Re: [PATCH v2 1/6] powerpc64/bpf: Move tail_call_cnt to bottom of stack frame

Quoted 104 lines · adubey
Quoted 103 lines · Hari Bathini
Quoted 101 lines · On 14/01/26 5:14 pm, [email protected] wrote:

From: Abhishek Dubey <[email protected]>

In the conventional stack frame, the position of tail_call_cnt is after the NVR save area (BPF_PPC_STACK_SAVE). Whereas, the offset of tail_call_cnt in the trampoline frame is after the stack alignment padding. BPF JIT logic could become complex when dealing with frame-sensitive offset calculation of tail_call_cnt. Having the same offset in both frames is the desired objective.

The trampoline frame does not have a BPF_PPC_STACK_SAVE area. Introducing it leads to under-utilization of extra memory meant only for the offset alignment of tail_call_cnt. Another challenge is the variable alignment padding sitting at the bottom of the trampoline frame, which requires additional handling to compute tail_call_cnt offset.

This patch addresses the above issues by moving tail_call_cnt to the bottom of the stack frame at offset 0 for both types of frames. This saves additional bytes required by BPF_PPC_STACK_SAVE in trampoline frame, and a common offset computation for tail_call_cnt serves both frames.

The changes in this patch are required by the third patch in the series, where the 'reference to tail_call_info' of the main frame is copied into the trampoline frame from the previous frame.

---
  arch/powerpc/net/bpf_jit.h        |  4 ++++ arch/powerpc/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++-----------

  2 files changed, 24 insertions(+), 11 deletions(-)
arch/powerpc/net/bpf_jit.h +4 -0
index 8334cd667bba..45d419c0ee73 100644
@@ -72,6 +72,10 @@
       } } while (0)
     #ifdef CONFIG_PPC64
+
+/* for tailcall counter */
+#define BPF_PPC_TAILCALL        8
+
   /* If dummy pass (!image), account for maximum possible instructions */
   #define PPC_LI64(d, i)        do {                          \
       if (!image)                                  \
arch/powerpc/net/bpf_jit_comp64.c +11 -5
index 1fe37128c876..39061cd742c1 100644
@@ -20,13 +20,15 @@
   #include "bpf_jit.h"
     /*
- * Stack layout:
+ * Stack layout 1:
+ * Layout when setting up our own stack frame.
+ * Note: r1 at bottom, component offsets positive wrt r1.
    * Ensure the top half (upto local_tmp_var) stays consistent
    * with our redzone usage.
    *
    *        [    prev sp        ] <-------------
- *        [   nv gpr save area    ] 6*8        |
    *        [    tail_call_cnt    ] 8        |
+ *        [   nv gpr save area    ] 6*8        |
    *        [    local_tmp_var    ] 24        |
    * fp (r31) -->    [   ebpf stack space    ] upto 512    |
    *        [     frame header    ] 32/112    |
@@ -36,10 +38,12 @@
   /* for gpr non volatile registers BPG_REG_6 to 10 */
   #define BPF_PPC_STACK_SAVE    (6*8)
   /* for bpf JIT code internal usage */
-#define BPF_PPC_STACK_LOCALS    32
+#define BPF_PPC_STACK_LOCALS    24
   /* stack frame excluding BPF stack, ensure this is quadword aligned */
   #define BPF_PPC_STACKFRAME    (STACK_FRAME_MIN_SIZE + \
-                 BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE)
+                 BPF_PPC_STACK_LOCALS + \
+                 BPF_PPC_STACK_SAVE   + \
+                 BPF_PPC_TAILCALL)
     /* BPF register usage */
   #define TMP_REG_1    (MAX_BPF_JIT_REG + 0)
@@ -87,27 +91,32 @@ static inline bool bpf_has_stack_frame(struct 
 codegen_context *ctx)
   }
 
 
   /*
+ * Stack layout 2:
    * When not setting up our own stackframe, the redzone (288 bytes) usage is:
+ * Note: r1 from prev frame. Component offset negative wrt r1.
    *
    *        [    prev sp        ] <-------------
    *        [      ...           ]         |
    * sp (r1) --->    [    stack pointer    ] --------------
- *        [   nv gpr save area    ] 6*8
    *        [    tail_call_cnt    ] 8
+ *        [   nv gpr save area    ] 6*8
    *        [    local_tmp_var    ] 24
    *        [   unused red zone    ] 224
    */
Quoted 4 lines
Quoted 3 lines

Calling it stack layout 1 & 2 is inappropriate. The stack layout is essentially the same. It just goes to show things with reference to r1 when stack is setup explicitly vs when redzone is being used...

Agree. I am using it as labels to refer in comment. Any better suggestions?

I think the comments could refer to has stack frame vs Redzone case..

- Hari

#7 lore raw

PATCH v2 2/6 powerpc64/bpf: Support tailcalls with subprogs

In reply to [email protected]

From: Abhishek Dubey <[email protected]>

Enabling tailcalls with subprog combinations by referencing method. The actual tailcall count is always maintained in the tail_call_info variable present in the frame of main function (also called entry function). The tail_call_info variables in the frames of all other subprog contains reference to the tail_call_info present in frame of main function.

Dynamic resolution interprets the tail_call_info either as value or reference depending on the context of active frame while tailcall is invoked.

---
 arch/powerpc/net/bpf_jit.h        | 12 +++++-
 arch/powerpc/net/bpf_jit_comp.c   | 10 ++++-
 arch/powerpc/net/bpf_jit_comp64.c | 68 +++++++++++++++++++++++--------
 3 files changed, 70 insertions(+), 20 deletions(-)
arch/powerpc/net/bpf_jit.h +10 -2
index 45d419c0ee73..5d735bc5e6bd 100644
@@ -51,6 +51,12 @@
 		EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));					\
 	} while (0)
 
+/* Same as PPC_BCC_SHORT, except valid dest is known prior to call. */
+#define PPC_COND_BRANCH(cond, dest)                                           \
+	do {								      \
+		long offset = (long)(dest) - CTX_NIA(ctx);		      \
+		EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));    \
+	} while (0)
 /*
  * Sign-extended 32-bit immediate load
  *
@@ -75,6 +81,8 @@
 
 /* for tailcall counter */
 #define BPF_PPC_TAILCALL        8
+/* for gpr non volatile registers BPG_REG_6 to 10 */
+#define BPF_PPC_STACK_SAVE      (6*8)
 
 /* If dummy pass (!image), account for maximum possible instructions */
 #define PPC_LI64(d, i)		do {					      \
@@ -170,6 +178,7 @@ struct codegen_context {
 	unsigned int alt_exit_addr;
 	u64 arena_vm_start;
 	u64 user_vm_start;
+	bool is_subprog;
 };
 
 #define bpf_to_ppc(r)	(ctx->b2p[r])
@@ -204,11 +213,10 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
 void bpf_jit_build_fentry_stubs(u32 *image, struct codegen_context *ctx);
 void bpf_jit_realloc_regs(struct codegen_context *ctx);
 int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr);
-
 int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass,
 			  struct codegen_context *ctx, int insn_idx,
 			  int jmp_off, int dst_reg, u32 code);
-
+int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx);
 #endif
 
 #endif
arch/powerpc/net/bpf_jit_comp.c +8 -2
index 5e976730b2f5..069a8822c30d 100644
@@ -206,6 +206,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 	cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
 	cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux->arena);
 	cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena);
+	cgctx.is_subprog = bpf_is_subprog(fp);
 
 	/* Scouting faux-generate pass 0 */
 	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
@@ -435,6 +436,11 @@ void bpf_jit_free(struct bpf_prog *fp)
 	bpf_prog_unlock_free(fp);
 }
 
+bool bpf_jit_supports_subprog_tailcalls(void)
+{
+	return IS_ENABLED(CONFIG_PPC64);
+}
+
 bool bpf_jit_supports_kfunc_call(void)
 {
 	return true;
@@ -604,7 +610,7 @@ static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_contex
 					       int func_frame_offset, int r4_off)
 {
 	if (IS_ENABLED(CONFIG_PPC64)) {
-		/* See bpf_jit_stack_tailcallcnt() */
+		/* See bpf_jit_stack_tailcallinfo_offset() */
 		int tailcallcnt_offset = 7 * 8;
 
 		EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
@@ -619,7 +625,7 @@ static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont
 						 int func_frame_offset, int r4_off)
 {
 	if (IS_ENABLED(CONFIG_PPC64)) {
-		/* See bpf_jit_stack_tailcallcnt() */
+		/* See bpf_jit_stack_tailcallinfo_offset() */
 		int tailcallcnt_offset = 7 * 8;
 
 		EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
arch/powerpc/net/bpf_jit_comp64.c +52 -16

118 lines not on this page · raw message

Signature · 1 line
2.48.1
#8 lore raw

Re: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs

Le 14/01/2026 à 12:44, [email protected] a écrit :

Quoted 25 lines

From: Abhishek Dubey <[email protected]>

Enabling tailcalls with subprog combinations by referencing method. The actual tailcall count is always maintained in the tail_call_info variable present in the frame of main function (also called entry function). The tail_call_info variables in the frames of all other subprog contains reference to the tail_call_info present in frame of main function.

Dynamic resolution interprets the tail_call_info either as value or reference depending on the context of active frame while tailcall is invoked.

---
  arch/powerpc/net/bpf_jit.h        | 12 +++++-
  arch/powerpc/net/bpf_jit_comp.c   | 10 ++++-
  arch/powerpc/net/bpf_jit_comp64.c | 68 +++++++++++++++++++++++--------
  3 files changed, 70 insertions(+), 20 deletions(-)
arch/powerpc/net/bpf_jit.h +0 -0
index 45d419c0ee73..5d735bc5e6bd 100644

arch/powerpc/net/bpf_jit.h line 59

Quoted 9 lines
arch/powerpc/net/bpf_jit.h +6 -0
@@ -51,6 +51,12 @@
  		EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));					\
  	} while (0)
 
+/* Same as PPC_BCC_SHORT, except valid dest is known prior to call. */
+#define PPC_COND_BRANCH(cond, dest)                                           \
+	do {								      \
+		long offset = (long)(dest) - CTX_NIA(ctx);		      \
+		EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));    \
+	} while (0)

I don't like the idea of duplicating PPC_BCC_SHORT() to just kick the verification out. Now we will have two macros doing the exact same thing with one handling failure case and one ignoring failure case. There is a big risk that one day or another someone will use the wrong macro.

Could you change bpf_jit_build_prologue() to return an int add use PPC_BCC_SHORT() instead of that new PPC_COND_BRANCH() ?

arch/powerpc/net/bpf_jit.h line 85

Quoted 9 lines
arch/powerpc/net/bpf_jit.h +2 -0
  /*
   * Sign-extended 32-bit immediate load
   *
@@ -75,6 +81,8 @@
 
  /* for tailcall counter */
  #define BPF_PPC_TAILCALL        8
+/* for gpr non volatile registers BPG_REG_6 to 10 */
+#define BPF_PPC_STACK_SAVE      (6*8)

Add spaces before and after the *

Quoted 12 lines
arch/powerpc/net/bpf_jit.h +1 -0
  /* If dummy pass (!image), account for maximum possible instructions */
  #define PPC_LI64(d, i)		do {					      \
@@ -170,6 +178,7 @@ struct codegen_context {
  	unsigned int alt_exit_addr;
  	u64 arena_vm_start;
  	u64 user_vm_start;
+	bool is_subprog;
  };
 
  #define bpf_to_ppc(r)	(ctx->b2p[r])

arch/powerpc/net/bpf_jit.h void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx); · line 211

Quoted 8 lines
arch/powerpc/net/bpf_jit.h +0 -2
@@ -204,11 +213,10 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
  void bpf_jit_build_fentry_stubs(u32 *image, struct codegen_context *ctx);
  void bpf_jit_realloc_regs(struct codegen_context *ctx);
  int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr);
-
  int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass,
  			  struct codegen_context *ctx, int insn_idx,
  			  int jmp_off, int dst_reg, u32 code);
-

Not sure why this patch needs to remove those blank lines here and above.

Quoted 174 lines
arch/powerpc/net/bpf_jit.h +1 -0
+int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx);
  #endif
 
  #endif
arch/powerpc/net/bpf_jit_comp.c +8 -2
index 5e976730b2f5..069a8822c30d 100644
@@ -206,6 +206,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
  	cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
  	cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux->arena);
  	cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena);
+	cgctx.is_subprog = bpf_is_subprog(fp);
 
  	/* Scouting faux-generate pass 0 */
  	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
@@ -435,6 +436,11 @@ void bpf_jit_free(struct bpf_prog *fp)
  	bpf_prog_unlock_free(fp);
  }
 
+bool bpf_jit_supports_subprog_tailcalls(void)
+{
+	return IS_ENABLED(CONFIG_PPC64);
+}
+
  bool bpf_jit_supports_kfunc_call(void)
  {
  	return true;
@@ -604,7 +610,7 @@ static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_contex
  					       int func_frame_offset, int r4_off)
  {
  	if (IS_ENABLED(CONFIG_PPC64)) {
-		/* See bpf_jit_stack_tailcallcnt() */
+		/* See bpf_jit_stack_tailcallinfo_offset() */
  		int tailcallcnt_offset = 7 * 8;
 
  		EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
@@ -619,7 +625,7 @@ static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont
  						 int func_frame_offset, int r4_off)
  {
  	if (IS_ENABLED(CONFIG_PPC64)) {
-		/* See bpf_jit_stack_tailcallcnt() */
+		/* See bpf_jit_stack_tailcallinfo_offset() */
  		int tailcallcnt_offset = 7 * 8;
 
  		EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
arch/powerpc/net/bpf_jit_comp64.c +52 -16
index 39061cd742c1..cebf81fbd59f 100644
@@ -26,8 +26,12 @@
   * Ensure the top half (upto local_tmp_var) stays consistent
   * with our redzone usage.
   *
+ * tail_call_info - stores tailcall count value in main program's
+ *                  frame, stores reference to tail_call_info of
+ *                  main's frame in sub-prog's frame.
+ *
   *		[	prev sp		] <-------------
- *		[    tail_call_cnt	] 8		|
+ *		[    tail_call_info	] 8		|
   *		[   nv gpr save area	] 6*8		|
   *		[    local_tmp_var	] 24		|
   * fp (r31) -->	[   ebpf stack space	] upto 512	|
@@ -35,8 +39,6 @@
   * sp (r1) --->	[    stack pointer	] --------------
   */
 
-/* for gpr non volatile registers BPG_REG_6 to 10 */
-#define BPF_PPC_STACK_SAVE	(6*8)
  /* for bpf JIT code internal usage */
  #define BPF_PPC_STACK_LOCALS	24
  /* stack frame excluding BPF stack, ensure this is quadword aligned */
@@ -98,7 +100,7 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
   *		[	prev sp		] <-------------
   *		[	  ...       	] 		|
   * sp (r1) --->	[    stack pointer	] --------------
- *		[    tail_call_cnt	] 8
+ *		[    tail_call_info	] 8
   *		[   nv gpr save area	] 6*8
   *		[    local_tmp_var	] 24
   *		[   unused red zone	] 224
@@ -114,7 +116,7 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)
  	}
  }
 
-static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx)
+int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)
  {
  	return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE;
  }
@@ -147,17 +149,32 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
  #endif
 
  	/*
-	 * Initialize tail_call_cnt if we do tail calls.
-	 * Otherwise, put in NOPs so that it can be skipped when we are
-	 * invoked through a tail call.
+	 * Tail call count(tcc) is saved & updated only in main
+	 * program's frame and the address of tcc in main program's
+	 * frame (tcc_ptr) is saved in subprogs frame.
+	 *
+	 * Offset of tail_call_info on any frame will be interpreted
+	 * as either tcc_ptr or tcc value depending on whether it is
+	 * greater than MAX_TAIL_CALL_CNT or not.
  	 */
-	if (ctx->seen & SEEN_TAILCALL) {
+	if (!ctx->is_subprog) {
  		EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0));
  		/* this goes in the redzone */
  		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
  	} else {
-		EMIT(PPC_RAW_NOP());
-		EMIT(PPC_RAW_NOP());
+		/*
+		 * if tail_call_info < MAX_TAIL_CALL_CNT
+		 *     main prog calling first subprog -> copy reference
+		 * else
+		 *     subsequent subprog calling another subprog -> directly copy content
+		 */
+		EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, 0));
+		EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), -(BPF_PPC_TAILCALL)));
+		EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
+		PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+		EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2),
+							-(BPF_PPC_TAILCALL)));
+		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
  	}
 
  	if (bpf_has_stack_frame(ctx)) {
@@ -352,19 +369,38 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
  	EMIT(PPC_RAW_CMPLW(b2p_index, bpf_to_ppc(TMP_REG_1)));
  	PPC_BCC_SHORT(COND_GE, out);
 
+	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+	EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
+	PPC_COND_BRANCH(COND_LE, CTX_NIA(ctx) + 8);
+
+	/* dereference TMP_REG_1 */
+	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 0));
+
  	/*
-	 * if (tail_call_cnt >= MAX_TAIL_CALL_CNT)
+	 * if (tail_call_info == MAX_TAIL_CALL_CNT)
  	 *   goto out;
  	 */
-	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
  	EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
-	PPC_BCC_SHORT(COND_GE, out);
+	PPC_COND_BRANCH(COND_EQ, out);
 
  	/*
-	 * tail_call_cnt++;
+	 * tail_call_info++; <- Actual value of tcc here
  	 */
  	EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 1));
-	EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
+
+	/*
+	 * Before writing updated tail_call_info, distinguish if current frame
+	 * is storing a reference to tail_call_info or actual tcc value in
+	 * tail_call_info.
+	 */
+	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+	EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_2), MAX_TAIL_CALL_CNT));
+	PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+
+	/* First get address of tail_call_info */
+	EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+	/* Writeback updated value to tail_call_info */
+	EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), 0));
 
  	/* prog = array->ptrs[index]; */
  	EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_1), b2p_index, 8));

A message that is not in the archive (sent off-list or not yet received): [email protected]

maintainer
#10 lore raw

Re: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs

Quoted 43 lines · adubey
Quoted 42 lines · Christophe Leroy (CS GROUP)

Le 14/01/2026 à 12:44, [email protected] a écrit :

Quoted 27 lines

From: Abhishek Dubey <[email protected]>

Enabling tailcalls with subprog combinations by referencing method. The actual tailcall count is always maintained in the tail_call_info variable present in the frame of main function (also called entry function). The tail_call_info variables in the frames of all other subprog contains reference to the tail_call_info present in frame of main function.

Dynamic resolution interprets the tail_call_info either as value or reference depending on the context of active frame while tailcall is invoked.

---
  arch/powerpc/net/bpf_jit.h        | 12 +++++- arch/powerpc/net/bpf_jit_comp.c   | 10 ++++- arch/powerpc/net/bpf_jit_comp64.c | 68 +++++++++++++++++++++++--------

  3 files changed, 70 insertions(+), 20 deletions(-)
arch/powerpc/net/bpf_jit.h +0 -0
index 45d419c0ee73..5d735bc5e6bd 100644
@@ -51,6 +51,12 @@
           EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));                    \
       } while (0)
Quoted 7 lines

  +/* Same as PPC_BCC_SHORT, except valid dest is known prior to call. */ +#define PPC_COND_BRANCH(cond, dest)         \
+    do {                                      \
+        long offset = (long)(dest) - CTX_NIA(ctx);              \
+        EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));    \
+    } while (0)

I don't like the idea of duplicating PPC_BCC_SHORT() to just kick the verification out. Now we will have two macros doing the exact same thing with one handling failure case and one ignoring failure case. There is a big risk that one day or another someone will use the wrong macro.

Quoted 6 lines
Quoted 2 lines

Could you change bpf_jit_build_prologue() to return an int add use PPC_BCC_SHORT() instead of that new PPC_COND_BRANCH() ?

I implemented exactly same change in bpf_jit_build_prologue(). But, during internal review, @HariBathini suggested to have separate macro with a caution note.

@Hari please suggest here!

Not just about the change of return type but the check seems like an overkill for cases where the offset is known and within branch range. How about using BUILD_BUG_ON() to avoid unecessary checks and return type change for places where the branch offset is known and is a constant?

Quoted 213 lines
Quoted 204 lines

arch/powerpc/net/bpf_jit.h line 84

Quoted 9 lines

  /*
   * Sign-extended 32-bit immediate load
   *

arch/powerpc/net/bpf_jit.h +2 -0
@@ -75,6 +81,8 @@
     /* for tailcall counter */
   #define BPF_PPC_TAILCALL        8
+/* for gpr non volatile registers BPG_REG_6 to 10 */
+#define BPF_PPC_STACK_SAVE      (6*8)

Add spaces before and after the *

Quoted 10 lines
arch/powerpc/net/bpf_jit.h +1 -0
     /* If dummy pass (!image), account for maximum possible instructions */
   #define PPC_LI64(d, i)        do {                          \
@@ -170,6 +178,7 @@ struct codegen_context {
       unsigned int alt_exit_addr;
       u64 arena_vm_start;
       u64 user_vm_start;
+    bool is_subprog;
   };
     #define bpf_to_ppc(r)    (ctx->b2p[r])

arch/powerpc/net/bpf_jit.h void bpf_jit_build_epilogue(u32 *image, struct · line 212

Quoted 9 lines
arch/powerpc/net/bpf_jit.h +0 -2
@@ -204,11 +213,10 @@ void bpf_jit_build_epilogue(u32 *image, struct 
 codegen_context *ctx);
   void bpf_jit_build_fentry_stubs(u32 *image, struct codegen_context *ctx);
   void bpf_jit_realloc_regs(struct codegen_context *ctx);
   int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr);
-
   int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass,
                 struct codegen_context *ctx, int insn_idx,
                 int jmp_off, int dst_reg, u32 code);
-

Not sure why this patch needs to remove those blank lines here and above.

Quoted 170 lines
arch/powerpc/net/bpf_jit.h +1 -0
+int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx);
   #endif
     #endif
arch/powerpc/net/bpf_jit_comp.c +0 -0
index 5e976730b2f5..069a8822c30d 100644
@@ -206,6 +206,7 @@ struct bpf_prog *bpf_int_jit_compile(struct 
 bpf_prog *fp)
       cgctx.stack_size = round_up(fp->aux->stack_depth, 16);

      cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux-

Quoted 1 line

arena);

      cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena); +    cgctx.is_subprog = bpf_is_subprog(fp);
        /* Scouting faux-generate pass 0 */
      if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {

arch/powerpc/net/bpf_jit_comp.c +4 -0
@@ -435,6 +436,11 @@ void bpf_jit_free(struct bpf_prog *fp)
       bpf_prog_unlock_free(fp);
   }
   +bool bpf_jit_supports_subprog_tailcalls(void)
+{
+    return IS_ENABLED(CONFIG_PPC64);
+}
+
   bool bpf_jit_supports_kfunc_call(void)
   {
       return true;
@@ -604,7 +610,7 @@ static void 

bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_contex int func_frame_offset, int r4_off)
  {
      if (IS_ENABLED(CONFIG_PPC64)) {
-        /* See bpf_jit_stack_tailcallcnt() */
+        /* See bpf_jit_stack_tailcallinfo_offset() */
          int tailcallcnt_offset = 7 * 8;
            EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));

arch/powerpc/net/bpf_jit_comp.c +0 -0
@@ -619,7 +625,7 @@ static void 

bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont int func_frame_offset, int r4_off) {
      if (IS_ENABLED(CONFIG_PPC64)) {
-        /* See bpf_jit_stack_tailcallcnt() */
+        /* See bpf_jit_stack_tailcallinfo_offset() */ int tailcallcnt_offset = 7 * 8; EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));

arch/powerpc/net/bpf_jit_comp64.c +5 -1
index 39061cd742c1..cebf81fbd59f 100644
@@ -26,8 +26,12 @@
    * Ensure the top half (upto local_tmp_var) stays consistent
    * with our redzone usage.
    *
+ * tail_call_info - stores tailcall count value in main program's
+ *                  frame, stores reference to tail_call_info of
+ *                  main's frame in sub-prog's frame.
+ *
    *        [    prev sp        ] <-------------
- *        [    tail_call_cnt    ] 8        |
+ *        [    tail_call_info    ] 8        |
    *        [   nv gpr save area    ] 6*8        |
    *        [    local_tmp_var    ] 24        |
    * fp (r31) -->    [   ebpf stack space    ] upto 512    |
@@ -35,8 +39,6 @@
    * sp (r1) --->    [    stack pointer    ] --------------
    */

  -/* for gpr non volatile registers BPG_REG_6 to 10 */ -#define BPF_PPC_STACK_SAVE    (6*8)
  /* for bpf JIT code internal usage */
  #define BPF_PPC_STACK_LOCALS    24
  /* stack frame excluding BPF stack, ensure this is quadword aligned */

arch/powerpc/net/bpf_jit_comp64.c +1 -1
@@ -98,7 +100,7 @@ static inline bool bpf_has_stack_frame(struct 
 codegen_context *ctx)
    *        [    prev sp        ] <-------------
    *        [      ...           ]         |
    * sp (r1) --->    [    stack pointer    ] --------------
- *        [    tail_call_cnt    ] 8
+ *        [    tail_call_info    ] 8
    *        [   nv gpr save area    ] 6*8
    *        [    local_tmp_var    ] 24

   *        [   unused red zone    ] 224

arch/powerpc/net/bpf_jit_comp64.c +1 -0
@@ -114,7 +116,7 @@ static int bpf_jit_stack_local(struct 
 codegen_context *ctx)
       }
   }
   -static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx)
+int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)
   {
       return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE;

  }

arch/powerpc/net/bpf_jit_comp64.c +13 -6
@@ -147,17 +149,32 @@ void bpf_jit_build_prologue(u32 *image, struct 
 codegen_context *ctx)
   #endif
         /*
-     * Initialize tail_call_cnt if we do tail calls.
-     * Otherwise, put in NOPs so that it can be skipped when we are
-     * invoked through a tail call.
+     * Tail call count(tcc) is saved & updated only in main
+     * program's frame and the address of tcc in main program's
+     * frame (tcc_ptr) is saved in subprogs frame.
+     *
+     * Offset of tail_call_info on any frame will be interpreted
+     * as either tcc_ptr or tcc value depending on whether it is
+     * greater than MAX_TAIL_CALL_CNT or not.
        */
-    if (ctx->seen & SEEN_TAILCALL) {
+    if (!ctx->is_subprog) {
           EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0));
           /* this goes in the redzone */
           EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, - (BPF_PPC_TAILCALL)));
       } else {
-        EMIT(PPC_RAW_NOP());
-        EMIT(PPC_RAW_NOP());
+        /*
+         * if tail_call_info < MAX_TAIL_CALL_CNT
+         *     main prog calling first subprog -> copy reference
+         * else
+         *     subsequent subprog calling another subprog -> 

directly copy content
+         */
+        EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, 0));
+        EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), -(BPF_PPC_TAILCALL)));
+        EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT)); +        PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+        EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), +                            -(BPF_PPC_TAILCALL)));
+        EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, - (BPF_PPC_TAILCALL)));
      }
        if (bpf_has_stack_frame(ctx)) {

arch/powerpc/net/bpf_jit_comp64.c +0 -0
@@ -352,19 +369,38 @@ static int bpf_jit_emit_tail_call(u32 *image, 

struct codegen_context *ctx, u32 o
      EMIT(PPC_RAW_CMPLW(b2p_index, bpf_to_ppc(TMP_REG_1))); PPC_BCC_SHORT(COND_GE, out);
  +    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallinfo_offset(ctx))); +    EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
+    PPC_COND_BRANCH(COND_LE, CTX_NIA(ctx) + 8);

arch/powerpc/net/bpf_jit_comp64.c +4 -0
+
+    /* dereference TMP_REG_1 */
+    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 0));
+

      /*
-     * if (tail_call_cnt >= MAX_TAIL_CALL_CNT)
+     * if (tail_call_info == MAX_TAIL_CALL_CNT)
       *   goto out;
       */
-    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx))); EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT)); -    PPC_BCC_SHORT(COND_GE, out);
+    PPC_COND_BRANCH(COND_EQ, out);
        /*
-     * tail_call_cnt++;
+     * tail_call_info++; <- Actual value of tcc here
       */
      EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 1)); -    EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
+
+    /*
+     * Before writing updated tail_call_info, distinguish if current frame
+     * is storing a reference to tail_call_info or actual tcc value in +     * tail_call_info.
+     */
+    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+    EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_2), MAX_TAIL_CALL_CNT)); +    PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);

Quoted 8 lines
Quoted 8 lines

+
+    /* First get address of tail_call_info */
+    EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+    /* Writeback updated value to tail_call_info */
+    EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), 0)); /* prog = array->ptrs[index]; */ EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_1), b2p_index, 8));

-Abhishek

#11 lore raw

Re: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs

Le 16/01/2026 à 05:50, Hari Bathini a écrit :

Not received this mail that Hari is reponding to.

Quoted 52 lines · adubey
Quoted 49 lines · Christophe Leroy (CS GROUP)
Quoted 43 lines

Le 14/01/2026 à 12:44, [email protected] a écrit :

Quoted 28 lines

From: Abhishek Dubey <[email protected]>

Enabling tailcalls with subprog combinations by referencing method. The actual tailcall count is always maintained in the tail_call_info variable present in the frame of main function (also called entry function). The tail_call_info variables in the frames of all other subprog contains reference to the tail_call_info present in frame of main function.

Dynamic resolution interprets the tail_call_info either as value or reference depending on the context of active frame while tailcall is invoked.

---
  arch/powerpc/net/bpf_jit.h        | 12 +++++- arch/powerpc/net/bpf_jit_comp.c   | 10 ++++- arch/powerpc/net/bpf_jit_comp64.c | 68 ++++++++++++++++++++++
+--------

  3 files changed, 70 insertions(+), 20 deletions(-)
arch/powerpc/net/bpf_jit.h +0 -0
index 45d419c0ee73..5d735bc5e6bd 100644
@@ -51,6 +51,12 @@
           EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));                    \
       } while (0)
Quoted 7 lines

  +/* Same as PPC_BCC_SHORT, except valid dest is known prior to call. */ +#define PPC_COND_BRANCH(cond, dest)         \
+    do {                                      \
+        long offset = (long)(dest) - CTX_NIA(ctx);              \
+        EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));    \
+    } while (0)

I don't like the idea of duplicating PPC_BCC_SHORT() to just kick the verification out. Now we will have two macros doing the exact same thing with one handling failure case and one ignoring failure case. There is a big risk that one day or another someone will use the wrong macro.

Quoted 2 lines

Could you change bpf_jit_build_prologue() to return an int add use PPC_BCC_SHORT() instead of that new PPC_COND_BRANCH() ?

I implemented exactly same change in bpf_jit_build_prologue(). But, during internal review, @HariBathini suggested to have separate macro with a caution note.

Quoted 7 lines
Quoted 1 line

@Hari please suggest here!

Not just about the change of return type but the check seems like an overkill for cases where the offset is known and within branch range. How about using BUILD_BUG_ON() to avoid unecessary checks and return type change for places where the branch offset is known and is a constant?

When offset is a constant known at build time, checks are eliminated by gcc at build, see exemple below from disasembly of bpf_jit_comp32.o, there are no checks.

                                PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
    36d8:       3c 80 41 81     lis     r4,16769
                                EMIT(PPC_RAW_CMPLW(src_reg, _R0));
    36dc:       81 3f 00 04     lwz     r9,4(r31)
                                PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
    36e0:       60 84 00 10     ori     r4,r4,16
                                EMIT(PPC_RAW_CMPLW(src_reg, _R0));
    36e4:       39 29 00 01     addi    r9,r9,1
                                PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
    36e8:       55 23 10 3a     slwi    r3,r9,2
                                EMIT(PPC_RAW_CMPLW(src_reg, _R0));
    36ec:       91 3f 00 04     stw     r9,4(r31)
                                PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
    36f0:       7c 97 19 2e     stwx    r4,r23,r3
                                EMIT(PPC_RAW_LI(dst_reg, 0));
    36f4:       55 49 a9 94     rlwinm  r9,r10,21,6,10
                                PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
    36f8:       80 9f 00 04     lwz     r4,4(r31)
                                EMIT(PPC_RAW_LI(dst_reg, 0));
    36fc:       65 29 38 00     oris    r9,r9,14336
                                PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
    3700:       38 84 00 01     addi    r4,r4,1
                                EMIT(PPC_RAW_LI(dst_reg, 0));
    3704:       54 83 10 3a     slwi    r3,r4,2
                                PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
    3708:       90 9f 00 04     stw     r4,4(r31)
                                EMIT(PPC_RAW_LI(dst_reg, 0));
    370c:       7d 37 19 2e     stwx    r9,r23,r3
Quoted 219 lines
Quoted 218 lines
Quoted 209 lines

arch/powerpc/net/bpf_jit.h line 84

Quoted 10 lines

  /*
   * Sign-extended 32-bit immediate load
   *

arch/powerpc/net/bpf_jit.h +2 -0
@@ -75,6 +81,8 @@
     /* for tailcall counter */
   #define BPF_PPC_TAILCALL        8
+/* for gpr non volatile registers BPG_REG_6 to 10 */
+#define BPF_PPC_STACK_SAVE      (6*8)

Add spaces before and after the *

Quoted 10 lines
arch/powerpc/net/bpf_jit.h +1 -0
     /* If dummy pass (!image), account for maximum possible instructions */
   #define PPC_LI64(d, i)        do {                          \
@@ -170,6 +178,7 @@ struct codegen_context {
       unsigned int alt_exit_addr;
       u64 arena_vm_start;
       u64 user_vm_start;
+    bool is_subprog;
   };
     #define bpf_to_ppc(r)    (ctx->b2p[r])

arch/powerpc/net/bpf_jit.h void bpf_jit_build_epilogue(u32 *image, struct · line 212

Quoted 9 lines
arch/powerpc/net/bpf_jit.h +0 -2
@@ -204,11 +213,10 @@ void bpf_jit_build_epilogue(u32 *image, struct 
 codegen_context *ctx);
   void bpf_jit_build_fentry_stubs(u32 *image, struct codegen_context *ctx);
   void bpf_jit_realloc_regs(struct codegen_context *ctx);
   int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr);
-
   int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass,
                 struct codegen_context *ctx, int insn_idx,
                 int jmp_off, int dst_reg, u32 code);
-

Not sure why this patch needs to remove those blank lines here and above.

Quoted 174 lines
arch/powerpc/net/bpf_jit.h +1 -0
+int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx);
   #endif
     #endif
arch/powerpc/net/bpf_jit_comp.c +0 -0
index 5e976730b2f5..069a8822c30d 100644
@@ -206,6 +206,7 @@ struct bpf_prog *bpf_int_jit_compile(struct 
 bpf_prog *fp)
       cgctx.stack_size = round_up(fp->aux->stack_depth, 16);

      cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux-

Quoted 1 line

arena);

      cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux-

Quoted 1 line

arena);

+    cgctx.is_subprog = bpf_is_subprog(fp);
        /* Scouting faux-generate pass 0 */
      if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {

arch/powerpc/net/bpf_jit_comp.c +4 -0
@@ -435,6 +436,11 @@ void bpf_jit_free(struct bpf_prog *fp)
       bpf_prog_unlock_free(fp);
   }
   +bool bpf_jit_supports_subprog_tailcalls(void)
+{
+    return IS_ENABLED(CONFIG_PPC64);
+}
+
   bool bpf_jit_supports_kfunc_call(void)
   {
       return true;
@@ -604,7 +610,7 @@ static void 

bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_contex int func_frame_offset, int r4_off)
  {
      if (IS_ENABLED(CONFIG_PPC64)) {
-        /* See bpf_jit_stack_tailcallcnt() */
+        /* See bpf_jit_stack_tailcallinfo_offset() */
          int tailcallcnt_offset = 7 * 8;
            EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));

arch/powerpc/net/bpf_jit_comp.c +0 -0
@@ -619,7 +625,7 @@ static void 

bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont int func_frame_offset, int r4_off) {
      if (IS_ENABLED(CONFIG_PPC64)) {
-        /* See bpf_jit_stack_tailcallcnt() */
+        /* See bpf_jit_stack_tailcallinfo_offset() */ int tailcallcnt_offset = 7 * 8; EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));

arch/powerpc/net/bpf_jit_comp64.c +5 -1
index 39061cd742c1..cebf81fbd59f 100644
@@ -26,8 +26,12 @@
    * Ensure the top half (upto local_tmp_var) stays consistent
    * with our redzone usage.
    *
+ * tail_call_info - stores tailcall count value in main program's
+ *                  frame, stores reference to tail_call_info of
+ *                  main's frame in sub-prog's frame.
+ *
    *        [    prev sp        ] <-------------
- *        [    tail_call_cnt    ] 8        |
+ *        [    tail_call_info    ] 8        |
    *        [   nv gpr save area    ] 6*8        |
    *        [    local_tmp_var    ] 24        |
    * fp (r31) -->    [   ebpf stack space    ] upto 512    |
@@ -35,8 +39,6 @@
    * sp (r1) --->    [    stack pointer    ] --------------
    */

  -/* for gpr non volatile registers BPG_REG_6 to 10 */ -#define BPF_PPC_STACK_SAVE    (6*8)
  /* for bpf JIT code internal usage */
  #define BPF_PPC_STACK_LOCALS    24
  /* stack frame excluding BPF stack, ensure this is quadword aligned */

arch/powerpc/net/bpf_jit_comp64.c +1 -1
@@ -98,7 +100,7 @@ static inline bool bpf_has_stack_frame(struct 
 codegen_context *ctx)
    *        [    prev sp        ] <-------------
    *        [      ...           ]         |
    * sp (r1) --->    [    stack pointer    ] --------------
- *        [    tail_call_cnt    ] 8
+ *        [    tail_call_info    ] 8
    *        [   nv gpr save area    ] 6*8
    *        [    local_tmp_var    ] 24

   *        [   unused red zone    ] 224

arch/powerpc/net/bpf_jit_comp64.c +1 -0
@@ -114,7 +116,7 @@ static int bpf_jit_stack_local(struct 
 codegen_context *ctx)
       }
   }
   -static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx)
+int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)
   {
       return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE;

  }

arch/powerpc/net/bpf_jit_comp64.c +13 -6
@@ -147,17 +149,32 @@ void bpf_jit_build_prologue(u32 *image, struct 
 codegen_context *ctx)
   #endif
         /*
-     * Initialize tail_call_cnt if we do tail calls.
-     * Otherwise, put in NOPs so that it can be skipped when we are
-     * invoked through a tail call.
+     * Tail call count(tcc) is saved & updated only in main
+     * program's frame and the address of tcc in main program's
+     * frame (tcc_ptr) is saved in subprogs frame.
+     *
+     * Offset of tail_call_info on any frame will be interpreted
+     * as either tcc_ptr or tcc value depending on whether it is
+     * greater than MAX_TAIL_CALL_CNT or not.
        */
-    if (ctx->seen & SEEN_TAILCALL) {
+    if (!ctx->is_subprog) {
           EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0));
           /* this goes in the redzone */
           EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, - (BPF_PPC_TAILCALL)));
       } else {
-        EMIT(PPC_RAW_NOP());
-        EMIT(PPC_RAW_NOP());
+        /*
+         * if tail_call_info < MAX_TAIL_CALL_CNT
+         *     main prog calling first subprog -> copy reference
+         * else
+         *     subsequent subprog calling another subprog -> 

directly copy content
+         */
+        EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, 0)); +        EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), -(BPF_PPC_TAILCALL)));
+        EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
+        PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8); +        EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2),
+                            -(BPF_PPC_TAILCALL))); +        EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, - (BPF_PPC_TAILCALL)));
      }
        if (bpf_has_stack_frame(ctx)) {

arch/powerpc/net/bpf_jit_comp64.c +0 -0
@@ -352,19 +369,38 @@ static int bpf_jit_emit_tail_call(u32 *image, 

struct codegen_context *ctx, u32 o
      EMIT(PPC_RAW_CMPLW(b2p_index, bpf_to_ppc(TMP_REG_1))); PPC_BCC_SHORT(COND_GE, out);
  +    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallinfo_offset(ctx))); +    EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
+    PPC_COND_BRANCH(COND_LE, CTX_NIA(ctx) + 8);

arch/powerpc/net/bpf_jit_comp64.c +4 -0
+
+    /* dereference TMP_REG_1 */
+    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 0));
+

      /*
-     * if (tail_call_cnt >= MAX_TAIL_CALL_CNT)
+     * if (tail_call_info == MAX_TAIL_CALL_CNT)
       *   goto out;
       */
-    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx))); EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT)); -    PPC_BCC_SHORT(COND_GE, out);
+    PPC_COND_BRANCH(COND_EQ, out);
        /*
-     * tail_call_cnt++;
+     * tail_call_info++; <- Actual value of tcc here
       */
      EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 1)); -    EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
+
+    /*
+     * Before writing updated tail_call_info, distinguish if current frame
+     * is storing a reference to tail_call_info or actual tcc value in +     * tail_call_info.
+     */
+    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+    EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_2), MAX_TAIL_CALL_CNT)); +    PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+

Quoted 8 lines
Quoted 8 lines

+    /* First get address of tail_call_info */
+    EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+    /* Writeback updated value to tail_call_info */
+    EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), 0));
        /* prog = array->ptrs[index]; */ EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_1), b2p_index, 8));

-Abhishek

maintainer
#12 lore raw

Re: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs

Quoted 5 lines · Christophe Leroy (CS GROUP)

Le 16/01/2026 à 05:50, Hari Bathini a écrit :

Not received this mail that Hari is reponding to.

That is weird.

Quoted 89 lines
Quoted 52 lines · adubey
Quoted 49 lines · Christophe Leroy (CS GROUP)
Quoted 43 lines

Le 14/01/2026 à 12:44, [email protected] a écrit :

Quoted 28 lines

From: Abhishek Dubey <[email protected]>

Enabling tailcalls with subprog combinations by referencing method. The actual tailcall count is always maintained in the tail_call_info variable present in the frame of main function (also called entry function). The tail_call_info variables in the frames of all other subprog contains reference to the tail_call_info present in frame of main function.

Dynamic resolution interprets the tail_call_info either as value or reference depending on the context of active frame while tailcall is invoked.

---
  arch/powerpc/net/bpf_jit.h        | 12 +++++- arch/powerpc/net/bpf_jit_comp.c   | 10 ++++- arch/powerpc/net/bpf_jit_comp64.c | 68 ++++++++++++++++++++++
+--------

  3 files changed, 70 insertions(+), 20 deletions(-)
arch/powerpc/net/bpf_jit.h +0 -0
index 45d419c0ee73..5d735bc5e6bd 100644
@@ -51,6 +51,12 @@
           EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));                    \
       } while (0)
Quoted 7 lines

  +/* Same as PPC_BCC_SHORT, except valid dest is known prior to call. */ +#define PPC_COND_BRANCH(cond, dest)         \
+    do {                                      \
+        long offset = (long)(dest) - CTX_NIA(ctx);              \
+        EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));    \
+    } while (0)

I don't like the idea of duplicating PPC_BCC_SHORT() to just kick the verification out. Now we will have two macros doing the exact same thing with one handling failure case and one ignoring failure case. There is a big risk that one day or another someone will use the wrong macro.

Quoted 2 lines

Could you change bpf_jit_build_prologue() to return an int add use PPC_BCC_SHORT() instead of that new PPC_COND_BRANCH() ?

I implemented exactly same change in bpf_jit_build_prologue(). But, during internal review, @HariBathini suggested to have separate macro with a caution note.

Quoted 7 lines
Quoted 1 line

@Hari please suggest here!

Not just about the change of return type but the check seems like an overkill for cases where the offset is known and within branch range. How about using BUILD_BUG_ON() to avoid unecessary checks and return type change for places where the branch offset is known and is a constant?

When offset is a constant known at build time, checks are eliminated by gcc at build, see exemple below from disasembly of bpf_jit_comp32.o, there are no checks.

                                PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) *
4);
    36d8:       3c 80 41 81     lis     r4,16769 EMIT(PPC_RAW_CMPLW(src_reg, _R0)); 36dc:       81 3f 00 04     lwz     r9,4(r31) PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) *
4);
    36e0:       60 84 00 10     ori     r4,r4,16 EMIT(PPC_RAW_CMPLW(src_reg, _R0)); 36e4:       39 29 00 01     addi    r9,r9,1 PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) *
4);
    36e8:       55 23 10 3a     slwi    r3,r9,2 EMIT(PPC_RAW_CMPLW(src_reg, _R0)); 36ec:       91 3f 00 04     stw     r9,4(r31) PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) *
4);
    36f0:       7c 97 19 2e     stwx    r4,r23,r3 EMIT(PPC_RAW_LI(dst_reg, 0)); 36f4:       55 49 a9 94     rlwinm  r9,r10,21,6,10 PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) *
4);
    36f8:       80 9f 00 04     lwz     r4,4(r31) EMIT(PPC_RAW_LI(dst_reg, 0)); 36fc:       65 29 38 00     oris    r9,r9,14336 PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) *
4);

Quoted 9 lines

    3700:       38 84 00 01     addi    r4,r4,1 EMIT(PPC_RAW_LI(dst_reg, 0)); 3704:       54 83 10 3a     slwi    r3,r4,2 PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) *
4);
    3708:       90 9f 00 04     stw     r4,4(r31) EMIT(PPC_RAW_LI(dst_reg, 0)); 370c:       7d 37 19 2e     stwx    r9,r23,r3

Interesting. I do see is_offset_in_cond_branch_range() in action with constant offsets too, on ppc64 compile at least. fwiw, I had this optimized version in mind for constant offset:

  #define PPC_BCC_CONST_SHORT(cond, offset) 
                    \
        do { 
                    \
                BUILD_BUG_ON(offset < -0x8000 || offset > 0x7fff || 
(offset & 0x3));        \
                EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | 
(offset & 0xfffc));  \
        } while (0)

With that, something like:

    PPC_BCC_SHORT(COND_NE, (ctx->idx + 3) * 4);

becomes

    PPC_BCC_CONST_SHORT(COND_NE, 12);

- Hari

maintainer
#13 lore raw

Re: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs

In reply to [email protected]
Quoted 8 lines · On 14/01/26 5:14 pm, [email protected] wrote:

From: Abhishek Dubey<[email protected]>

Enabling tailcalls with subprog combinations by referencing method. The actual tailcall count is always maintained in the tail_call_info variable present in the frame of main function (also called entry function). The tail_call_info variables in the frames of all other subprog contains reference to the tail_call_info present in frame of main function.

I would rephrase:

"The tail_call_info variables in the frames of all other subprog contains reference to the tail_call_info present in frame of main function."

to:

"The tail_call_info field in the stack frame of subprogs contains reference to the tail_call_info field in the stack frame of main BPF program."

- Hari

#14 lore raw

PATCH v2 3/6 powerpc64/bpf: Tailcall handling with trampolines

In reply to [email protected]

From: Abhishek Dubey <[email protected]>

The trampoline mechanism sets up its own stack frame and an additional dummy frame. We need to have additional JIT instructions handling tailcall dereferencing in the trampoline's context.

We don't add the two stack frames pointed above, rather add space for tail_call_info at bottom in trampoline frame for ppc64. This makes the trampoline's frame consistent with layout of all other frames wrt tail_call_info offset.

---
 arch/powerpc/net/bpf_jit_comp.c | 83 ++++++++++++++++++++++-----------
 1 file changed, 56 insertions(+), 27 deletions(-)
arch/powerpc/net/bpf_jit_comp.c +56 -27
index 069a8822c30d..e3088cf089d1 100644
@@ -606,33 +606,58 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context
 	return 0;
 }
 
-static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_context *ctx,
-					       int func_frame_offset, int r4_off)
-{
-	if (IS_ENABLED(CONFIG_PPC64)) {
-		/* See bpf_jit_stack_tailcallinfo_offset() */
-		int tailcallcnt_offset = 7 * 8;
-
-		EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
-		EMIT(PPC_RAW_STL(_R3, _R1, -tailcallcnt_offset));
-	} else {
-		/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
-		EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
-	}
-}
+/*
+ * Refer the label 'Generated stack layout' in this file for actual stack
+ * layout during trampoline invocation.
+ *
+ * Refer __arch_prepare_bpf_trampoline() for stack component details.
+ *
+ * The tailcall count/reference is present in caller's stack frame. Its required
+ * to copy the content of tail_call_info before calling the actual function
+ * to which the trampoline is attached.
+ *
+ */
 
-static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_context *ctx,
-						 int func_frame_offset, int r4_off)
+static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx,
+					       int func_frame_offset,
+					       int bpf_dummy_frame_size, int r4_off)
 {
 	if (IS_ENABLED(CONFIG_PPC64)) {
 		/* See bpf_jit_stack_tailcallinfo_offset() */
-		int tailcallcnt_offset = 7 * 8;
+		int tailcallinfo_offset = BPF_PPC_TAILCALL;
+		/*
+		 * func_frame_offset =                                   ...(1)
+		 *     bpf_dummy_frame_size + trampoline_frame_size
+		 */
+		EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset));
+		EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset));
+
+		/*
+		 * Setting the tail_call_info in trampoline's frame
+		 * depending on if previous frame had value or reference.
+		 */
+		EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT));
+		PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+		EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx)));
+		/*
+		 * From ...(1) above:
+		 * trampoline_frame_bottom =                            ...(2)
+		 *     func_frame_offset - bpf_dummy_frame_size
+		 *
+		 * Using ...(2) derived above:
+		 *  trampoline_tail_call_info_offset =                  ...(3)
+		 *      trampoline_frame_bottom - tailcallinfo_offset
+		 *
+		 * From ...(3):
+		 * Use trampoline_tail_call_info_offset to write reference of main's
+		 * tail_call_info in trampoline frame.
+		 */
+		EMIT(PPC_RAW_STL(_R3, _R1, (func_frame_offset - bpf_dummy_frame_size)
+					- tailcallinfo_offset));
 
-		EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
-		EMIT(PPC_RAW_STL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
 	} else {
 		/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
-		EMIT(PPC_RAW_STL(_R4, _R1, r4_off));
+		EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
 	}
 }
 
@@ -720,6 +745,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	 * LR save area                 [ r0 save (64-bit)  ]   | header
 	 *                              [ r0 save (32-bit)  ]   |
 	 * dummy frame for unwind       [ back chain 1      ] --
+	 *                              [ tail_call_info    ] non optional - 64-bit powerpc
 	 *                              [ padding           ] align stack frame
 	 *       r4_off                 [ r4 (tailcallcnt)  ] optional - 32-bit powerpc
 	 *       alt_lr_off             [ real lr (ool stub)] optional - actual lr
@@ -801,8 +827,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		}
 	}
 
-	/* Padding to align stack frame, if any */
-	bpf_frame_size = round_up(bpf_frame_size, SZL * 2);
+	if (!(bpf_frame_size % (2 * SZL))) {
+		/* Stack is 16-byte aligned */
+		/* Room for padding followed by 64-bit tail_call_info */
+		bpf_frame_size += SZL + BPF_PPC_TAILCALL;
+	} else {
+		/* Room for 64-bit tail_call_info */
+		bpf_frame_size += BPF_PPC_TAILCALL;
+	}
 
 	/* Dummy frame size for proper unwind - includes 64-bytes red zone for 64-bit powerpc */
 	bpf_dummy_frame_size = STACK_FRAME_MIN_SIZE + 64;
@@ -902,7 +934,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 
 		/* Replicate tail_call_cnt before calling the original BPF prog */
 		if (flags & BPF_TRAMP_F_TAIL_CALL_CTX)
-			bpf_trampoline_setup_tail_call_cnt(image, ctx, func_frame_offset, r4_off);
+			bpf_trampoline_setup_tail_call_info(image, ctx, func_frame_offset,
+							   bpf_dummy_frame_size, r4_off);
 
 		/* Restore args */
 		bpf_trampoline_restore_args_stack(image, ctx, func_frame_offset, nr_regs, regs_off);
@@ -917,10 +950,6 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		/* Store return value for bpf prog to access */
 		EMIT(PPC_RAW_STL(_R3, _R1, retval_off));
 
-		/* Restore updated tail_call_cnt */
-		if (flags & BPF_TRAMP_F_TAIL_CALL_CTX)
-			bpf_trampoline_restore_tail_call_cnt(image, ctx, func_frame_offset, r4_off);
-
 		/* Reserve space to patch branch instruction to skip fexit progs */
 		if (ro_image) /* image is NULL for dummy pass */
 			im->ip_after_call = &((u32 *)ro_image)[ctx->idx];
Signature · 1 line
2.48.1
#15 lore raw

Re: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines

Should this LD instruction be an ADDI instead? Looking at the stack frame setup, after the two STLU instructions in __arch_prepare_bpf_trampoline():
Quoted 4 lines
arch/powerpc/net/bpf_jit_comp.c +0 -0
index 069a8822c30d..e3088cf089d1 100644

[ ... ]

arch/powerpc/net/bpf_jit_comp.c static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context · line 633

Quoted 13 lines
arch/powerpc/net/bpf_jit_comp.c +13 -0
+static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx,
+					       int func_frame_offset,
+					       int bpf_dummy_frame_size, int r4_off)
+{
+	if (IS_ENABLED(CONFIG_PPC64)) {
+		/* See bpf_jit_stack_tailcallinfo_offset() */
+		int tailcallinfo_offset = BPF_PPC_TAILCALL;
+		/*
+		 * func_frame_offset =                                   ...(1)
+		 *     bpf_dummy_frame_size + trampoline_frame_size
+		 */
+		EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset));
+		EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset));
                              ^^^^

Should this LD instruction be an ADDI instead? Looking at the stack frame setup, after the two STLU instructions in __arch_prepare_bpf_trampoline():

  STLU _R1, _R1, -bpf_dummy_frame_size  ->  stores original _R1 at dummy frame
  STLU _R1, _R1, -bpf_frame_size        ->  stores dummy frame ptr at tramp frame

From the trampoline frame, the back chain stored in the dummy frame at offset bpf_frame_size points to the caller's frame. But func_frame_offset equals (bpf_dummy_frame_size + bpf_frame_size), which points into the caller's frame itself. Loading from that address returns the caller's back chain (pointing to the grandparent frame), not a pointer to the caller's frame.

This appears to cause the subsequent load at [_R4 - 8] to read the grandparent's tail_call_info rather than the caller's. Would loading from [_R1 + bpf_frame_size] give the correct pointer to the caller's frame?

For comparison, the subprogram prologue in bpf_jit_build_prologue() uses [_R1 + 0] to get the back chain pointing to its caller, which seems like the right pattern.

arch/powerpc/net/bpf_jit_comp.c static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context · line 641

Quoted 8 lines
arch/powerpc/net/bpf_jit_comp.c +8 -0
+
+		/*
+		 * Setting the tail_call_info in trampoline's frame
+		 * depending on if previous frame had value or reference.
+		 */
+		EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT));
+		PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+		EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx)));

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/20993216190

#16 lore raw

Re: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines

In reply to [email protected]
kernel test robot noticed the following build errors:

Hi,

kernel test robot noticed the following build errors:

[auto build test ERROR on powerpc/next]
[also build test ERROR on powerpc/fixes linus/master v6.19-rc5 next-20260114] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/adubey-linux-ibm-com/powerpc64-bpf-Move-tail_call_cnt-to-bottom-of-stack-frame/20260114-195044
base:   https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next
patch link:    https://lore.kernel.org/r/20260114114450.30405-4-adubey%40linux.ibm.com
patch subject: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines
config: powerpc-randconfig-002-20260114 (https://download.01.org/0day-ci/archive/20260115/[email protected]/config)
compiler: powerpc-linux-gcc (GCC) 12.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260115/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

arch/powerpc/net/bpf_jit_comp.c: In function 'bpf_trampoline_setup_tail_call_info':

>> arch/powerpc/net/bpf_jit_comp.c:644:43: error: 'BPF_PPC_TAILCALL' undeclared (first use in this function); did you mean 'BPF_TAIL_CALL'?
     644 |                 int tailcallinfo_offset = BPF_PPC_TAILCALL;
         |                                           ^~~~~~~~~~~~~~~~
         |                                           BPF_TAIL_CALL
   arch/powerpc/net/bpf_jit_comp.c:644:43: note: each undeclared identifier is reported only once for each function it appears in
   arch/powerpc/net/bpf_jit_comp.c: In function '__arch_prepare_bpf_trampoline':
   arch/powerpc/net/bpf_jit_comp.c:850:41: error: 'BPF_PPC_TAILCALL' undeclared (first use in this function); did you mean 'BPF_TAIL_CALL'?
     850 |                 bpf_frame_size += SZL + BPF_PPC_TAILCALL;
         |                                         ^~~~~~~~~~~~~~~~
         |                                         BPF_TAIL_CALL

vim +644 arch/powerpc/net/bpf_jit_comp.c

625
626 /*
627 * Refer the label 'Generated stack layout' in this file for actual stack
628 * layout during trampoline invocation.
629 *
630 * Refer __arch_prepare_bpf_trampoline() for stack component details.
631 *
632 * The tailcall count/reference is present in caller's stack frame. Its required
633 * to copy the content of tail_call_info before calling the actual function
634 * to which the trampoline is attached.
635 *
636 */
637
638 static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx,
639 int func_frame_offset,
640 int bpf_dummy_frame_size, int r4_off)
641 {
642 if (IS_ENABLED(CONFIG_PPC64)) {
643 /* See bpf_jit_stack_tailcallinfo_offset() */

 > 644			int tailcallinfo_offset = BPF_PPC_TAILCALL;

645 /*
646 * func_frame_offset = ...(1)
647 * bpf_dummy_frame_size + trampoline_frame_size
648 */
649 EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset));
650 EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset));
651
652 /*
653 * Setting the tail_call_info in trampoline's frame
654 * depending on if previous frame had value or reference.
655 */
656 EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT));
657 PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
658 EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx)));
659 /*
660 * From ...(1) above:
661 * trampoline_frame_bottom = ...(2)
662 * func_frame_offset - bpf_dummy_frame_size
663 *
664 * Using ...(2) derived above:
665 * trampoline_tail_call_info_offset = ...(3)
666 * trampoline_frame_bottom - tailcallinfo_offset
667 *
668 * From ...(3):
669 * Use trampoline_tail_call_info_offset to write reference of main's
670 * tail_call_info in trampoline frame.
671 */
672 EMIT(PPC_RAW_STL(_R3, _R1, (func_frame_offset - bpf_dummy_frame_size)
673 - tailcallinfo_offset));
674
675 } else {
676 /* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
677 EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
678 }
679 }
680

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
maintainer
#17 lore raw

Re: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines

In reply to [email protected]
Quoted 52 lines · On 14/01/26 5:14 pm, [email protected] wrote:

From: Abhishek Dubey <[email protected]>

The trampoline mechanism sets up its own stack frame and an additional dummy frame. We need to have additional JIT instructions handling tailcall dereferencing in the trampoline's context.

We don't add the two stack frames pointed above, rather add space for tail_call_info at bottom in trampoline frame for ppc64. This makes the trampoline's frame consistent with layout of all other frames wrt tail_call_info offset.

---
  arch/powerpc/net/bpf_jit_comp.c | 83 ++++++++++++++++++++++-----------
  1 file changed, 56 insertions(+), 27 deletions(-)
arch/powerpc/net/bpf_jit_comp.c +11 -15
index 069a8822c30d..e3088cf089d1 100644
@@ -606,33 +606,58 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context
  	return 0;
  }
 
-static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_context *ctx,
-					       int func_frame_offset, int r4_off)
-{
-	if (IS_ENABLED(CONFIG_PPC64)) {
-		/* See bpf_jit_stack_tailcallinfo_offset() */
-		int tailcallcnt_offset = 7 * 8;
-
-		EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
-		EMIT(PPC_RAW_STL(_R3, _R1, -tailcallcnt_offset));
-	} else {
-		/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
-		EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
-	}
-}
+/*
+ * Refer the label 'Generated stack layout' in this file for actual stack
+ * layout during trampoline invocation.
+ *
+ * Refer __arch_prepare_bpf_trampoline() for stack component details.
+ *
+ * The tailcall count/reference is present in caller's stack frame. Its required
+ * to copy the content of tail_call_info before calling the actual function
+ * to which the trampoline is attached.
+ *
+ */
 
-static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_context *ctx,

arch/powerpc/net/bpf_jit_comp.c static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context · line 628

Quoted 10 lines
arch/powerpc/net/bpf_jit_comp.c +4 -2
-						 int func_frame_offset, int r4_off)
+static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx,
+					       int func_frame_offset,
+					       int bpf_dummy_frame_size, int r4_off)
  {
  	if (IS_ENABLED(CONFIG_PPC64)) {
  		/* See bpf_jit_stack_tailcallinfo_offset() */
 
-		int tailcallcnt_offset = 7 * 8;
+		int tailcallinfo_offset = BPF_PPC_TAILCALL;

This offset update should have been part of patch#1

Quoted 55 lines
arch/powerpc/net/bpf_jit_comp.c +32 -5
+		/*
+		 * func_frame_offset =                                   ...(1)
+		 *     bpf_dummy_frame_size + trampoline_frame_size
+		 */
+		EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset));
+		EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset));
+
+		/*
+		 * Setting the tail_call_info in trampoline's frame
+		 * depending on if previous frame had value or reference.
+		 */
+		EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT));
+		PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+		EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx)));
+		/*
+		 * From ...(1) above:
+		 * trampoline_frame_bottom =                            ...(2)
+		 *     func_frame_offset - bpf_dummy_frame_size
+		 *
+		 * Using ...(2) derived above:
+		 *  trampoline_tail_call_info_offset =                  ...(3)
+		 *      trampoline_frame_bottom - tailcallinfo_offset
+		 *
+		 * From ...(3):
+		 * Use trampoline_tail_call_info_offset to write reference of main's
+		 * tail_call_info in trampoline frame.
+		 */
+		EMIT(PPC_RAW_STL(_R3, _R1, (func_frame_offset - bpf_dummy_frame_size)
+					- tailcallinfo_offset));
 
-		EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
-		EMIT(PPC_RAW_STL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
  	} else {
  		/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
-		EMIT(PPC_RAW_STL(_R4, _R1, r4_off));
+		EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
  	}
  }
@@ -720,6 +745,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
  	 * LR save area                 [ r0 save (64-bit)  ]   | header
  	 *                              [ r0 save (32-bit)  ]   |
  	 * dummy frame for unwind       [ back chain 1      ] --
+	 *                              [ tail_call_info    ] non optional - 64-bit powerpc
  	 *                              [ padding           ] align stack frame
  	 *       r4_off                 [ r4 (tailcallcnt)  ] optional - 32-bit powerpc
  	 *       alt_lr_off             [ real lr (ool stub)] optional - actual lr
@@ -801,8 +827,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
  		}
  	}
 
-	/* Padding to align stack frame, if any */
-	bpf_frame_size = round_up(bpf_frame_size, SZL * 2);
+	if (!(bpf_frame_size % (2 * SZL))) {

arch/powerpc/net/bpf_jit_comp.c static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · line 840

Quoted 10 lines
arch/powerpc/net/bpf_jit_comp.c +7 -0
+		/* Stack is 16-byte aligned */
+		/* Room for padding followed by 64-bit tail_call_info */
+		bpf_frame_size += SZL + BPF_PPC_TAILCALL;
+	} else {
+		/* Room for 64-bit tail_call_info */
+		bpf_frame_size += BPF_PPC_TAILCALL;
+	}
 
  	/* Dummy frame size for proper unwind - includes 64-bytes red zone for 64-bit powerpc */
  	bpf_dummy_frame_size = STACK_FRAME_MIN_SIZE + 64;

This change assumes the size is at least 8-byte aligned which is true today but better skip that assumption by not touching the padding part. The above hunk could simply be:

arch/powerpc/net/bpf_jit_comp.c +4 -0
index 5e976730b2f5..266cc6f17dcc 100644
@@ -795,6 +795,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
                }
        }
 
+       /* Save tailcall count pointer at the same offset on the stack where subprogs expect it */
+       if ((flags & BPF_TRAMP_F_CALL_ORIG) && (flags & BPF_TRAMP_F_TAIL_CALL_CTX))
+               bpf_frame_size += SZL;
+
        /* Padding to align stack frame, if any */
        bpf_frame_size = round_up(bpf_frame_size, SZL * 2);
 

Patch#2 is not complete without this change. Please fold this patch into patch#2 itself.

- Hari

maintainer
#18 lore raw

Re: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines

Quoted 69 lines · Hari Bathini
Quoted 65 lines · On 14/01/26 5:14 pm, [email protected] wrote:

From: Abhishek Dubey <[email protected]>

The trampoline mechanism sets up its own stack frame and an additional dummy frame. We need to have additional JIT instructions handling tailcall dereferencing in the trampoline's context.

We don't add the two stack frames pointed above, rather add space for tail_call_info at bottom in trampoline frame for ppc64. This makes the trampoline's frame consistent with layout of all other frames wrt tail_call_info offset.

---
  arch/powerpc/net/bpf_jit_comp.c | 83 ++++++++++++++++++++++-----------

  1 file changed, 56 insertions(+), 27 deletions(-)
arch/powerpc/net/bpf_jit_comp.c +2 -14
index 069a8822c30d..e3088cf089d1 100644
@@ -606,33 +606,58 @@ static int invoke_bpf_mod_ret(u32 *image, u32 
 *ro_image, struct codegen_context
       return 0;
   }
-static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct 
 codegen_context *ctx,
-                           int func_frame_offset, int r4_off)
-{
-    if (IS_ENABLED(CONFIG_PPC64)) {
-        /* See bpf_jit_stack_tailcallinfo_offset() */
-        int tailcallcnt_offset = 7 * 8;
-
-        EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - 
 tailcallcnt_offset));
-        EMIT(PPC_RAW_STL(_R3, _R1, -tailcallcnt_offset));
-    } else {
-        /* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
-        EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
-    }
-}
+/*
+ * Refer the label 'Generated stack layout' in this file for actual 

stack
+ * layout during trampoline invocation.
+ *
+ * Refer __arch_prepare_bpf_trampoline() for stack component details.
+ *
+ * The tailcall count/reference is present in caller's stack frame. Its required
+ * to copy the content of tail_call_info before calling the actual function
+ * to which the trampoline is attached.
+ *
+ */
-static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_context *ctx,
-                         int func_frame_offset, int r4_off) +static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx,
+                           int func_frame_offset,
+                           int bpf_dummy_frame_size, int r4_off) {
      if (IS_ENABLED(CONFIG_PPC64)) {
          /* See bpf_jit_stack_tailcallinfo_offset() */

Quoted 54 lines
Quoted 2 lines

-        int tailcallcnt_offset = 7 * 8;
+        int tailcallinfo_offset = BPF_PPC_TAILCALL;

This offset update should have been part of patch#1

Quoted 49 lines

+        /*
+         * func_frame_offset =                                   ...(1) +         *     bpf_dummy_frame_size + trampoline_frame_size +         */
+        EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset));
+        EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset));
+
+        /*
+         * Setting the tail_call_info in trampoline's frame +         * depending on if previous frame had value or reference. +         */
+        EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT));
+        PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+        EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx)));
+        /*
+         * From ...(1) above:
+         * trampoline_frame_bottom =                            ...(2) +         *     func_frame_offset - bpf_dummy_frame_size
+         *
+         * Using ...(2) derived above:
+         *  trampoline_tail_call_info_offset =                  ...(3) +         *      trampoline_frame_bottom - tailcallinfo_offset +         *
+         * From ...(3):
+         * Use trampoline_tail_call_info_offset to write reference of main's
+         * tail_call_info in trampoline frame.
+         */
+        EMIT(PPC_RAW_STL(_R3, _R1, (func_frame_offset - bpf_dummy_frame_size)
+                    - tailcallinfo_offset));
-        EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
-        EMIT(PPC_RAW_STL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
      } else {
          /* See bpf_jit_stack_offsetof() and BPF_PPC_TC */ -        EMIT(PPC_RAW_STL(_R4, _R1, r4_off));
+        EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
      }
  }

arch/powerpc/net/bpf_jit_comp.c +0 -0
@@ -720,6 +745,7 @@ static int __arch_prepare_bpf_trampoline(struct 

bpf_tramp_image *im, void *rw_im
       * LR save area                 [ r0 save (64-bit)  ]   | header *                              [ r0 save (32-bit)  ]   |
       * dummy frame for unwind       [ back chain 1      ] --

+     *                              [ tail_call_info    ] non optional - 64-bit powerpc

Also, why non-optional? This can be optional on BPF_TRAMP_F_CALL_ORIG and BPF_TRAMP_F_TAIL_CALL_CTX flags?

Quoted 44 lines
Quoted 12 lines

       *                              [ padding           ] align stack frame *       r4_off                 [ r4 (tailcallcnt)  ] optional - 32-bit powerpc *       alt_lr_off             [ real lr (ool stub)] optional - actual lr

arch/powerpc/net/bpf_jit_comp.c +0 -0
@@ -801,8 +827,14 @@ static int __arch_prepare_bpf_trampoline(struct 

bpf_tramp_image *im, void *rw_im
          }
      }
-    /* Padding to align stack frame, if any */ -    bpf_frame_size = round_up(bpf_frame_size, SZL * 2); +    if (!(bpf_frame_size % (2 * SZL))) {
+        /* Stack is 16-byte aligned */

Quoted 8 lines

+        /* Room for padding followed by 64-bit tail_call_info */
+        bpf_frame_size += SZL + BPF_PPC_TAILCALL;
+    } else {
+        /* Room for 64-bit tail_call_info */
+        bpf_frame_size += BPF_PPC_TAILCALL;
+    }
      /* Dummy frame size for proper unwind - includes 64-bytes red zone for 64-bit powerpc */ bpf_dummy_frame_size = STACK_FRAME_MIN_SIZE + 64;

This change assumes the size is at least 8-byte aligned which is true today but better skip that assumption by not touching the padding part. The above hunk could simply be:

arch/powerpc/net/bpf_jit_comp.c +0 -0
index 5e976730b2f5..266cc6f17dcc 100644
@@ -795,6 +795,10 @@ static int __arch_prepare_bpf_trampoline(struct 

bpf_tramp_image *im, void *rw_im
                }
        }

+       /* Save tailcall count pointer at the same offset on the stack where subprogs expect it */
+       if ((flags & BPF_TRAMP_F_CALL_ORIG) && (flags & BPF_TRAMP_F_TAIL_CALL_CTX))
+               bpf_frame_size += SZL;
+
        /* Padding to align stack frame, if any */ bpf_frame_size = round_up(bpf_frame_size, SZL * 2);

Quoted 3 lines

Patch#2 is not complete without this change. Please fold this patch into patch#2 itself.

- Hari

#19 lore raw

PATCH v2 4/6 powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT

In reply to [email protected]

From: Abhishek Dubey <[email protected]>

This function is used by bpf_throw() to unwind the stack until frame of exception-boundary during BPF exception handling.

This function is necessary to support BPF exceptions on PowerPC.

---
 arch/powerpc/net/bpf_jit_comp64.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
arch/powerpc/net/bpf_jit_comp64.c +28 -0
index cebf81fbd59f..ec58395f74f7 100644
@@ -247,6 +247,34 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)
 	bpf_jit_build_fentry_stubs(image, ctx);
 }
 
+void arch_bpf_stack_walk(bool (*consume_fn)(void *, u64, u64, u64), void *cookie)
+{
+	// callback processing always in current context
+	unsigned long fp = current_stack_frame();
+
+	for (;;) {
+		unsigned long *frame = (unsigned long *) fp;
+		unsigned long ip;
+
+		if (!validate_sp(fp, current))
+			return;
+
+		ip = frame[STACK_FRAME_LR_SAVE];
+		if (!ip)
+			break;
+
+		/*
+		 * consume_fn common code expects stack pointer(sp) in third
+		 * argument. There is no sp in ppc64, rather pass frame
+		 * pointer.
+		 */
+		if (ip && !consume_fn(cookie, ip, fp, fp))
+			break;
+
+		fp = frame[0];
+	}
+}
+
 int bpf_jit_emit_func_call_rel(u32 *image, u32 *fimage, struct codegen_context *ctx, u64 func)
 {
 	unsigned long func_addr = func ? ppc_function_entry((void *)func) : 0;
Signature · 1 line
2.48.1
#20 lore raw

Re: [PATCH v2 4/6] powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT

Le 14/01/2026 à 12:44, [email protected] a écrit :

Quoted 40 lines

From: Abhishek Dubey <[email protected]>

This function is used by bpf_throw() to unwind the stack until frame of exception-boundary during BPF exception handling.

This function is necessary to support BPF exceptions on PowerPC.

---
  arch/powerpc/net/bpf_jit_comp64.c | 28 ++++++++++++++++++++++++++++
  1 file changed, 28 insertions(+)
arch/powerpc/net/bpf_jit_comp64.c +18 -0
index cebf81fbd59f..ec58395f74f7 100644
@@ -247,6 +247,34 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)
  	bpf_jit_build_fentry_stubs(image, ctx);
  }
 
+void arch_bpf_stack_walk(bool (*consume_fn)(void *, u64, u64, u64), void *cookie)
+{
+	// callback processing always in current context
+	unsigned long fp = current_stack_frame();
+
+	for (;;) {
+		unsigned long *frame = (unsigned long *) fp;
+		unsigned long ip;
+
+		if (!validate_sp(fp, current))
+			return;
+
+		ip = frame[STACK_FRAME_LR_SAVE];
+		if (!ip)
+			break;
+
+		/*
+		 * consume_fn common code expects stack pointer(sp) in third

arch/powerpc/net/bpf_jit_comp64.c void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx) · line 277

Quoted 10 lines
arch/powerpc/net/bpf_jit_comp64.c +10 -0
+		 * argument. There is no sp in ppc64, rather pass frame
+		 * pointer.
+		 */
+		if (ip && !consume_fn(cookie, ip, fp, fp))
+			break;
+
+		fp = frame[0];
+	}
+}
+

This fonction looks very close to arch_stack_walk(). Would it be possible to refactor and have a common part used by both functions, like ARM64 for instance ?

Quoted 3 lines
arch/powerpc/net/bpf_jit_comp64.c +0 -0
  int bpf_jit_emit_func_call_rel(u32 *image, u32 *fimage, struct codegen_context *ctx, u64 func)
  {
  	unsigned long func_addr = func ? ppc_function_entry((void *)func) : 0;

A message that is not in the archive (sent off-list or not yet received): [email protected]

maintainer
#22 lore raw

Re: [PATCH v2 4/6] powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT

Quoted 56 lines · adubey
Quoted 47 lines · Christophe Leroy (CS GROUP)

Le 14/01/2026 à 12:44, [email protected] a écrit :

Quoted 46 lines

From: Abhishek Dubey <[email protected]>

This function is used by bpf_throw() to unwind the stack until frame of exception-boundary during BPF exception handling.

This function is necessary to support BPF exceptions on PowerPC.

---
  arch/powerpc/net/bpf_jit_comp64.c | 28 ++++++++++++++++++++++++++++

  1 file changed, 28 insertions(+)
arch/powerpc/net/bpf_jit_comp64.c +23 -0
index cebf81fbd59f..ec58395f74f7 100644
@@ -247,6 +247,34 @@ void bpf_jit_build_epilogue(u32 *image, struct 
 codegen_context *ctx)
       bpf_jit_build_fentry_stubs(image, ctx);
   }
   +void arch_bpf_stack_walk(bool (*consume_fn)(void *, u64, u64, u64), void *cookie)
+{
+    // callback processing always in current context
+    unsigned long fp = current_stack_frame();
+
+    for (;;) {
+        unsigned long *frame = (unsigned long *) fp;
+        unsigned long ip;
+
+        if (!validate_sp(fp, current))
+            return;
+
+        ip = frame[STACK_FRAME_LR_SAVE];
+        if (!ip)
+            break;
+
+        /*
+         * consume_fn common code expects stack pointer(sp) in third
+         * argument. There is no sp in ppc64, rather pass frame
+         * pointer.
+         */
+        if (ip && !consume_fn(cookie, ip, fp, fp))
+            break;
+
Quoted 8 lines

arch/powerpc/net/bpf_jit_comp64.c void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx) · line 277

Quoted 4 lines
arch/powerpc/net/bpf_jit_comp64.c +4 -0
+        fp = frame[0];
+    }
+}
+

This fonction looks very close to arch_stack_walk(). Would it be possible to refactor and have a common part used by both functions, like ARM64 for instance ?

Quoted 4 lines

Yes, its inspired from arch_stack_walk(). consume_entry() have different parameter count in both cases. If merged, it need additional handling to identify which call_back to invoke.
Also, we need to define arch-specific weak function arch_bpf_stack_walk(), so renaming of arch_stack_walk is needed on merge. Stack walker logic with "bpf" name might be confusing when used at other places. Thoughts?

Not sure what you mean by renaming of arch_stack_walk is needed on merge but refactoring does not have to change API signature or any common code for that matter..

- Hari

#23 lore raw

PATCH v2 5/6 powerpc64/bpf: Support exceptions

In reply to [email protected]

From: Abhishek Dubey <[email protected]>

The modified prologue/epilogue generation code now enables exception-callback to use the stack frame of the program marked as exception boundary, where callee saved registers are stored.

As per ppc64 ABIv2 documentation[1], r14-r31 are callee saved registers. BPF programs on ppc64 already saves r26-r31 registers. Saving the remaining set of callee saved registers(r14-r25) is handled in the next patch.

[1] https://ftp.rtems.org/pub/rtems/people/sebh/ABI64BitOpenPOWERv1.1_16July2015_pub.pdf

---
 arch/powerpc/net/bpf_jit.h        |  2 ++
 arch/powerpc/net/bpf_jit_comp.c   |  7 ++++
 arch/powerpc/net/bpf_jit_comp64.c | 53 +++++++++++++++++++++----------
 3 files changed, 45 insertions(+), 17 deletions(-)
arch/powerpc/net/bpf_jit.h +2 -0
index 5d735bc5e6bd..fb548ae5d143 100644
@@ -179,6 +179,8 @@ struct codegen_context {
 	u64 arena_vm_start;
 	u64 user_vm_start;
 	bool is_subprog;
+	bool exception_boundary;
+	bool exception_cb;
 };
 
 #define bpf_to_ppc(r)	(ctx->b2p[r])
arch/powerpc/net/bpf_jit_comp.c +7 -0
index e3088cf089d1..26991940d36e 100644
@@ -207,6 +207,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 	cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux->arena);
 	cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena);
 	cgctx.is_subprog = bpf_is_subprog(fp);
+	cgctx.exception_boundary = fp->aux->exception_boundary;
+	cgctx.exception_cb = fp->aux->exception_cb;
 
 	/* Scouting faux-generate pass 0 */
 	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
@@ -436,6 +438,11 @@ void bpf_jit_free(struct bpf_prog *fp)
 	bpf_prog_unlock_free(fp);
 }
 
+bool bpf_jit_supports_exceptions(void)
+{
+	return IS_ENABLED(CONFIG_PPC64);
+}
+
 bool bpf_jit_supports_subprog_tailcalls(void)
 {
 	return IS_ENABLED(CONFIG_PPC64);
arch/powerpc/net/bpf_jit_comp64.c +36 -17

83 lines not on this page · raw message

Signature · 1 line
2.48.1
maintainer
#24 lore raw

Re: [PATCH v2 5/6] powerpc64/bpf: Support exceptions

Quoted 132 lines · On 14/01/26 5:14 pm, [email protected] wrote:

From: Abhishek Dubey <[email protected]>

The modified prologue/epilogue generation code now enables exception-callback to use the stack frame of the program marked as exception boundary, where callee saved registers are stored.

As per ppc64 ABIv2 documentation[1], r14-r31 are callee saved registers. BPF programs on ppc64 already saves r26-r31 registers. Saving the remaining set of callee saved registers(r14-r25) is handled in the next patch.

[1] https://ftp.rtems.org/pub/rtems/people/sebh/ABI64BitOpenPOWERv1.1_16July2015_pub.pdf

---
  arch/powerpc/net/bpf_jit.h        |  2 ++
  arch/powerpc/net/bpf_jit_comp.c   |  7 ++++
  arch/powerpc/net/bpf_jit_comp64.c | 53 +++++++++++++++++++++----------
  3 files changed, 45 insertions(+), 17 deletions(-)
arch/powerpc/net/bpf_jit.h +2 -0
index 5d735bc5e6bd..fb548ae5d143 100644
@@ -179,6 +179,8 @@ struct codegen_context {
  	u64 arena_vm_start;
  	u64 user_vm_start;
  	bool is_subprog;
+	bool exception_boundary;
+	bool exception_cb;
  };
 
  #define bpf_to_ppc(r)	(ctx->b2p[r])
arch/powerpc/net/bpf_jit_comp.c +7 -0
index e3088cf089d1..26991940d36e 100644
@@ -207,6 +207,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
  	cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux->arena);
  	cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena);
  	cgctx.is_subprog = bpf_is_subprog(fp);
+	cgctx.exception_boundary = fp->aux->exception_boundary;
+	cgctx.exception_cb = fp->aux->exception_cb;
 
  	/* Scouting faux-generate pass 0 */
  	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
@@ -436,6 +438,11 @@ void bpf_jit_free(struct bpf_prog *fp)
  	bpf_prog_unlock_free(fp);
  }
 
+bool bpf_jit_supports_exceptions(void)
+{
+	return IS_ENABLED(CONFIG_PPC64);
+}
+
  bool bpf_jit_supports_subprog_tailcalls(void)
  {
  	return IS_ENABLED(CONFIG_PPC64);
arch/powerpc/net/bpf_jit_comp64.c +27 -16
index ec58395f74f7..a6083dd9786c 100644
@@ -89,7 +89,9 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
  	 * - the bpf program uses its stack area
  	 * The latter condition is deduced from the usage of BPF_REG_FP
  	 */
-	return ctx->seen & SEEN_FUNC || bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP));
+	return ctx->seen & SEEN_FUNC ||
+	       bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)) ||
+	       ctx->exception_cb;
  }
 
  /*
@@ -190,23 +192,32 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
  		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
  	}
 
-	/*
-	 * Back up non-volatile regs -- BPF registers 6-10
-	 * If we haven't created our own stack frame, we save these
-	 * in the protected zone below the previous stack frame
-	 */
-	for (i = BPF_REG_6; i <= BPF_REG_10; i++)
-		if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
-			EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
+	if (!ctx->exception_cb) {
+		/*
+		 * Back up non-volatile regs -- BPF registers 6-10
+		 * If we haven't created our own stack frame, we save these
+		 * in the protected zone below the previous stack frame
+		 */
+		for (i = BPF_REG_6; i <= BPF_REG_10; i++)
+			if (ctx->exception_boundary || bpf_is_seen_register(ctx, bpf_to_ppc(i)))
+				EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1,
+					bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
 
-	if (ctx->arena_vm_start)
-		EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1,
+		if (ctx->exception_boundary || ctx->arena_vm_start)
+			EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1,
  				 bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
 
-	/* Setup frame pointer to point to the bpf stack area */
-	if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
-		EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
+		/* Setup frame pointer to point to the bpf stack area */
+		if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
+			EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
  				STACK_FRAME_MIN_SIZE + ctx->stack_size));
+	} else {
+		/*
+		 * Exception callback receives Frame Pointer of main
+		 * program as third arg
+		 */
+		EMIT(PPC_RAW_MR(_R1, _R5));
+	}
 
  	if (ctx->arena_vm_start)
  		PPC_LI64(bpf_to_ppc(ARENA_VM_START), ctx->arena_vm_start);
@@ -218,17 +229,25 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx
 
  	/* Restore NVRs */
  	for (i = BPF_REG_6; i <= BPF_REG_10; i++)
-		if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
+		if (ctx->exception_cb || bpf_is_seen_register(ctx, bpf_to_ppc(i)))
  			EMIT(PPC_RAW_LD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
 
-	if (ctx->arena_vm_start)
+	if (ctx->exception_cb || ctx->arena_vm_start)
  		EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1,
  				bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));

arch/powerpc/net/bpf_jit_comp64.c static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx · line 247

Quoted 10 lines
arch/powerpc/net/bpf_jit_comp64.c +8 -0
 
 
+	if (ctx->exception_cb) {
+		/*
+		 * LR value from boundary-frame is received as second parameter
+		 * in exception callback.
+		 */
+		EMIT(PPC_RAW_MTLR(_R4));
+	}
+

No. Both second and third parameter of exception_cb() are stack pointer (or frame pointer, if you prefer that) and not LR.
The above hunk is wrong. It still worked because of mtlr instruction below that restored LR from PPC_LR_STKOFF offset on the stack. Please drop the above hunk.

Quoted 8 lines
arch/powerpc/net/bpf_jit_comp64.c +1 -1
  	/* Tear down our stack frame */
  	if (bpf_has_stack_frame(ctx)) {
  		EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size));
-		if (ctx->seen & SEEN_FUNC) {
+		if (ctx->seen & SEEN_FUNC || ctx->exception_cb) {
  			EMIT(PPC_RAW_LD(_R0, _R1, PPC_LR_STKOFF));
  			EMIT(PPC_RAW_MTLR(_R0));
  		}

- Hari

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

maintainer
#27 lore raw

Re: [PATCH v2 5/6] powerpc64/bpf: Support exceptions

Quoted 149 lines · adubey
Quoted 148 lines · adubey
Quoted 147 lines · Hari Bathini
Quoted 145 lines · On 14/01/26 5:14 pm, [email protected] wrote:

From: Abhishek Dubey <[email protected]>

The modified prologue/epilogue generation code now enables exception-callback to use the stack frame of the program marked as exception boundary, where callee saved registers are stored.

As per ppc64 ABIv2 documentation[1], r14-r31 are callee saved registers. BPF programs on ppc64 already saves r26-r31 registers. Saving the remaining set of callee saved registers(r14-r25) is handled in the next patch.

[1] https://ftp.rtems.org/pub/rtems/people/sebh/ ABI64BitOpenPOWERv1.1_16July2015_pub.pdf

---
  arch/powerpc/net/bpf_jit.h        |  2 ++ arch/powerpc/net/bpf_jit_comp.c   |  7 ++++ arch/powerpc/net/bpf_jit_comp64.c | 53 ++++++++++++++++++++
+----------

  3 files changed, 45 insertions(+), 17 deletions(-)
arch/powerpc/net/bpf_jit.h +2 -0
index 5d735bc5e6bd..fb548ae5d143 100644
@@ -179,6 +179,8 @@ struct codegen_context {
       u64 arena_vm_start;
       u64 user_vm_start;
       bool is_subprog;
+    bool exception_boundary;
+    bool exception_cb;
   };
     #define bpf_to_ppc(r)    (ctx->b2p[r])
arch/powerpc/net/bpf_jit_comp.c +0 -0
index e3088cf089d1..26991940d36e 100644
@@ -207,6 +207,8 @@ struct bpf_prog *bpf_int_jit_compile(struct 
 bpf_prog *fp)

      cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux-

Quoted 1 line

arena);

      cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux-

Quoted 1 line

arena);

      cgctx.is_subprog = bpf_is_subprog(fp);
+    cgctx.exception_boundary = fp->aux->exception_boundary; +    cgctx.exception_cb = fp->aux->exception_cb;
        /* Scouting faux-generate pass 0 */
      if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {

arch/powerpc/net/bpf_jit_comp.c +4 -0
@@ -436,6 +438,11 @@ void bpf_jit_free(struct bpf_prog *fp)
       bpf_prog_unlock_free(fp);
   }
   +bool bpf_jit_supports_exceptions(void)
+{
+    return IS_ENABLED(CONFIG_PPC64);
+}
+
   bool bpf_jit_supports_subprog_tailcalls(void)
   {
       return IS_ENABLED(CONFIG_PPC64);
arch/powerpc/net/bpf_jit_comp64.c +3 -1
index ec58395f74f7..a6083dd9786c 100644
@@ -89,7 +89,9 @@ static inline bool bpf_has_stack_frame(struct 
 codegen_context *ctx)
        * - the bpf program uses its stack area
        * The latter condition is deduced from the usage of BPF_REG_FP
        */
-    return ctx->seen & SEEN_FUNC || bpf_is_seen_register(ctx, 
 bpf_to_ppc(BPF_REG_FP));
+    return ctx->seen & SEEN_FUNC ||
+           bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)) ||
+           ctx->exception_cb;
   }

    /*

arch/powerpc/net/bpf_jit_comp64.c +0 -0
@@ -190,23 +192,32 @@ void bpf_jit_build_prologue(u32 *image, struct 
 codegen_context *ctx)

          EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx-

Quoted 1 line

stack_size)));

      }
  -    /*
-     * Back up non-volatile regs -- BPF registers 6-10
-     * If we haven't created our own stack frame, we save these -     * in the protected zone below the previous stack frame
-     */
-    for (i = BPF_REG_6; i <= BPF_REG_10; i++)
-        if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
-            EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
+    if (!ctx->exception_cb) {
+        /*
+         * Back up non-volatile regs -- BPF registers 6-10
+         * If we haven't created our own stack frame, we save these +         * in the protected zone below the previous stack frame +         */
+        for (i = BPF_REG_6; i <= BPF_REG_10; i++)
+            if (ctx->exception_boundary || bpf_is_seen_register(ctx, bpf_to_ppc(i)))
+                EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1,
+                    bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i)))); -    if (ctx->arena_vm_start)
-        EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1,
+        if (ctx->exception_boundary || ctx->arena_vm_start)
+            EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START)))); -    /* Setup frame pointer to point to the bpf stack area */
-    if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
-        EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
+        /* Setup frame pointer to point to the bpf stack area */ +        if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
+            EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1, STACK_FRAME_MIN_SIZE + ctx->stack_size));
+    } else {
+        /*
+         * Exception callback receives Frame Pointer of main
+         * program as third arg
+         */
+        EMIT(PPC_RAW_MR(_R1, _R5));
+    }
        if (ctx->arena_vm_start) PPC_LI64(bpf_to_ppc(ARENA_VM_START), ctx->arena_vm_start);

arch/powerpc/net/bpf_jit_comp64.c +5 -1
@@ -218,17 +229,25 @@ static void bpf_jit_emit_common_epilogue(u32 
 *image, struct codegen_context *ctx
         /* Restore NVRs */
       for (i = BPF_REG_6; i <= BPF_REG_10; i++)
-        if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
+        if (ctx->exception_cb || bpf_is_seen_register(ctx, 
 bpf_to_ppc(i)))
               EMIT(PPC_RAW_LD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
   -    if (ctx->arena_vm_start)
+    if (ctx->exception_cb || ctx->arena_vm_start)
           EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1,
                   bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
 
 
+    if (ctx->exception_cb) {
+        /*
+         * LR value from boundary-frame is received as second 

parameter
+         * in exception callback.
+         */
+        EMIT(PPC_RAW_MTLR(_R4));
+    }
+

Quoted 7 lines
Quoted 6 lines
Quoted 5 lines

No. Both second and third parameter of exception_cb() are stack pointer (or frame pointer, if you prefer that) and not LR.
The above hunk is wrong. It still worked because of mtlr instruction below that restored LR from PPC_LR_STKOFF offset on the stack. Please drop the above hunk.

Okay, but from patch 3/6:

My mistake, its patch 4/6.

Please note the consume_fn() is not exception_cb(). It is bpf_stack_walker() that helps with the stack walk to be precise. You can find the exception_cb() call in bpf_throw() kfunc as:

  ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
Quoted 11 lines
Quoted 11 lines

ip = frame[STACK_FRAME_LR_SAVE];
...
if (ip && !consume_fn(cookie, ip, fp, fp))
we send cookie as first, LR as second parameter and fp as third & fourth both. Default callback expects 4 parameter, so we copy fp in both.

Quoted 4 lines
Quoted 4 lines

      /* Tear down our stack frame */
      if (bpf_has_stack_frame(ctx)) { EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx-

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 1 line

stack_size));

-        if (ctx->seen & SEEN_FUNC) {
+        if (ctx->seen & SEEN_FUNC || ctx->exception_cb) { EMIT(PPC_RAW_LD(_R0, _R1, PPC_LR_STKOFF)); EMIT(PPC_RAW_MTLR(_R0));
          }

- Hari

#28 lore raw

PATCH v2 6/6 powerpc64/bpf: Additional NVR handling for bpf_throw

In reply to [email protected]

From: Abhishek Dubey <[email protected]>

The bpf_throw() function never returns, if it has clobbered any callee-saved register, those will remain clobbered. The prologue must take care of saving all callee-saved registers in the frame of exception boundary program. Later these additional non volatile registers R14-R25 along with other NVRs are restored back in the epilogue of exception callback.

To achieve above objective the frame size is determined dynamically to accommodate additional non volatile registers in exception boundary's frame. For non-exception boundary program, the frame size remains optimal. The additional instructions to save & restore r14-r25 registers are emitted only during exception boundary and exception callback respectively.

---
 arch/powerpc/net/bpf_jit_comp64.c | 70 +++++++++++++++++++++++++++----
 1 file changed, 63 insertions(+), 7 deletions(-)
arch/powerpc/net/bpf_jit_comp64.c +63 -7
index a6083dd9786c..941e0818c9ec 100644
@@ -32,21 +32,37 @@
  *
  *		[	prev sp		] <-------------
  *		[    tail_call_info	] 8		|
- *		[   nv gpr save area	] 6*8		|
+ *		[   nv gpr save area	] 6*8 + (12*8)	|
  *		[    local_tmp_var	] 24		|
  * fp (r31) -->	[   ebpf stack space	] upto 512	|
  *		[     frame header	] 32/112	|
  * sp (r1) --->	[    stack pointer	] --------------
+ *
+ * Additional (12*8) in 'nv gpr save area' only in case of
+ * exception boundary.
  */
 
 /* for bpf JIT code internal usage */
 #define BPF_PPC_STACK_LOCALS	24
+/*
+ * for additional non volatile registers(r14-r25) to be saved
+ * at exception boundary
+ */
+#define BPF_PPC_EXC_STACK_SAVE (12*8)
+
 /* stack frame excluding BPF stack, ensure this is quadword aligned */
 #define BPF_PPC_STACKFRAME	(STACK_FRAME_MIN_SIZE + \
 				 BPF_PPC_STACK_LOCALS + \
 				 BPF_PPC_STACK_SAVE   + \
 				 BPF_PPC_TAILCALL)
 
+/*
+ * same as BPF_PPC_STACKFRAME with save area for additional
+ * non volatile registers saved at exception boundary.
+ * This is quad-word aligned.
+ */
+#define BPF_PPC_EXC_STACKFRAME (BPF_PPC_STACKFRAME + BPF_PPC_EXC_STACK_SAVE)
+
 /* BPF register usage */
 #define TMP_REG_1	(MAX_BPF_JIT_REG + 0)
 #define TMP_REG_2	(MAX_BPF_JIT_REG + 1)
@@ -103,9 +119,12 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
  *		[	  ...       	] 		|
  * sp (r1) --->	[    stack pointer	] --------------
  *		[    tail_call_info	] 8
- *		[   nv gpr save area	] 6*8
+ *		[   nv gpr save area	] 6*8 + (12*8)
  *		[    local_tmp_var	] 24
  *		[   unused red zone	] 224
+ *
+ * Additional (12*8) in 'nv gpr save area' only in case of
+ * exception boundary.
  */
 static int bpf_jit_stack_local(struct codegen_context *ctx)
 {
@@ -114,7 +133,11 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)
 		return STACK_FRAME_MIN_SIZE + ctx->stack_size;
 	} else {
 		/* Stack layout 2 */
-		return -(BPF_PPC_TAILCALL + BPF_PPC_STACK_SAVE + BPF_PPC_STACK_LOCALS);
+		return -(BPF_PPC_TAILCALL
+			+ BPF_PPC_STACK_SAVE
+			+ (ctx->exception_boundary || ctx->exception_cb ?
+							BPF_PPC_EXC_STACK_SAVE:0)
+			+ BPF_PPC_STACK_LOCALS);
 	}
 }
 
@@ -125,9 +148,19 @@ int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)
 
 static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
 {
-	if (reg >= BPF_PPC_NVR_MIN && reg < 32)
+	int min_valid_nvreg = BPF_PPC_NVR_MIN;
+	/* Default frame size for all cases except exception boundary */
+	int frame_nvr_size = BPF_PPC_STACKFRAME;
+
+	/* Consider all nv regs for handling exceptions */
+	if (ctx->exception_boundary || ctx->exception_cb) {
+		min_valid_nvreg = _R14;
+		frame_nvr_size = BPF_PPC_EXC_STACKFRAME;
+	}
+
+	if (reg >= min_valid_nvreg && reg < 32)
 		return (bpf_has_stack_frame(ctx) ?
-			(BPF_PPC_STACKFRAME + ctx->stack_size) : 0)
+			(frame_nvr_size + ctx->stack_size) : 0)
 				- (8 * (32 - reg)) - BPF_PPC_TAILCALL;
 
 	pr_err("BPF JIT is asking about unknown registers");
@@ -189,7 +222,20 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 			EMIT(PPC_RAW_STD(_R0, _R1, PPC_LR_STKOFF));
 		}
 
-		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
+		int stack_expand = ctx->exception_boundary || ctx->exception_cb ?
+					BPF_PPC_EXC_STACKFRAME : BPF_PPC_STACKFRAME;
+		EMIT(PPC_RAW_STDU(_R1, _R1, -(stack_expand + ctx->stack_size)));
+	}
+
+	/*
+	 * Program acting as exception boundary pushes R14..R25 in addition to
+	 * BPF callee-saved non volatile registers. Exception callback uses
+	 * the boundary program's stack frame, recover additionally saved
+	 * registers in epilogue of exception callback.
+	 */
+	if (ctx->exception_boundary) {
+		for (i = _R14; i <= _R25; i++)
+			EMIT(PPC_RAW_STD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
 	}
 
 	if (!ctx->exception_cb) {
@@ -237,6 +283,13 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx
 				bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
 
 	if (ctx->exception_cb) {
+		/*
+		 * Recover additionally saved non volatile registers from stack
+		 * frame of exception boundary program.
+		 */
+		for (i = _R14; i <= _R25; i++)
+			EMIT(PPC_RAW_LD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
+
 		/*
 		 * LR value from boundary-frame is received as second parameter
 		 * in exception callback.
@@ -246,7 +299,10 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx
 
 	/* Tear down our stack frame */
 	if (bpf_has_stack_frame(ctx)) {
-		EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size));
+		int stack_shrink = ctx->exception_cb || ctx->exception_boundary ?
+					BPF_PPC_EXC_STACKFRAME : BPF_PPC_STACKFRAME;
+		EMIT(PPC_RAW_ADDI(_R1, _R1, stack_shrink + ctx->stack_size));
+
 		if (ctx->seen & SEEN_FUNC || ctx->exception_cb) {
 			EMIT(PPC_RAW_LD(_R0, _R1, PPC_LR_STKOFF));
 			EMIT(PPC_RAW_MTLR(_R0));
Signature · 1 line
2.48.1
#29 lore raw

Re: [PATCH v2 6/6] powerpc64/bpf: Additional NVR handling for bpf_throw

Does this need to account for BPF_PPC_EXC_STACK_SAVE when exception_boundary or exception_cb is true?
Quoted 4 lines
arch/powerpc/net/bpf_jit_comp64.c +0 -0
index a6083dd9786c..941e0818c9ec 100644

[ ... ]

Quoted 15 lines
arch/powerpc/net/bpf_jit_comp64.c +4 -1
@@ -103,9 +119,12 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
 *		[	  ...       	] 		|
 * sp (r1) --->	[    stack pointer	] --------------
 *		[    tail_call_info	] 8
- *		[   nv gpr save area	] 6*8
+ *		[   nv gpr save area	] 6*8 + (12*8)
 *		[    local_tmp_var	] 24
 *		[   unused red zone	] 224
+ *
+ * Additional (12*8) in 'nv gpr save area' only in case of
+ * exception boundary.
 */
 static int bpf_jit_stack_local(struct codegen_context *ctx)
 {

arch/powerpc/net/bpf_jit_comp64.c static int bpf_jit_stack_local(struct codegen_context *ctx) · line 133

Quoted 1 line
arch/powerpc/net/bpf_jit_comp64.c +0 -0
@@ -114,7 +133,11 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)
 		return STACK_FRAME_MIN_SIZE + ctx->stack_size;

Does this need to account for BPF_PPC_EXC_STACK_SAVE when exception_boundary or exception_cb is true?

In the else branch below, the exception case is handled, but the bpf_has_stack_frame() case returns the same value regardless of exception_boundary. However, bpf_jit_build_prologue() creates a larger frame using BPF_PPC_EXC_STACKFRAME when exception_boundary is true.

This inconsistency appears to cause bpf_jit_stack_tailcallinfo_offset() to return an incorrect offset when both bpf_has_stack_frame() returns true AND exception_boundary is true. The offset would be off by 96 bytes (BPF_PPC_EXC_STACK_SAVE).

For example, if a program uses bpf_throw (setting exception_boundary=true) and also calls helper functions (setting SEEN_FUNC, making bpf_has_stack_frame() return true), and uses tail calls, the tail call handling in bpf_jit_emit_tail_call() would access the wrong stack location.

Quoted 10 lines
arch/powerpc/net/bpf_jit_comp64.c +5 -1
 	} else {
 		/* Stack layout 2 */
-		return -(BPF_PPC_TAILCALL + BPF_PPC_STACK_SAVE + BPF_PPC_STACK_LOCALS);
+		return -(BPF_PPC_TAILCALL
+			+ BPF_PPC_STACK_SAVE
+			+ (ctx->exception_boundary || ctx->exception_cb ?
+							BPF_PPC_EXC_STACK_SAVE:0)
+			+ BPF_PPC_STACK_LOCALS);
 	}
 }

[ ... ]

arch/powerpc/net/bpf_jit_comp64.c void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) · line 227

Quoted 8 lines
arch/powerpc/net/bpf_jit_comp64.c +3 -1
@@ -189,7 +222,20 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 			EMIT(PPC_RAW_STD(_R0, _R1, PPC_LR_STKOFF));
 		}
 
-		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
+		int stack_expand = ctx->exception_boundary || ctx->exception_cb ?
+					BPF_PPC_EXC_STACKFRAME : BPF_PPC_STACKFRAME;
+		EMIT(PPC_RAW_STDU(_R1, _R1, -(stack_expand + ctx->stack_size)));

The stack frame expansion here uses BPF_PPC_EXC_STACKFRAME for exception cases, but bpf_jit_stack_local() in the bpf_has_stack_frame() branch does not account for this larger frame size.

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/20993216190

maintainer
#30 lore raw

Re: [PATCH v2 6/6] powerpc64/bpf: Additional NVR handling for bpf_throw

In reply to [email protected]
Quoted 125 lines · On 14/01/26 5:14 pm, [email protected] wrote:

From: Abhishek Dubey <[email protected]>

The bpf_throw() function never returns, if it has clobbered any callee-saved register, those will remain clobbered. The prologue must take care of saving all callee-saved registers in the frame of exception boundary program. Later these additional non volatile registers R14-R25 along with other NVRs are restored back in the epilogue of exception callback.

To achieve above objective the frame size is determined dynamically to accommodate additional non volatile registers in exception boundary's frame. For non-exception boundary program, the frame size remains optimal. The additional instructions to save & restore r14-r25 registers are emitted only during exception boundary and exception callback respectively.

---
  arch/powerpc/net/bpf_jit_comp64.c | 70 +++++++++++++++++++++++++++----
  1 file changed, 63 insertions(+), 7 deletions(-)
arch/powerpc/net/bpf_jit_comp64.c +41 -6
index a6083dd9786c..941e0818c9ec 100644
@@ -32,21 +32,37 @@
   *
   *		[	prev sp		] <-------------
   *		[    tail_call_info	] 8		|
- *		[   nv gpr save area	] 6*8		|
+ *		[   nv gpr save area	] 6*8 + (12*8)	|
   *		[    local_tmp_var	] 24		|
   * fp (r31) -->	[   ebpf stack space	] upto 512	|
   *		[     frame header	] 32/112	|
   * sp (r1) --->	[    stack pointer	] --------------
+ *
+ * Additional (12*8) in 'nv gpr save area' only in case of
+ * exception boundary.
   */
 
  /* for bpf JIT code internal usage */
  #define BPF_PPC_STACK_LOCALS	24
+/*
+ * for additional non volatile registers(r14-r25) to be saved
+ * at exception boundary
+ */
+#define BPF_PPC_EXC_STACK_SAVE (12*8)
+
  /* stack frame excluding BPF stack, ensure this is quadword aligned */
  #define BPF_PPC_STACKFRAME	(STACK_FRAME_MIN_SIZE + \
  				 BPF_PPC_STACK_LOCALS + \
  				 BPF_PPC_STACK_SAVE   + \
  				 BPF_PPC_TAILCALL)
 
+/*
+ * same as BPF_PPC_STACKFRAME with save area for additional
+ * non volatile registers saved at exception boundary.
+ * This is quad-word aligned.
+ */
+#define BPF_PPC_EXC_STACKFRAME (BPF_PPC_STACKFRAME + BPF_PPC_EXC_STACK_SAVE)
+
  /* BPF register usage */
  #define TMP_REG_1	(MAX_BPF_JIT_REG + 0)
  #define TMP_REG_2	(MAX_BPF_JIT_REG + 1)
@@ -103,9 +119,12 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
   *		[	  ...       	] 		|
   * sp (r1) --->	[    stack pointer	] --------------
   *		[    tail_call_info	] 8
- *		[   nv gpr save area	] 6*8
+ *		[   nv gpr save area	] 6*8 + (12*8)
   *		[    local_tmp_var	] 24
   *		[   unused red zone	] 224
+ *
+ * Additional (12*8) in 'nv gpr save area' only in case of
+ * exception boundary.
   */
  static int bpf_jit_stack_local(struct codegen_context *ctx)
  {
@@ -114,7 +133,11 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)
  		return STACK_FRAME_MIN_SIZE + ctx->stack_size;
  	} else {
  		/* Stack layout 2 */
-		return -(BPF_PPC_TAILCALL + BPF_PPC_STACK_SAVE + BPF_PPC_STACK_LOCALS);
+		return -(BPF_PPC_TAILCALL
+			+ BPF_PPC_STACK_SAVE
+			+ (ctx->exception_boundary || ctx->exception_cb ?
+							BPF_PPC_EXC_STACK_SAVE:0)
+			+ BPF_PPC_STACK_LOCALS);
  	}
  }
 
@@ -125,9 +148,19 @@ int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)
 
  static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
  {
-	if (reg >= BPF_PPC_NVR_MIN && reg < 32)
+	int min_valid_nvreg = BPF_PPC_NVR_MIN;
+	/* Default frame size for all cases except exception boundary */
+	int frame_nvr_size = BPF_PPC_STACKFRAME;
+
+	/* Consider all nv regs for handling exceptions */
+	if (ctx->exception_boundary || ctx->exception_cb) {
+		min_valid_nvreg = _R14;
+		frame_nvr_size = BPF_PPC_EXC_STACKFRAME;
+	}
+
+	if (reg >= min_valid_nvreg && reg < 32)
  		return (bpf_has_stack_frame(ctx) ?
-			(BPF_PPC_STACKFRAME + ctx->stack_size) : 0)
+			(frame_nvr_size + ctx->stack_size) : 0)
  				- (8 * (32 - reg)) - BPF_PPC_TAILCALL;
 
  	pr_err("BPF JIT is asking about unknown registers");
@@ -189,7 +222,20 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
  			EMIT(PPC_RAW_STD(_R0, _R1, PPC_LR_STKOFF));
  		}
 
 
-		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
+		int stack_expand = ctx->exception_boundary || ctx->exception_cb ?
+					BPF_PPC_EXC_STACKFRAME : BPF_PPC_STACKFRAME;
+		EMIT(PPC_RAW_STDU(_R1, _R1, -(stack_expand + ctx->stack_size)));

[...]

arch/powerpc/net/bpf_jit_comp64.c static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx · line 305

Quoted 5 lines
arch/powerpc/net/bpf_jit_comp64.c +4 -1
-		EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size));
+		int stack_shrink = ctx->exception_cb || ctx->exception_boundary ?
+					BPF_PPC_EXC_STACKFRAME : BPF_PPC_STACKFRAME;
+		EMIT(PPC_RAW_ADDI(_R1, _R1, stack_shrink + ctx->stack_size));
+

An inline helper bpf_jit_stack_size() defined to return the stack size in both prologue and epilogue while setting up and tearing down the stack should be more elegant.

Also, IIUC, the JIT code to handle tailcall info is irrelevant for all subprogs of a BPF program with seen_exception. JIT code in the prologue for tailcall count handling can be skipped for exception_cb at least?

- Hari

#31 lore raw
In reply to [email protected]

Le 14/01/2026 à 12:44, [email protected] a écrit :

Quoted 7 lines

From: Abhishek Dubey <[email protected]>

This patch series enables support for two BPF JIT features on powerpc64. The first three patches target support for tail calls with subprogram combinations. The first patch supports realignment of tail_call_cnt offset in stack frame. Implementation details are provided in the commit messages.

Did you consider doing it also for powerpc32 ?

Christophe