powerpc64/bpf: Support tailcalls with subprogs & BPF exceptions
Subsystems (4)
v2 cover powerpc64/bpf: Support tailcalls with subprogs & BPF exceptions
From: Abhishek Dubey <[email protected]>
This patch series enables support for two BPF JIT features on powerpc64. The first three patches target support for tail calls with subprogram combinations. The first patch supports realignment of tail_call_cnt offset in stack frame. Implementation details are provided in the commit messages.
The last three patches add support for BPF exceptions. An architecture-specific stack walker is implemented to assist with stack walk during exceptions.
All selftests related to tailcalls and exceptions are passing:
# ./test_progs -t tailcalls #442/1 tailcalls/tailcall_1:OK #442/2 tailcalls/tailcall_2:OK #442/3 tailcalls/tailcall_3:OK #442/4 tailcalls/tailcall_4:OK #442/5 tailcalls/tailcall_5:OK #442/6 tailcalls/tailcall_6:OK #442/7 tailcalls/tailcall_bpf2bpf_1:OK #442/8 tailcalls/tailcall_bpf2bpf_2:OK #442/9 tailcalls/tailcall_bpf2bpf_3:OK #442/10 tailcalls/tailcall_bpf2bpf_4:OK #442/11 tailcalls/tailcall_bpf2bpf_5:OK #442/12 tailcalls/tailcall_bpf2bpf_6:OK #442/13 tailcalls/tailcall_bpf2bpf_fentry:OK #442/14 tailcalls/tailcall_bpf2bpf_fexit:OK #442/15 tailcalls/tailcall_bpf2bpf_fentry_fexit:OK #442/16 tailcalls/tailcall_bpf2bpf_fentry_entry:OK #442/17 tailcalls/tailcall_poke:OK #442/18 tailcalls/tailcall_bpf2bpf_hierarchy_1:OK #442/19 tailcalls/tailcall_bpf2bpf_hierarchy_fentry:OK #442/20 tailcalls/tailcall_bpf2bpf_hierarchy_fexit:OK #442/21 tailcalls/tailcall_bpf2bpf_hierarchy_fentry_fexit:OK #442/22 tailcalls/tailcall_bpf2bpf_hierarchy_fentry_entry:OK #442/23 tailcalls/tailcall_bpf2bpf_hierarchy_2:OK #442/24 tailcalls/tailcall_bpf2bpf_hierarchy_3:OK #442/25 tailcalls/tailcall_freplace:OK #442/26 tailcalls/tailcall_bpf2bpf_freplace:OK #442/27 tailcalls/tailcall_failure:OK #442/28 tailcalls/reject_tail_call_spin_lock:OK #442/29 tailcalls/reject_tail_call_rcu_lock:OK #442/30 tailcalls/reject_tail_call_preempt_lock:OK #442/31 tailcalls/reject_tail_call_ref:OK #442 tailcalls:OK Summary: 1/31 PASSED, 0 SKIPPED, 0 FAILED
# ./test_progs -t exceptions #105/1 exceptions/exception_throw_always_1:OK #105/2 exceptions/exception_throw_always_2:OK #105/3 exceptions/exception_throw_unwind_1:OK #105/4 exceptions/exception_throw_unwind_2:OK #105/5 exceptions/exception_throw_default:OK #105/6 exceptions/exception_throw_default_value:OK #105/7 exceptions/exception_tail_call:OK #105/8 exceptions/exception_ext:OK #105/9 exceptions/exception_ext_mod_cb_runtime:OK #105/10 exceptions/exception_throw_subprog:OK #105/11 exceptions/exception_assert_nz_gfunc:OK #105/12 exceptions/exception_assert_zero_gfunc:OK #105/13 exceptions/exception_assert_neg_gfunc:OK #105/14 exceptions/exception_assert_pos_gfunc:OK #105/15 exceptions/exception_assert_negeq_gfunc:OK #105/16 exceptions/exception_assert_poseq_gfunc:OK #105/17 exceptions/exception_assert_nz_gfunc_with:OK #105/18 exceptions/exception_assert_zero_gfunc_with:OK #105/19 exceptions/exception_assert_neg_gfunc_with:OK #105/20 exceptions/exception_assert_pos_gfunc_with:OK #105/21 exceptions/exception_assert_negeq_gfunc_with:OK #105/22 exceptions/exception_assert_poseq_gfunc_with:OK #105/23 exceptions/exception_bad_assert_nz_gfunc:OK #105/24 exceptions/exception_bad_assert_zero_gfunc:OK #105/25 exceptions/exception_bad_assert_neg_gfunc:OK #105/26 exceptions/exception_bad_assert_pos_gfunc:OK #105/27 exceptions/exception_bad_assert_negeq_gfunc:OK #105/28 exceptions/exception_bad_assert_poseq_gfunc:OK #105/29 exceptions/exception_bad_assert_nz_gfunc_with:OK #105/30 exceptions/exception_bad_assert_zero_gfunc_with:OK #105/31 exceptions/exception_bad_assert_neg_gfunc_with:OK #105/32 exceptions/exception_bad_assert_pos_gfunc_with:OK #105/33 exceptions/exception_bad_assert_negeq_gfunc_with:OK #105/34 exceptions/exception_bad_assert_poseq_gfunc_with:OK #105/35 exceptions/exception_assert_range:OK #105/36 exceptions/exception_assert_range_with:OK #105/37 exceptions/exception_bad_assert_range:OK #105/38 exceptions/exception_bad_assert_range_with:OK #105/39 exceptions/non-throwing fentry -> exception_cb:OK #105/40 exceptions/throwing fentry -> exception_cb:OK #105/41 exceptions/non-throwing fexit -> exception_cb:OK #105/42 exceptions/throwing fexit -> exception_cb:OK #105/43 exceptions/throwing extension (with custom cb) -> exception_cb:OK #105/44 exceptions/throwing extension -> global func in exception_cb:OK #105/45 exceptions/exception_ext_mod_cb_runtime:OK #105/46 exceptions/throwing extension (with custom cb) -> global func in exception_cb:OK #105/47 exceptions/exception_ext:OK #105/48 exceptions/non-throwing fentry -> non-throwing subprog:OK #105/49 exceptions/throwing fentry -> non-throwing subprog:OK #105/50 exceptions/non-throwing fentry -> throwing subprog:OK #105/51 exceptions/throwing fentry -> throwing subprog:OK #105/52 exceptions/non-throwing fexit -> non-throwing subprog:OK #105/53 exceptions/throwing fexit -> non-throwing subprog:OK #105/54 exceptions/non-throwing fexit -> throwing subprog:OK #105/55 exceptions/throwing fexit -> throwing subprog:OK #105/56 exceptions/non-throwing fmod_ret -> non-throwing subprog:OK #105/57 exceptions/non-throwing fmod_ret -> non-throwing global subprog:OK #105/58 exceptions/non-throwing extension -> non-throwing subprog:OK #105/59 exceptions/non-throwing extension -> throwing subprog:OK #105/60 exceptions/non-throwing extension -> non-throwing subprog:OK #105/61 exceptions/non-throwing extension -> throwing global subprog:OK #105/62 exceptions/throwing extension -> throwing global subprog:OK #105/63 exceptions/throwing extension -> non-throwing global subprog:OK #105/64 exceptions/non-throwing extension -> main subprog:OK #105/65 exceptions/throwing extension -> main subprog:OK #105/66 exceptions/reject_exception_cb_type_1:OK #105/67 exceptions/reject_exception_cb_type_2:OK #105/68 exceptions/reject_exception_cb_type_3:OK #105/69 exceptions/reject_exception_cb_type_4:OK #105/70 exceptions/reject_async_callback_throw:OK #105/71 exceptions/reject_with_lock:OK #105/72 exceptions/reject_subprog_with_lock:OK #105/73 exceptions/reject_with_rcu_read_lock:OK #105/74 exceptions/reject_subprog_with_rcu_read_lock:OK #105/75 exceptions/reject_with_rbtree_add_throw:OK #105/76 exceptions/reject_with_reference:OK #105/77 exceptions/reject_with_cb_reference:OK #105/78 exceptions/reject_with_cb:OK #105/79 exceptions/reject_with_subprog_reference:OK #105/80 exceptions/reject_throwing_exception_cb:OK #105/81 exceptions/reject_exception_cb_call_global_func:OK #105/82 exceptions/reject_exception_cb_call_static_func:OK #105/83 exceptions/reject_multiple_exception_cb:OK #105/84 exceptions/reject_exception_throw_cb:OK #105/85 exceptions/reject_exception_throw_cb_diff:OK #105/86 exceptions/reject_set_exception_cb_bad_ret1:OK #105/87 exceptions/reject_set_exception_cb_bad_ret2:OK #105/88 exceptions/check_assert_eq_int_min:OK #105/89 exceptions/check_assert_eq_int_max:OK #105/90 exceptions/check_assert_eq_zero:OK #105/91 exceptions/check_assert_eq_llong_min:OK #105/92 exceptions/check_assert_eq_llong_max:OK #105/93 exceptions/check_assert_lt_pos:OK #105/94 exceptions/check_assert_lt_zero:OK #105/95 exceptions/check_assert_lt_neg:OK #105/96 exceptions/check_assert_le_pos:OK #105/97 exceptions/check_assert_le_zero:OK #105/98 exceptions/check_assert_le_neg:OK #105/99 exceptions/check_assert_gt_pos:OK #105/100 exceptions/check_assert_gt_zero:OK #105/101 exceptions/check_assert_gt_neg:OK #105/102 exceptions/check_assert_ge_pos:OK #105/103 exceptions/check_assert_ge_zero:OK #105/104 exceptions/check_assert_ge_neg:OK #105/105 exceptions/check_assert_range_s64:OK #105/106 exceptions/check_assert_range_u64:OK #105/107 exceptions/check_assert_single_range_s64:OK #105/108 exceptions/check_assert_single_range_u64:OK #105/109 exceptions/check_assert_generic:OK #105/110 exceptions/check_assert_with_return:OK #105 exceptions:OK Summary: 1/110 PASSED, 0 SKIPPED, 0 FAILED
[v1]: https://lore.kernel.org/all/[email protected]/
Changes v1->v2:
Move tail_call_cnt to offset 0 in stack frame Remove trampoline NVR remapping-patch3/6[v1]
Abhishek Dubey (6):
powerpc64/bpf: Move tail_call_cnt to bottom of stack frame powerpc64/bpf: Support tailcalls with subprogs powerpc64/bpf: Tailcall handling with trampolines powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT powerpc64/bpf: Support exceptions
powerpc64/bpf: Additional NVR handling for bpf_throw
arch/powerpc/net/bpf_jit.h | 18 ++- arch/powerpc/net/bpf_jit_comp.c | 98 ++++++++---- arch/powerpc/net/bpf_jit_comp64.c | 240 ++++++++++++++++++++++++------ 3 files changed, 280 insertions(+), 76 deletions(-)
Signature · 1 line
2.48.1
PATCH v2 1/6 powerpc64/bpf: Move tail_call_cnt to bottom of stack frame
From: Abhishek Dubey <[email protected]>
In the conventional stack frame, the position of tail_call_cnt is after the NVR save area (BPF_PPC_STACK_SAVE). Whereas, the offset of tail_call_cnt in the trampoline frame is after the stack alignment padding. BPF JIT logic could become complex when dealing with frame-sensitive offset calculation of tail_call_cnt. Having the same offset in both frames is the desired objective.
The trampoline frame does not have a BPF_PPC_STACK_SAVE area. Introducing it leads to under-utilization of extra memory meant only for the offset alignment of tail_call_cnt. Another challenge is the variable alignment padding sitting at the bottom of the trampoline frame, which requires additional handling to compute tail_call_cnt offset.
This patch addresses the above issues by moving tail_call_cnt to the bottom of the stack frame at offset 0 for both types of frames. This saves additional bytes required by BPF_PPC_STACK_SAVE in trampoline frame, and a common offset computation for tail_call_cnt serves both frames.
The changes in this patch are required by the third patch in the series, where the 'reference to tail_call_info' of the main frame is copied into the trampoline frame from the previous frame.
--- arch/powerpc/net/bpf_jit.h | 4 ++++ arch/powerpc/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++----------- 2 files changed, 24 insertions(+), 11 deletions(-)
arch/powerpc/net/bpf_jit.h +4 -0
@@ -72,6 +72,10 @@ } } while (0)#ifdef CONFIG_PPC64 + +/* for tailcall counter */ +#define BPF_PPC_TAILCALL 8 +/* If dummy pass (!image), account for maximum possible instructions */ #define PPC_LI64 (d, i) do { \ if (!image) \
arch/powerpc/net/bpf_jit_comp64.c +20 -11
@@ -20,13 +20,15 @@#include "bpf_jit.h" /* -* Stack layout: +* Stack layout 1: +* Layout when setting up our own stack frame. +* Note: r1 at bottom, component offsets positive wrt r1. * Ensure the top half (upto local_tmp_var) stays consistent * with our redzone usage. * * [ prev sp ] <------------- -* [ nv gpr save area ] 6*8 | * [ tail_call_cnt ] 8 | +* [ nv gpr save area ] 6*8 | * [ local_tmp_var ] 24 | * fp (r31) --> [ ebpf stack space ] upto 512 | * [ frame header ] 32/112 |
@@ -36,10 +38,12 @@/* for gpr non volatile registers BPG_REG_6 to 10 */ #define BPF_PPC_STACK_SAVE (6*8)/* for bpf JIT code internal usage */ -#define BPF_PPC_STACK_LOCALS 32 +#define BPF_PPC_STACK_LOCALS 24/* stack frame excluding BPF stack, ensure this is quadword aligned */ #define BPF_PPC_STACKFRAME (STACK_FRAME_MIN_SIZE + \ - BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE) + BPF_PPC_STACK_LOCALS + \ + BPF_PPC_STACK_SAVE + \ + BPF_PPC_TAILCALL)/* BPF register usage */ #define TMP_REG_1 (MAX_BPF_JIT_REG + 0)
@@ -87,27 +91,32 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx) }/* +* Stack layout 2: * When not setting up our own stackframe, the redzone (288 bytes) usage is: +* Note: r1 from prev frame. Component offset negative wrt r1. * * [ prev sp ] <------------- * [ ... ] | * sp (r1) ---> [ stack pointer ] -------------- -* [ nv gpr save area ] 6*8 * [ tail_call_cnt ] 8 +* [ nv gpr save area ] 6*8 * [ local_tmp_var ] 24 * [ unused red zone ] 224 */ static int bpf_jit_stack_local (struct codegen_context *ctx) { -if (bpf_has_stack_frame (ctx)) +if (bpf_has_stack_frame (ctx)) { +/* Stack layout 1 */ return STACK_FRAME_MIN_SIZE + ctx->stack_size; -else -return -(BPF_PPC_STACK_SAVE +32 ); + }else { +/* Stack layout 2 */ +return -(BPF_PPC_TAILCALL +BPF_PPC_STACK_SAVE +BPF_PPC_STACK_LOCALS ); + } }static int bpf_jit_stack_tailcallcnt (struct codegen_context *ctx) { -return bpf_jit_stack_local (ctx) +24 ; +return bpf_jit_stack_local (ctx) +BPF_PPC_STACK_LOCALS +BPF_PPC_STACK_SAVE ; }static int bpf_jit_stack_offsetof (struct codegen_context *ctx,int reg)
@@ -115,7 +124,7 @@ static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)if (reg >=BPF_PPC_NVR_MIN && reg <32 )return (bpf_has_stack_frame (ctx) ? (BPF_PPC_STACKFRAME + ctx->stack_size) :0 ) - - (8 * (32 - reg)); + - (8 * (32 - reg)) -BPF_PPC_TAILCALL ;pr_err ("BPF JIT is asking about unknown registers" );BUG ();
@@ -145,7 +154,7 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)if (ctx->seen &SEEN_TAILCALL ) {EMIT (PPC_RAW_LI (bpf_to_ppc (TMP_REG_1 ),0 ));/* this goes in the redzone */ -EMIT (PPC_RAW_STD (bpf_to_ppc (TMP_REG_1 ), _R1, -(BPF_PPC_STACK_SAVE +8 ))); +EMIT (PPC_RAW_STD (bpf_to_ppc (TMP_REG_1 ), _R1, -(BPF_PPC_TAILCALL ))); }else {EMIT (PPC_RAW_NOP ());EMIT (PPC_RAW_NOP ());
Signature · 1 line
2.48.1
[email protected]
AI reviewRe: [PATCH v2 1/6] powerpc64/bpf: Move tail_call_cnt to bottom of stack frame
This new constant defines tail_call_cnt offset as 8 bytes from prev sp.
Quoted 4 lines
arch/powerpc/net/bpf_jit.h +0
-0
[ ... ]
arch/powerpc/net/bpf_jit.h line 77
Quoted 2 lines
arch/powerpc/net/bpf_jit.h +2
-0+/* for tailcall counter */ +#define BPF_PPC_TAILCALL 8
This new constant defines tail_call_cnt offset as 8 bytes from prev sp.
Quoted 4 lines
arch/powerpc/net/bpf_jit_comp64.c +0
-0
[ ... ]
Quoted 5 lines
static int
bpf_jit_stack_tailcallcnt(struct codegen_context *ctx){arch/powerpc/net/bpf_jit_comp64.c +1
-1-return bpf_jit_stack_local (ctx) +24 ; +return bpf_jit_stack_local (ctx) +BPF_PPC_STACK_LOCALS +BPF_PPC_STACK_SAVE ;}
The tail_call_cnt offset changes from +24 to +72 (24 + 48) relative to bpf_jit_stack_local().
There appears to be an inconsistency with the trampoline code in bpf_jit_comp.c. The function bpf_trampoline_setup_tail_call_cnt() and bpf_trampoline_restore_tail_call_cnt() use a hardcoded offset:
int tailcallcnt_offset = 7 * 8; /* = 56 */
The comment says "See bpf_jit_stack_tailcallcnt()" but after this patch, bpf_jit_stack_tailcallcnt() returns an offset that corresponds to 8 bytes from prev sp (BPF_PPC_TAILCALL), not 56 bytes. When BPF_TRAMP_F_TAIL_CALL_CTX is set, will the trampoline read/write to the wrong stack location?
Note: This appears to be fixed later in the same series by commit 6889ec3d3e10 ("powerpc64/bpf: Tailcall handling with trampolines") which updates the trampoline code to use BPF_PPC_TAILCALL instead of the hardcoded 7 * 8.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/20993216190
Christophe Leroy (CS GROUP)
maintainerRe: [PATCH v2 1/6] powerpc64/bpf: Move tail_call_cnt to bottom of stack frame
In reply to [email protected]Le 14/01/2026 à 12:44, [email protected] a écrit :
Quoted 38 lines
From: Abhishek Dubey <[email protected]>
In the conventional stack frame, the position of
tail_call_cntis after the NVR save area (BPF_PPC_STACK_SAVE). Whereas, the offset oftail_call_cntin the trampoline frame is after the stack alignment padding. BPF JIT logic could become complex when dealing with frame-sensitive offset calculation oftail_call_cnt. Having the same offset in both frames is the desired objective.The trampoline frame does not have a
BPF_PPC_STACK_SAVEarea. Introducing it leads to under-utilization of extra memory meant only for the offset alignment oftail_call_cnt. Another challenge is the variable alignment padding sitting at the bottom of the trampoline frame, which requires additional handling to computetail_call_cntoffset.This patch addresses the above issues by moving
tail_call_cntto the bottom of the stack frame at offset 0 for both types of frames. This saves additional bytes required byBPF_PPC_STACK_SAVEin trampoline frame, and a common offset computation fortail_call_cntserves both frames.The changes in this patch are required by the third patch in the series, where the 'reference to
tail_call_info' of the main frame is copied into the trampoline frame from the previous frame.Signed-off-by Abhishek Dubey <[email protected]>--- arch/powerpc/net/bpf_jit.h | 4 ++++ arch/powerpc/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++----------- 2 files changed, 24 insertions(+), 11 deletions(-)arch/powerpc/net/bpf_jit.h +0
-0
arch/powerpc/net/bpf_jit.h line 77
Quoted 6 lines
arch/powerpc/net/bpf_jit.h +3
-0@@ -72,6 +72,10 @@ } } while (0)#ifdef CONFIG_PPC64 + +/* for tailcall counter */ +#define BPF_PPC_TAILCALL 8
This needs to be defined outside of CONFIG_PPC64 ifdef because from patch 3 it is used in bpf_jit_comp.c which is also built on powerpc32.
Quoted 4 lines
arch/powerpc/net/bpf_jit.h +1
-0+/* If dummy pass (!image), account for maximum possible instructions */ #define PPC_LI64 (d, i) do { \ if (!image) \
Hari Bathini
maintainerRe: [PATCH v2 1/6] powerpc64/bpf: Move tail_call_cnt to bottom of stack frame
In reply to [email protected]Quoted 93 lines · On 14/01/26 5:14 pm, [email protected] wrote:
From: Abhishek Dubey <[email protected]>
In the conventional stack frame, the position of
tail_call_cntis after the NVR save area (BPF_PPC_STACK_SAVE). Whereas, the offset oftail_call_cntin the trampoline frame is after the stack alignment padding. BPF JIT logic could become complex when dealing with frame-sensitive offset calculation oftail_call_cnt. Having the same offset in both frames is the desired objective.The trampoline frame does not have a
BPF_PPC_STACK_SAVEarea. Introducing it leads to under-utilization of extra memory meant only for the offset alignment oftail_call_cnt. Another challenge is the variable alignment padding sitting at the bottom of the trampoline frame, which requires additional handling to computetail_call_cntoffset.This patch addresses the above issues by moving
tail_call_cntto the bottom of the stack frame at offset 0 for both types of frames. This saves additional bytes required byBPF_PPC_STACK_SAVEin trampoline frame, and a common offset computation fortail_call_cntserves both frames.The changes in this patch are required by the third patch in the series, where the 'reference to
tail_call_info' of the main frame is copied into the trampoline frame from the previous frame.Signed-off-by Abhishek Dubey <[email protected]>--- arch/powerpc/net/bpf_jit.h | 4 ++++ arch/powerpc/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++----------- 2 files changed, 24 insertions(+), 11 deletions(-)arch/powerpc/net/bpf_jit.h +4
-0@@ -72,6 +72,10 @@ } } while (0)#ifdef CONFIG_PPC64 + +/* for tailcall counter */ +#define BPF_PPC_TAILCALL 8 +/* If dummy pass (!image), account for maximum possible instructions */ #define PPC_LI64 (d, i) do { \ if (!image) \arch/powerpc/net/bpf_jit_comp64.c +10
-4@@ -20,13 +20,15 @@#include "bpf_jit.h" /* -* Stack layout: +* Stack layout 1: +* Layout when setting up our own stack frame. +* Note: r1 at bottom, component offsets positive wrt r1. * Ensure the top half (upto local_tmp_var) stays consistent * with our redzone usage. * * [ prev sp ] <------------- -* [ nv gpr save area ] 6*8 | * [ tail_call_cnt ] 8 | +* [ nv gpr save area ] 6*8 | * [ local_tmp_var ] 24 | * fp (r31) --> [ ebpf stack space ] upto 512 | * [ frame header ] 32/112 | @@ -36,10 +38,12 @@/* for gpr non volatile registers BPG_REG_6 to 10 */ #define BPF_PPC_STACK_SAVE (6*8)/* for bpf JIT code internal usage */ -#define BPF_PPC_STACK_LOCALS 32 +#define BPF_PPC_STACK_LOCALS 24/* stack frame excluding BPF stack, ensure this is quadword aligned */ #define BPF_PPC_STACKFRAME (STACK_FRAME_MIN_SIZE + \ - BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE) + BPF_PPC_STACK_LOCALS + \ + BPF_PPC_STACK_SAVE + \ + BPF_PPC_TAILCALL)/* BPF register usage */ #define TMP_REG_1 (MAX_BPF_JIT_REG + 0)@@ -87,27 +91,32 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx) }/* +* Stack layout 2: * When not setting up our own stackframe, the redzone (288 bytes) usage is: +* Note: r1 from prev frame. Component offset negative wrt r1.
arch/powerpc/net/bpf_jit_comp64.c static inline bool bpf_has_stack_frame(struct codegen_context *ctx) · line 106
Quoted 10 lines
arch/powerpc/net/bpf_jit_comp64.c +1
-1* * [ prev sp ] <------------- * [ ... ] | * sp (r1) ---> [ stack pointer ] -------------- -* [ nv gpr save area ] 6*8 * [ tail_call_cnt ] 8 +* [ nv gpr save area ] 6*8 * [ local_tmp_var ] 24 * [ unused red zone ] 224 */
Calling it stack layout 1 & 2 is inappropriate. The stack layout is essentially the same. It just goes to show things with reference to r1 when stack is setup explicitly vs when redzone is being used...
- Hari
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hari Bathini
maintainerRe: [PATCH v2 1/6] powerpc64/bpf: Move tail_call_cnt to bottom of stack frame
Quoted 104 lines · adubey
Quoted 103 lines · Hari Bathini
Quoted 101 lines · On 14/01/26 5:14 pm, [email protected] wrote:
From: Abhishek Dubey <[email protected]>
In the conventional stack frame, the position of
tail_call_cntis after the NVR save area (BPF_PPC_STACK_SAVE). Whereas, the offset oftail_call_cntin the trampoline frame is after the stack alignment padding. BPF JIT logic could become complex when dealing with frame-sensitive offset calculation oftail_call_cnt. Having the same offset in both frames is the desired objective.The trampoline frame does not have a
BPF_PPC_STACK_SAVEarea. Introducing it leads to under-utilization of extra memory meant only for the offset alignment oftail_call_cnt. Another challenge is the variable alignment padding sitting at the bottom of the trampoline frame, which requires additional handling to computetail_call_cntoffset.This patch addresses the above issues by moving
tail_call_cntto the bottom of the stack frame at offset 0 for both types of frames. This saves additional bytes required byBPF_PPC_STACK_SAVEin trampoline frame, and a common offset computation fortail_call_cntserves both frames.The changes in this patch are required by the third patch in the series, where the 'reference to
tail_call_info' of the main frame is copied into the trampoline frame from the previous frame.Signed-off-by Abhishek Dubey <[email protected]>---
arch/powerpc/net/bpf_jit.h | 4 ++++arch/powerpc/net/bpf_jit_comp64.c| 31 ++++++++++++++++++++-----------2 files changed, 24 insertions(+), 11 deletions(-)arch/powerpc/net/bpf_jit.h +4
-0@@ -72,6 +72,10 @@ } } while (0) #ifdef CONFIG_PPC64 + +/* for tailcall counter */ +#define BPF_PPC_TAILCALL 8 + /* If dummy pass (!image), account for maximum possible instructions */ #define PPC_LI64(d, i) do { \ if (!image) \arch/powerpc/net/bpf_jit_comp64.c +11
-5@@ -20,13 +20,15 @@ #include "bpf_jit.h" /* -* Stack layout: +* Stack layout 1: +* Layout when setting up our own stack frame. +* Note: r1 at bottom, component offsets positive wrt r1. * Ensure the top half (upto local_tmp_var) stays consistent * with our redzone usage. * * [ prev sp ] <------------- -* [ nv gpr save area ] 6*8 | * [ tail_call_cnt ] 8 | +* [ nv gpr save area ] 6*8 | * [ local_tmp_var ] 24 | * fp (r31) --> [ ebpf stack space ] upto 512 | * [ frame header ] 32/112 | @@ -36,10 +38,12 @@ /* for gpr non volatile registers BPG_REG_6 to 10 */ #define BPF_PPC_STACK_SAVE (6*8) /* for bpf JIT code internal usage */ -#define BPF_PPC_STACK_LOCALS 32 +#define BPF_PPC_STACK_LOCALS 24 /* stack frame excluding BPF stack, ensure this is quadword aligned */ #define BPF_PPC_STACKFRAME (STACK_FRAME_MIN_SIZE + \ - BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE) + BPF_PPC_STACK_LOCALS + \ + BPF_PPC_STACK_SAVE + \ + BPF_PPC_TAILCALL) /* BPF register usage */ #define TMP_REG_1 (MAX_BPF_JIT_REG + 0)@@ -87,27 +91,32 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx) } /* +* Stack layout 2: * When not setting up our own stackframe, the redzone (288 bytes) usage is: +* Note: r1 from prev frame. Component offset negative wrt r1. * * [ prev sp ] <------------- * [ ... ] | * sp (r1) ---> [ stack pointer ] -------------- -* [ nv gpr save area ] 6*8 * [ tail_call_cnt ] 8 +* [ nv gpr save area ] 6*8 * [ local_tmp_var ] 24 * [ unused red zone ] 224 */
Quoted 4 lines
Quoted 3 lines
Calling it stack layout 1 & 2 is inappropriate. The stack layout is essentially the same. It just goes to show things with reference to r1 when stack is setup explicitly vs when redzone is being used...
Agree. I am using it as labels to refer in comment. Any better suggestions?
I think the comments could refer to has stack frame vs Redzone case..
- Hari
PATCH v2 2/6 powerpc64/bpf: Support tailcalls with subprogs
In reply to [email protected]From: Abhishek Dubey <[email protected]>
Enabling tailcalls with subprog combinations by referencing method. The actual tailcall count is always maintained in the tail_call_info variable present in the frame of main function (also called entry function). The tail_call_info variables in the frames of all other subprog contains reference to the tail_call_info present in frame of main function.
Dynamic resolution interprets the tail_call_info either as value or reference depending on the context of active frame while tailcall is invoked.
--- arch/powerpc/net/bpf_jit.h | 12 +++++- arch/powerpc/net/bpf_jit_comp.c | 10 ++++- arch/powerpc/net/bpf_jit_comp64.c | 68 +++++++++++++++++++++++-------- 3 files changed, 70 insertions(+), 20 deletions(-)
arch/powerpc/net/bpf_jit.h +10 -2
@@ -51,6 +51,12 @@EMIT (PPC_INST_BRANCH_COND | (((cond) &0x3ff ) <<16 ) | (offset &0xfffc )); \ }while (0 ) +/* Same as PPC_BCC_SHORT, except valid dest is known prior to call. */ +#define PPC_COND_BRANCH (cond, dest) \ + do { \ + long offset = (long)(dest) - CTX_NIA(ctx); \ + EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc)); \ + } while (0)/* * Sign-extended 32-bit immediate load *
@@ -75,6 +81,8 @@/* for tailcall counter */ #define BPF_PPC_TAILCALL 8 +/* for gpr non volatile registers BPG_REG_6 to 10 */ +#define BPF_PPC_STACK_SAVE (6*8)/* If dummy pass (!image), account for maximum possible instructions */ #define PPC_LI64 (d, i) do { \
@@ -170,6 +178,7 @@ struct codegen_context {unsigned int alt_exit_addr;u64 arena_vm_start;u64 user_vm_start; +bool is_subprog; };#define bpf_to_ppc (r) (ctx->b2p[r])
@@ -204,11 +213,10 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);void bpf_jit_build_fentry_stubs (u32 *image,struct codegen_context *ctx);void bpf_jit_realloc_regs (struct codegen_context *ctx);int bpf_jit_emit_exit_insn (u32 *image,struct codegen_context *ctx,int tmp_reg,long exit_addr); -int bpf_add_extable_entry (struct bpf_prog *fp,u32 *image,u32 *fimage,int pass,struct codegen_context *ctx,int insn_idx,int jmp_off,int dst_reg,u32 code); - +int bpf_jit_stack_tailcallinfo_offset (struct codegen_context *ctx);#endif #endif
arch/powerpc/net/bpf_jit_comp.c +8 -2
@@ -206,6 +206,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) cgctx.stack_size =round_up (fp->aux->stack_depth,16 ); cgctx.arena_vm_start =bpf_arena_get_kern_vm_start (fp->aux->arena); cgctx.user_vm_start =bpf_arena_get_user_vm_start (fp->aux->arena); + cgctx.is_subprog =bpf_is_subprog (fp);/* Scouting faux-generate pass 0 */ if (bpf_jit_build_body (fp,NULL ,NULL , &cgctx, addrs,0 , false)) {
@@ -435,6 +436,11 @@ void bpf_jit_free(struct bpf_prog *fp)bpf_prog_unlock_free (fp); } +bool bpf_jit_supports_subprog_tailcalls (void ) +{ +return IS_ENABLED (CONFIG_PPC64 ); +} +bool bpf_jit_supports_kfunc_call (void ) {return true;
@@ -604,7 +610,7 @@ static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_contex int func_frame_offset, int r4_off) { if (IS_ENABLED(CONFIG_PPC64)) { - /* See bpf_jit_stack_tailcallcnt() */ + /* See bpf_jit_stack_tailcallinfo_offset() */ int tailcallcnt_offset = 7 * 8;EMIT (PPC_RAW_LL (_R3, _R1, func_frame_offset - tailcallcnt_offset));
@@ -619,7 +625,7 @@ static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont int func_frame_offset, int r4_off) { if (IS_ENABLED(CONFIG_PPC64)) { - /* See bpf_jit_stack_tailcallcnt() */ + /* See bpf_jit_stack_tailcallinfo_offset() */ int tailcallcnt_offset = 7 * 8;EMIT (PPC_RAW_LL (_R3, _R1, -tailcallcnt_offset));
arch/powerpc/net/bpf_jit_comp64.c +52 -16
118 lines not on this page · raw message
Signature · 1 line
2.48.1
Christophe Leroy (CS GROUP)
maintainerRe: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs
Le 14/01/2026 à 12:44, [email protected] a écrit :
Quoted 25 lines
From: Abhishek Dubey <[email protected]>
Enabling tailcalls with subprog combinations by referencing method. The actual tailcall count is always maintained in the
tail_call_infovariable present in the frame of main function (also called entry function). Thetail_call_infovariables in the frames of all other subprog contains reference to thetail_call_infopresent in frame of main function.Dynamic resolution interprets the
tail_call_infoeither as value or reference depending on the context of active frame while tailcall is invoked.Signed-off-by Abhishek Dubey <[email protected]>--- arch/powerpc/net/bpf_jit.h | 12 +++++- arch/powerpc/net/bpf_jit_comp.c | 10 ++++- arch/powerpc/net/bpf_jit_comp64.c | 68 +++++++++++++++++++++++-------- 3 files changed, 70 insertions(+), 20 deletions(-)arch/powerpc/net/bpf_jit.h +0
-0
arch/powerpc/net/bpf_jit.h line 59
Quoted 9 lines
arch/powerpc/net/bpf_jit.h +6
-0@@ -51,6 +51,12 @@EMIT (PPC_INST_BRANCH_COND | (((cond) &0x3ff ) <<16 ) | (offset &0xfffc )); \ }while (0 ) +/* Same as PPC_BCC_SHORT, except valid dest is known prior to call. */ +#define PPC_COND_BRANCH (cond, dest) \ + do { \ + long offset = (long)(dest) - CTX_NIA(ctx); \ + EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc)); \ + } while (0)
I don't like the idea of duplicating PPC_BCC_SHORT() to just kick the verification out. Now we will have two macros doing the exact same thing with one handling failure case and one ignoring failure case. There is a big risk that one day or another someone will use the wrong macro.
Could you change bpf_jit_build_prologue() to return an int add use PPC_BCC_SHORT() instead of that new PPC_COND_BRANCH() ?
arch/powerpc/net/bpf_jit.h line 85
Quoted 9 lines
arch/powerpc/net/bpf_jit.h +2
-0/* * Sign-extended 32-bit immediate load * @@ -75,6 +81,8 @@/* for tailcall counter */ #define BPF_PPC_TAILCALL 8 +/* for gpr non volatile registers BPG_REG_6 to 10 */ +#define BPF_PPC_STACK_SAVE (6*8)
Add spaces before and after the *
Quoted 12 lines
arch/powerpc/net/bpf_jit.h +1
-0/* If dummy pass (!image), account for maximum possible instructions */ #define PPC_LI64 (d, i) do { \@@ -170,6 +178,7 @@ struct codegen_context {unsigned int alt_exit_addr;u64 arena_vm_start;u64 user_vm_start; +bool is_subprog; };#define bpf_to_ppc (r) (ctx->b2p[r])
arch/powerpc/net/bpf_jit.h void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx); · line 211
Quoted 8 lines
arch/powerpc/net/bpf_jit.h +0
-2@@ -204,11 +213,10 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);void bpf_jit_build_fentry_stubs (u32 *image,struct codegen_context *ctx);void bpf_jit_realloc_regs (struct codegen_context *ctx);int bpf_jit_emit_exit_insn (u32 *image,struct codegen_context *ctx,int tmp_reg,long exit_addr); -int bpf_add_extable_entry (struct bpf_prog *fp,u32 *image,u32 *fimage,int pass,struct codegen_context *ctx,int insn_idx,int jmp_off,int dst_reg,u32 code); -
Not sure why this patch needs to remove those blank lines here and above.
Quoted 174 lines
arch/powerpc/net/bpf_jit.h +1
-0+int bpf_jit_stack_tailcallinfo_offset (struct codegen_context *ctx);#endif #endif arch/powerpc/net/bpf_jit_comp.c +8
-2@@ -206,6 +206,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) cgctx.stack_size =round_up (fp->aux->stack_depth,16 ); cgctx.arena_vm_start =bpf_arena_get_kern_vm_start (fp->aux->arena); cgctx.user_vm_start =bpf_arena_get_user_vm_start (fp->aux->arena); + cgctx.is_subprog =bpf_is_subprog (fp);/* Scouting faux-generate pass 0 */ if (bpf_jit_build_body (fp,NULL ,NULL , &cgctx, addrs,0 , false)) {@@ -435,6 +436,11 @@ void bpf_jit_free(struct bpf_prog *fp)bpf_prog_unlock_free (fp); } +bool bpf_jit_supports_subprog_tailcalls (void ) +{ +return IS_ENABLED (CONFIG_PPC64 ); +} +bool bpf_jit_supports_kfunc_call (void ) {return true;@@ -604,7 +610,7 @@ static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_contex int func_frame_offset, int r4_off) { if (IS_ENABLED(CONFIG_PPC64)) { - /* See bpf_jit_stack_tailcallcnt() */ + /* See bpf_jit_stack_tailcallinfo_offset() */ int tailcallcnt_offset = 7 * 8;EMIT (PPC_RAW_LL (_R3, _R1, func_frame_offset - tailcallcnt_offset));@@ -619,7 +625,7 @@ static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont int func_frame_offset, int r4_off) { if (IS_ENABLED(CONFIG_PPC64)) { - /* See bpf_jit_stack_tailcallcnt() */ + /* See bpf_jit_stack_tailcallinfo_offset() */ int tailcallcnt_offset = 7 * 8;EMIT (PPC_RAW_LL (_R3, _R1, -tailcallcnt_offset));arch/powerpc/net/bpf_jit_comp64.c +52
-16@@ -26,8 +26,12 @@* Ensure the top half (upto local_tmp_var) stays consistent * with our redzone usage. * +* tail_call_info - stores tailcall count value in main program's +* frame, stores reference to tail_call_info of +* main's frame in sub-prog's frame. +* * [ prev sp ] <------------- -* [ tail_call_cnt ] 8 | +* [ tail_call_info ] 8 | * [ nv gpr save area ] 6*8 | * [ local_tmp_var ] 24 | * fp (r31) --> [ ebpf stack space ] upto 512 | @@ -35,8 +39,6 @@* sp (r1) ---> [ stack pointer ] -------------- */ -/* for gpr non volatile registers BPG_REG_6 to 10 */ -#define BPF_PPC_STACK_SAVE (6*8)/* for bpf JIT code internal usage */ #define BPF_PPC_STACK_LOCALS 24/* stack frame excluding BPF stack, ensure this is quadword aligned */ @@ -98,7 +100,7 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)* [ prev sp ] <------------- * [ ... ] | * sp (r1) ---> [ stack pointer ] -------------- -* [ tail_call_cnt ] 8 +* [ tail_call_info ] 8 * [ nv gpr save area ] 6*8 * [ local_tmp_var ] 24 * [ unused red zone ] 224 @@ -114,7 +116,7 @@ static int bpf_jit_stack_local(struct codegen_context *ctx) } } -static int bpf_jit_stack_tailcallcnt (struct codegen_context *ctx) +int bpf_jit_stack_tailcallinfo_offset (struct codegen_context *ctx) {return bpf_jit_stack_local (ctx) +BPF_PPC_STACK_LOCALS +BPF_PPC_STACK_SAVE ; }@@ -147,17 +149,32 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)#endif /* -* Initialize tail_call_cnt if we do tail calls. -* Otherwise, put in NOPs so that it can be skipped when we are -* invoked through a tail call. +* Tail call count(tcc) is saved & updated only in main +* program's frame and the address of tcc in main program's +* frame (tcc_ptr) is saved in subprogs frame. +* +* Offset of tail_call_info on any frame will be interpreted +* as either tcc_ptr or tcc value depending on whether it is +* greater than MAX_TAIL_CALL_CNT or not. */ -if (ctx->seen &SEEN_TAILCALL ) { +if (!ctx->is_subprog) {EMIT (PPC_RAW_LI (bpf_to_ppc (TMP_REG_1 ),0 ));/* this goes in the redzone */ EMIT (PPC_RAW_STD (bpf_to_ppc (TMP_REG_1 ), _R1, -(BPF_PPC_TAILCALL ))); }else { -EMIT (PPC_RAW_NOP ()); -EMIT (PPC_RAW_NOP ()); +/* +* if tail_call_info < MAX_TAIL_CALL_CNT +* main prog calling first subprog -> copy reference +* else +* subsequent subprog calling another subprog -> directly copy content +*/ +EMIT (PPC_RAW_LD (bpf_to_ppc (TMP_REG_2 ), _R1,0 )); +EMIT (PPC_RAW_LD (bpf_to_ppc (TMP_REG_1 ),bpf_to_ppc (TMP_REG_2 ), -(BPF_PPC_TAILCALL ))); +EMIT (PPC_RAW_CMPLWI (bpf_to_ppc (TMP_REG_1 ),MAX_TAIL_CALL_CNT )); +PPC_COND_BRANCH (COND_GT ,CTX_NIA (ctx) +8 ); +EMIT (PPC_RAW_ADDI (bpf_to_ppc (TMP_REG_1 ),bpf_to_ppc (TMP_REG_2 ), + -(BPF_PPC_TAILCALL ))); +EMIT (PPC_RAW_STD (bpf_to_ppc (TMP_REG_1 ), _R1, -(BPF_PPC_TAILCALL ))); }if (bpf_has_stack_frame (ctx)) {@@ -352,19 +369,38 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 oEMIT (PPC_RAW_CMPLW (b2p_index,bpf_to_ppc (TMP_REG_1 )));PPC_BCC_SHORT (COND_GE , out); +EMIT (PPC_RAW_LD (bpf_to_ppc (TMP_REG_1 ), _R1,bpf_jit_stack_tailcallinfo_offset (ctx))); +EMIT (PPC_RAW_CMPLWI (bpf_to_ppc (TMP_REG_1 ),MAX_TAIL_CALL_CNT )); +PPC_COND_BRANCH (COND_LE ,CTX_NIA (ctx) +8 ); + +/* dereference TMP_REG_1 */ +EMIT (PPC_RAW_LD (bpf_to_ppc (TMP_REG_1 ),bpf_to_ppc (TMP_REG_1 ),0 )); +/* -* if (tail_call_cnt >= MAX_TAIL_CALL_CNT) +* if (tail_call_info == MAX_TAIL_CALL_CNT) * goto out; */ -EMIT (PPC_RAW_LD (bpf_to_ppc (TMP_REG_1 ), _R1,bpf_jit_stack_tailcallcnt (ctx)));EMIT (PPC_RAW_CMPLWI (bpf_to_ppc (TMP_REG_1 ),MAX_TAIL_CALL_CNT )); -PPC_BCC_SHORT (COND_GE , out); +PPC_COND_BRANCH (COND_EQ , out);/* -* tail_call_cnt++; +* tail_call_info++; <- Actual value of tcc here */ EMIT (PPC_RAW_ADDI (bpf_to_ppc (TMP_REG_1 ),bpf_to_ppc (TMP_REG_1 ),1 )); -EMIT (PPC_RAW_STD (bpf_to_ppc (TMP_REG_1 ), _R1,bpf_jit_stack_tailcallcnt (ctx))); + +/* +* Before writing updated tail_call_info, distinguish if current frame +* is storing a reference to tail_call_info or actual tcc value in +* tail_call_info. +*/ +EMIT (PPC_RAW_LD (bpf_to_ppc (TMP_REG_2 ), _R1,bpf_jit_stack_tailcallinfo_offset (ctx))); +EMIT (PPC_RAW_CMPLWI (bpf_to_ppc (TMP_REG_2 ),MAX_TAIL_CALL_CNT )); +PPC_COND_BRANCH (COND_GT ,CTX_NIA (ctx) +8 ); + +/* First get address of tail_call_info */ +EMIT (PPC_RAW_ADDI (bpf_to_ppc (TMP_REG_2 ), _R1,bpf_jit_stack_tailcallinfo_offset (ctx))); +/* Writeback updated value to tail_call_info */ +EMIT (PPC_RAW_STD (bpf_to_ppc (TMP_REG_1 ),bpf_to_ppc (TMP_REG_2 ),0 ));/* prog = array->ptrs[index]; */ EMIT (PPC_RAW_MULI (bpf_to_ppc (TMP_REG_1 ), b2p_index,8 ));
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hari Bathini
maintainerRe: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs
Quoted 43 lines · adubey
Quoted 42 lines · Christophe Leroy (CS GROUP)
Le 14/01/2026 à 12:44, [email protected] a écrit :
Quoted 27 lines
From: Abhishek Dubey <[email protected]>
Enabling tailcalls with subprog combinations by referencing method. The actual tailcall count is always maintained in the
tail_call_infovariable present in the frame of main function (also called entry function). Thetail_call_infovariables in the frames of all other subprog contains reference to thetail_call_infopresent in frame of main function.Dynamic resolution interprets the
tail_call_infoeither as value or reference depending on the context of active frame while tailcall is invoked.Signed-off-by Abhishek Dubey <[email protected]>---
arch/powerpc/net/bpf_jit.h | 12 +++++- arch/powerpc/net/bpf_jit_comp.c | 10 ++++-arch/powerpc/net/bpf_jit_comp64.c| 68 +++++++++++++++++++++++--------3 files changed, 70 insertions(+), 20 deletions(-)arch/powerpc/net/bpf_jit.h +0
-0@@ -51,6 +51,12 @@ EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc)); \ } while (0)Quoted 7 lines
+/* Same as
PPC_BCC_SHORT, except valid dest is known prior to call. */ +#definePPC_COND_BRANCH(cond, dest)\
+ do { \
+ long offset = (long)(dest) - CTX_NIA(ctx); \
+EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc)); \
+ } while (0)I don't like the idea of duplicating
PPC_BCC_SHORT()to just kick the verification out. Now we will have two macros doing the exact same thing with one handling failure case and one ignoring failure case. There is a big risk that one day or another someone will use the wrong macro.
Quoted 6 lines
Quoted 2 lines
Could you change
bpf_jit_build_prologue()to return an int add usePPC_BCC_SHORT()instead of that newPPC_COND_BRANCH()?I implemented exactly same change in
bpf_jit_build_prologue(). But, during internal review, @HariBathini suggested to have separate macro with a caution note.@Hari please suggest here!
Not just about the change of return type but the check seems like an overkill for cases where the offset is known and within branch range. How about using BUILD_BUG_ON() to avoid unecessary checks and return type change for places where the branch offset is known and is a constant?
Quoted 213 lines
Quoted 204 lines
arch/powerpc/net/bpf_jit.h line 84
Quoted 9 lines
/*
* Sign-extended 32-bit immediate load
*arch/powerpc/net/bpf_jit.h +2
-0@@ -75,6 +81,8 @@ /* for tailcall counter */ #define BPF_PPC_TAILCALL 8 +/* for gpr non volatile registers BPG_REG_6 to 10 */ +#define BPF_PPC_STACK_SAVE (6*8)Add spaces before and after the *
Quoted 10 lines
arch/powerpc/net/bpf_jit.h +1
-0/* If dummy pass (!image), account for maximum possible instructions */ #define PPC_LI64(d, i) do { \@@ -170,6 +178,7 @@ struct codegen_context { unsigned int alt_exit_addr; u64 arena_vm_start; u64 user_vm_start; + bool is_subprog; }; #define bpf_to_ppc(r) (ctx->b2p[r])arch/powerpc/net/bpf_jit.h void bpf_jit_build_epilogue(u32 *image, struct · line 212
Quoted 9 lines
arch/powerpc/net/bpf_jit.h +0
-2@@ -204,11 +213,10 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx); void bpf_jit_build_fentry_stubs(u32 *image, struct codegen_context *ctx); void bpf_jit_realloc_regs(struct codegen_context *ctx); int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr); - int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass, struct codegen_context *ctx, int insn_idx, int jmp_off, int dst_reg, u32 code); -Not sure why this patch needs to remove those blank lines here and above.
Quoted 170 lines
arch/powerpc/net/bpf_jit.h +1
-0+int bpf_jit_stack_tailcallinfo_offset (struct codegen_context *ctx); #endif #endifarch/powerpc/net/bpf_jit_comp.c +0
-0@@ -206,6 +206,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) cgctx.stack_size = round_up(fp->aux->stack_depth, 16);cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux-
Quoted 1 line
arena);
cgctx.user_vm_start =
bpf_arena_get_user_vm_start(fp->aux->arena); + cgctx.is_subprog =bpf_is_subprog(fp);
/* Scouting faux-generate pass 0 */
if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {arch/powerpc/net/bpf_jit_comp.c +4
-0@@ -435,6 +436,11 @@ void bpf_jit_free(struct bpf_prog *fp) bpf_prog_unlock_free(fp); } +bool bpf_jit_supports_subprog_tailcalls(void) +{ + return IS_ENABLED(CONFIG_PPC64); +} + bool bpf_jit_supports_kfunc_call(void) { return true;@@ -604,7 +610,7 @@ static void
bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_contex int func_frame_offset, int r4_off)
{
if (IS_ENABLED(CONFIG_PPC64)) {
- /* Seebpf_jit_stack_tailcallcnt()*/
+ /* Seebpf_jit_stack_tailcallinfo_offset()*/
inttailcallcnt_offset= 7 * 8;
EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));arch/powerpc/net/bpf_jit_comp.c +0
-0@@ -619,7 +625,7 @@ static void
bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont int func_frame_offset, int r4_off){
if (IS_ENABLED(CONFIG_PPC64)) {
- /* Seebpf_jit_stack_tailcallcnt()*/
+ /* Seebpf_jit_stack_tailcallinfo_offset()*/ inttailcallcnt_offset= 7 * 8;EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));arch/powerpc/net/bpf_jit_comp64.c +5
-1@@ -26,8 +26,12 @@ * Ensure the top half (upto local_tmp_var) stays consistent * with our redzone usage. * + * tail_call_info - stores tailcall count value in main program's + * frame, stores reference to tail_call_info of + * main's frame in sub-prog's frame. + * * [ prev sp ] <------------- - * [ tail_call_cnt ] 8 | + * [ tail_call_info ] 8 | * [ nv gpr save area ] 6*8 | * [ local_tmp_var ] 24 | * fp (r31) --> [ ebpf stack space ] upto 512 |@@ -35,8 +39,6 @@* sp (r1) ---> [ stack pointer ] -------------- */ -/* for gpr non volatile registers
BPG_REG_6to 10 */ -#define BPF_PPC_STACK_SAVE (6*8)
/* for bpf JIT code internal usage */
#define BPF_PPC_STACK_LOCALS 24
/* stack frame excluding BPF stack, ensure this is quadword aligned */arch/powerpc/net/bpf_jit_comp64.c +1
-1@@ -98,7 +100,7 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx) * [ prev sp ] <------------- * [ ... ] | * sp (r1) ---> [ stack pointer ] -------------- - * [ tail_call_cnt ] 8 + * [ tail_call_info ] 8 * [ nv gpr save area ] 6*8 * [ local_tmp_var ] 24* [ unused red zone ] 224
arch/powerpc/net/bpf_jit_comp64.c +1
-0@@ -114,7 +116,7 @@ static int bpf_jit_stack_local(struct codegen_context *ctx) } } -static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx) +int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx) { return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE;}
arch/powerpc/net/bpf_jit_comp64.c +13
-6@@ -147,17 +149,32 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) #endif /* - * Initialize tail_call_cnt if we do tail calls. - * Otherwise, put in NOPs so that it can be skipped when we are - * invoked through a tail call. + * Tail call count(tcc) is saved & updated only in main + * program's frame and the address of tcc in main program's + * frame (tcc_ptr) is saved in subprogs frame. + * + * Offset of tail_call_info on any frame will be interpreted + * as either tcc_ptr or tcc value depending on whether it is + * greater than MAX_TAIL_CALL_CNT or not. */ - if (ctx->seen & SEEN_TAILCALL) { + if (!ctx->is_subprog) { EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0)); /* this goes in the redzone */ EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, - (BPF_PPC_TAILCALL))); } else { - EMIT(PPC_RAW_NOP()); - EMIT(PPC_RAW_NOP()); + /* +* if tail_call_info < MAX_TAIL_CALL_CNT +* main prog calling first subprog -> copy reference +* else +* subsequent subprog calling another subprog -> directly copy content
+ */
+EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, 0));
+EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), -(BPF_PPC_TAILCALL)));
+EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT)); +PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+ EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1),bpf_to_ppc(TMP_REG_2), + -(BPF_PPC_TAILCALL)));
+EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, - (BPF_PPC_TAILCALL)));
}
if (bpf_has_stack_frame(ctx)) {arch/powerpc/net/bpf_jit_comp64.c +0
-0@@ -352,19 +369,38 @@ static int bpf_jit_emit_tail_call(u32 *image,struct
codegen_context*ctx, u32 o
EMIT(PPC_RAW_CMPLW(b2p_index, bpf_to_ppc(TMP_REG_1)));PPC_BCC_SHORT(COND_GE, out);
+EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallinfo_offset(ctx))); +EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
+PPC_COND_BRANCH(COND_LE, CTX_NIA(ctx) + 8);arch/powerpc/net/bpf_jit_comp64.c +4
-0+ + /* dereference TMP_REG_1 */ + EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 0)); +/*
- * if (tail_call_cnt>=MAX_TAIL_CALL_CNT)
+ * if (tail_call_info==MAX_TAIL_CALL_CNT)
* goto out;
*/
-EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT)); -PPC_BCC_SHORT(COND_GE, out);
+PPC_COND_BRANCH(COND_EQ, out);
/*
- * tail_call_cnt++;
+ * tail_call_info++; <- Actual value of tcc here
*/
EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 1)); -EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
+
+ /*
+ * Before writing updatedtail_call_info, distinguish if current frame
+ * is storing a reference totail_call_infoor actual tcc value in + *tail_call_info.
+ */
+EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_2), MAX_TAIL_CALL_CNT)); +PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);Quoted 8 lines
Quoted 8 lines
+
+ /* First get address oftail_call_info*/
+EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+ /* Writeback updated value totail_call_info*/
+EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), 0)); /* prog = array->ptrs[index]; */EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_1), b2p_index, 8));-Abhishek
Christophe Leroy (CS GROUP)
maintainerRe: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs
Le 16/01/2026 à 05:50, Hari Bathini a écrit :
Not received this mail that Hari is reponding to.
Quoted 52 lines · adubey
Quoted 49 lines · Christophe Leroy (CS GROUP)
Quoted 43 lines
Le 14/01/2026 à 12:44, [email protected] a écrit :
Quoted 28 lines
From: Abhishek Dubey <[email protected]>
Enabling tailcalls with subprog combinations by referencing method. The actual tailcall count is always maintained in the
tail_call_infovariable present in the frame of main function (also called entry function). Thetail_call_infovariables in the frames of all other subprog contains reference to thetail_call_infopresent in frame of main function.Dynamic resolution interprets the
tail_call_infoeither as value or reference depending on the context of active frame while tailcall is invoked.Signed-off-by Abhishek Dubey <[email protected]>---
arch/powerpc/net/bpf_jit.h | 12 +++++- arch/powerpc/net/bpf_jit_comp.c | 10 ++++-arch/powerpc/net/bpf_jit_comp64.c| 68 ++++++++++++++++++++++
+--------3 files changed, 70 insertions(+), 20 deletions(-)arch/powerpc/net/bpf_jit.h +0
-0@@ -51,6 +51,12 @@ EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc)); \ } while (0)Quoted 7 lines
+/* Same as
PPC_BCC_SHORT, except valid dest is known prior to call. */ +#definePPC_COND_BRANCH(cond, dest)\
+ do { \
+ long offset = (long)(dest) - CTX_NIA(ctx); \
+EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc)); \
+ } while (0)I don't like the idea of duplicating
PPC_BCC_SHORT()to just kick the verification out. Now we will have two macros doing the exact same thing with one handling failure case and one ignoring failure case. There is a big risk that one day or another someone will use the wrong macro.Quoted 2 lines
Could you change
bpf_jit_build_prologue()to return an int add usePPC_BCC_SHORT()instead of that newPPC_COND_BRANCH()?I implemented exactly same change in
bpf_jit_build_prologue(). But, during internal review, @HariBathini suggested to have separate macro with a caution note.
Quoted 7 lines
Quoted 1 line
@Hari please suggest here!
Not just about the change of return type but the check seems like an overkill for cases where the offset is known and within branch range. How about using
BUILD_BUG_ON()to avoid unecessary checks and return type change for places where the branch offset is known and is a constant?
When offset is a constant known at build time, checks are eliminated by gcc at build, see exemple below from disasembly of bpf_jit_comp32.o, there are no checks.
PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
36d8: 3c 80 41 81 lis r4,16769
EMIT(PPC_RAW_CMPLW(src_reg, _R0));
36dc: 81 3f 00 04 lwz r9,4(r31)
PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
36e0: 60 84 00 10 ori r4,r4,16
EMIT(PPC_RAW_CMPLW(src_reg, _R0));
36e4: 39 29 00 01 addi r9,r9,1
PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
36e8: 55 23 10 3a slwi r3,r9,2
EMIT(PPC_RAW_CMPLW(src_reg, _R0));
36ec: 91 3f 00 04 stw r9,4(r31)
PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
36f0: 7c 97 19 2e stwx r4,r23,r3
EMIT(PPC_RAW_LI(dst_reg, 0));
36f4: 55 49 a9 94 rlwinm r9,r10,21,6,10
PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
36f8: 80 9f 00 04 lwz r4,4(r31)
EMIT(PPC_RAW_LI(dst_reg, 0));
36fc: 65 29 38 00 oris r9,r9,14336
PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
3700: 38 84 00 01 addi r4,r4,1
EMIT(PPC_RAW_LI(dst_reg, 0));
3704: 54 83 10 3a slwi r3,r4,2
PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
3708: 90 9f 00 04 stw r4,4(r31)
EMIT(PPC_RAW_LI(dst_reg, 0));
370c: 7d 37 19 2e stwx r9,r23,r3Quoted 219 lines
Quoted 218 lines
Quoted 209 lines
arch/powerpc/net/bpf_jit.h line 84
Quoted 10 lines
/*
* Sign-extended 32-bit immediate load
*arch/powerpc/net/bpf_jit.h +2
-0@@ -75,6 +81,8 @@ /* for tailcall counter */ #define BPF_PPC_TAILCALL 8 +/* for gpr non volatile registers BPG_REG_6 to 10 */ +#define BPF_PPC_STACK_SAVE (6*8)Add spaces before and after the *
Quoted 10 lines
arch/powerpc/net/bpf_jit.h +1
-0/* If dummy pass (!image), account for maximum possible instructions */ #define PPC_LI64(d, i) do { \@@ -170,6 +178,7 @@ struct codegen_context { unsigned int alt_exit_addr; u64 arena_vm_start; u64 user_vm_start; + bool is_subprog; }; #define bpf_to_ppc(r) (ctx->b2p[r])arch/powerpc/net/bpf_jit.h void bpf_jit_build_epilogue(u32 *image, struct · line 212
Quoted 9 lines
arch/powerpc/net/bpf_jit.h +0
-2@@ -204,11 +213,10 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx); void bpf_jit_build_fentry_stubs(u32 *image, struct codegen_context *ctx); void bpf_jit_realloc_regs(struct codegen_context *ctx); int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr); - int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass, struct codegen_context *ctx, int insn_idx, int jmp_off, int dst_reg, u32 code); -Not sure why this patch needs to remove those blank lines here and above.
Quoted 174 lines
arch/powerpc/net/bpf_jit.h +1
-0+int bpf_jit_stack_tailcallinfo_offset (struct codegen_context *ctx); #endif #endifarch/powerpc/net/bpf_jit_comp.c +0
-0@@ -206,6 +206,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) cgctx.stack_size = round_up(fp->aux->stack_depth, 16);cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux-
Quoted 1 line
arena);
cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux-
Quoted 1 line
arena);
+ cgctx.is_subprog =
bpf_is_subprog(fp);
/* Scouting faux-generate pass 0 */
if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {arch/powerpc/net/bpf_jit_comp.c +4
-0@@ -435,6 +436,11 @@ void bpf_jit_free(struct bpf_prog *fp) bpf_prog_unlock_free(fp); } +bool bpf_jit_supports_subprog_tailcalls(void) +{ + return IS_ENABLED(CONFIG_PPC64); +} + bool bpf_jit_supports_kfunc_call(void) { return true;@@ -604,7 +610,7 @@ static void
bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_contex int func_frame_offset, int r4_off)
{
if (IS_ENABLED(CONFIG_PPC64)) {
- /* Seebpf_jit_stack_tailcallcnt()*/
+ /* Seebpf_jit_stack_tailcallinfo_offset()*/
inttailcallcnt_offset= 7 * 8;
EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));arch/powerpc/net/bpf_jit_comp.c +0
-0@@ -619,7 +625,7 @@ static void
bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont int func_frame_offset, int r4_off){
if (IS_ENABLED(CONFIG_PPC64)) {
- /* Seebpf_jit_stack_tailcallcnt()*/
+ /* Seebpf_jit_stack_tailcallinfo_offset()*/ inttailcallcnt_offset= 7 * 8;EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));arch/powerpc/net/bpf_jit_comp64.c +5
-1@@ -26,8 +26,12 @@ * Ensure the top half (upto local_tmp_var) stays consistent * with our redzone usage. * + * tail_call_info - stores tailcall count value in main program's + * frame, stores reference to tail_call_info of + * main's frame in sub-prog's frame. + * * [ prev sp ] <------------- - * [ tail_call_cnt ] 8 | + * [ tail_call_info ] 8 | * [ nv gpr save area ] 6*8 | * [ local_tmp_var ] 24 | * fp (r31) --> [ ebpf stack space ] upto 512 |@@ -35,8 +39,6 @@* sp (r1) ---> [ stack pointer ] -------------- */ -/* for gpr non volatile registers
BPG_REG_6to 10 */ -#define BPF_PPC_STACK_SAVE (6*8)
/* for bpf JIT code internal usage */
#define BPF_PPC_STACK_LOCALS 24
/* stack frame excluding BPF stack, ensure this is quadword aligned */arch/powerpc/net/bpf_jit_comp64.c +1
-1@@ -98,7 +100,7 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx) * [ prev sp ] <------------- * [ ... ] | * sp (r1) ---> [ stack pointer ] -------------- - * [ tail_call_cnt ] 8 + * [ tail_call_info ] 8 * [ nv gpr save area ] 6*8 * [ local_tmp_var ] 24* [ unused red zone ] 224
arch/powerpc/net/bpf_jit_comp64.c +1
-0@@ -114,7 +116,7 @@ static int bpf_jit_stack_local(struct codegen_context *ctx) } } -static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx) +int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx) { return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE;}
arch/powerpc/net/bpf_jit_comp64.c +13
-6@@ -147,17 +149,32 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) #endif /* - * Initialize tail_call_cnt if we do tail calls. - * Otherwise, put in NOPs so that it can be skipped when we are - * invoked through a tail call. + * Tail call count(tcc) is saved & updated only in main + * program's frame and the address of tcc in main program's + * frame (tcc_ptr) is saved in subprogs frame. + * + * Offset of tail_call_info on any frame will be interpreted + * as either tcc_ptr or tcc value depending on whether it is + * greater than MAX_TAIL_CALL_CNT or not. */ - if (ctx->seen & SEEN_TAILCALL) { + if (!ctx->is_subprog) { EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0)); /* this goes in the redzone */ EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, - (BPF_PPC_TAILCALL))); } else { - EMIT(PPC_RAW_NOP()); - EMIT(PPC_RAW_NOP()); + /* +* if tail_call_info < MAX_TAIL_CALL_CNT +* main prog calling first subprog -> copy reference +* else +* subsequent subprog calling another subprog -> directly copy content
+ */
+EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, 0)); +EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), -(BPF_PPC_TAILCALL)));
+EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
+PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8); + EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1),bpf_to_ppc(TMP_REG_2),
+ -(BPF_PPC_TAILCALL))); +EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, - (BPF_PPC_TAILCALL)));
}
if (bpf_has_stack_frame(ctx)) {arch/powerpc/net/bpf_jit_comp64.c +0
-0@@ -352,19 +369,38 @@ static int bpf_jit_emit_tail_call(u32 *image,struct
codegen_context*ctx, u32 o
EMIT(PPC_RAW_CMPLW(b2p_index, bpf_to_ppc(TMP_REG_1)));PPC_BCC_SHORT(COND_GE, out);
+EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallinfo_offset(ctx))); +EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
+PPC_COND_BRANCH(COND_LE, CTX_NIA(ctx) + 8);arch/powerpc/net/bpf_jit_comp64.c +4
-0+ + /* dereference TMP_REG_1 */ + EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 0)); +/*
- * if (tail_call_cnt>=MAX_TAIL_CALL_CNT)
+ * if (tail_call_info==MAX_TAIL_CALL_CNT)
* goto out;
*/
-EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT)); -PPC_BCC_SHORT(COND_GE, out);
+PPC_COND_BRANCH(COND_EQ, out);
/*
- * tail_call_cnt++;
+ * tail_call_info++; <- Actual value of tcc here
*/
EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 1)); -EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
+
+ /*
+ * Before writing updatedtail_call_info, distinguish if current frame
+ * is storing a reference totail_call_infoor actual tcc value in + *tail_call_info.
+ */
+EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_2), MAX_TAIL_CALL_CNT)); +PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+Quoted 8 lines
Quoted 8 lines
+ /* First get address of
tail_call_info*/
+EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+ /* Writeback updated value totail_call_info*/
+EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), 0));
/* prog = array->ptrs[index]; */EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_1), b2p_index, 8));-Abhishek
Hari Bathini
maintainerRe: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs
Quoted 5 lines · Christophe Leroy (CS GROUP)
Le 16/01/2026 à 05:50, Hari Bathini a écrit :
Not received this mail that Hari is reponding to.
That is weird.
Quoted 89 lines
Quoted 52 lines · adubey
Quoted 49 lines · Christophe Leroy (CS GROUP)
Quoted 43 lines
Le 14/01/2026 à 12:44, [email protected] a écrit :
Quoted 28 lines
From: Abhishek Dubey <[email protected]>
Enabling tailcalls with subprog combinations by referencing method. The actual tailcall count is always maintained in the
tail_call_infovariable present in the frame of main function (also called entry function). Thetail_call_infovariables in the frames of all other subprog contains reference to thetail_call_infopresent in frame of main function.Dynamic resolution interprets the
tail_call_infoeither as value or reference depending on the context of active frame while tailcall is invoked.Signed-off-by Abhishek Dubey <[email protected]>---
arch/powerpc/net/bpf_jit.h | 12 +++++- arch/powerpc/net/bpf_jit_comp.c | 10 ++++-arch/powerpc/net/bpf_jit_comp64.c| 68 ++++++++++++++++++++++
+--------3 files changed, 70 insertions(+), 20 deletions(-)arch/powerpc/net/bpf_jit.h +0
-0@@ -51,6 +51,12 @@ EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc)); \ } while (0)Quoted 7 lines
+/* Same as
PPC_BCC_SHORT, except valid dest is known prior to call. */ +#definePPC_COND_BRANCH(cond, dest)\
+ do { \
+ long offset = (long)(dest) - CTX_NIA(ctx); \
+EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc)); \
+ } while (0)I don't like the idea of duplicating
PPC_BCC_SHORT()to just kick the verification out. Now we will have two macros doing the exact same thing with one handling failure case and one ignoring failure case. There is a big risk that one day or another someone will use the wrong macro.Quoted 2 lines
Could you change
bpf_jit_build_prologue()to return an int add usePPC_BCC_SHORT()instead of that newPPC_COND_BRANCH()?I implemented exactly same change in
bpf_jit_build_prologue(). But, during internal review, @HariBathini suggested to have separate macro with a caution note.Quoted 7 lines
Quoted 1 line
@Hari please suggest here!
Not just about the change of return type but the check seems like an overkill for cases where the offset is known and within branch range. How about using
BUILD_BUG_ON()to avoid unecessary checks and return type change for places where the branch offset is known and is a constant?When offset is a constant known at build time, checks are eliminated by gcc at build, see exemple below from disasembly of bpf_jit_comp32.o, there are no checks.
PPC_BCC_SHORT(COND_GT, (
ctx->idx+ 4) *
4);
36d8: 3c 80 41 81 lis r4,16769EMIT(PPC_RAW_CMPLW(src_reg, _R0)); 36dc: 81 3f 00 04 lwz r9,4(r31) PPC_BCC_SHORT(COND_GT, (ctx->idx+ 4) *
4);
36e0: 60 84 00 10 ori r4,r4,16EMIT(PPC_RAW_CMPLW(src_reg, _R0)); 36e4: 39 29 00 01 addi r9,r9,1 PPC_BCC_SHORT(COND_GT, (ctx->idx+ 4) *
4);
36e8: 55 23 10 3a slwi r3,r9,2EMIT(PPC_RAW_CMPLW(src_reg, _R0)); 36ec: 91 3f 00 04 stw r9,4(r31) PPC_BCC_SHORT(COND_GT, (ctx->idx+ 4) *
4);
36f0: 7c 97 19 2e stwx r4,r23,r3EMIT(PPC_RAW_LI(dst_reg, 0)); 36f4: 55 49 a9 94 rlwinm r9,r10,21,6,10 PPC_BCC_SHORT(COND_GT, (ctx->idx+ 4) *
4);
36f8: 80 9f 00 04 lwz r4,4(r31)EMIT(PPC_RAW_LI(dst_reg, 0)); 36fc: 65 29 38 00 oris r9,r9,14336 PPC_BCC_SHORT(COND_GT, (ctx->idx+ 4) *
4);
Quoted 9 lines
3700: 38 84 00 01 addi r4,r4,1
EMIT(PPC_RAW_LI(dst_reg, 0)); 3704: 54 83 10 3a slwi r3,r4,2 PPC_BCC_SHORT(COND_GT, (ctx->idx+ 4) *
4);
3708: 90 9f 00 04 stw r4,4(r31)EMIT(PPC_RAW_LI(dst_reg, 0)); 370c: 7d 37 19 2e stwx r9,r23,r3
Interesting. I do see is_offset_in_cond_branch_range() in action with constant offsets too, on ppc64 compile at least. fwiw, I had this optimized version in mind for constant offset:
#define PPC_BCC_CONST_SHORT(cond, offset)
\
do {
\
BUILD_BUG_ON(offset < -0x8000 || offset > 0x7fff ||
(offset & 0x3)); \
EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) |
(offset & 0xfffc)); \
} while (0)With that, something like:
PPC_BCC_SHORT(COND_NE, (ctx->idx + 3) * 4);
becomes
PPC_BCC_CONST_SHORT(COND_NE, 12);
- Hari
Hari Bathini
maintainerRe: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs
In reply to [email protected]Quoted 8 lines · On 14/01/26 5:14 pm, [email protected] wrote:
From: Abhishek Dubey<[email protected]>
Enabling tailcalls with subprog combinations by referencing method. The actual tailcall count is always maintained in the
tail_call_infovariable present in the frame of main function (also called entry function). Thetail_call_infovariables in the frames of all other subprog contains reference to thetail_call_infopresent in frame of main function.
I would rephrase:
"The tail_call_info variables in the frames of all other subprog contains reference to the tail_call_info present in frame of main function."
to:
"The tail_call_info field in the stack frame of subprogs contains reference to the tail_call_info field in the stack frame of main BPF program."
- Hari
PATCH v2 3/6 powerpc64/bpf: Tailcall handling with trampolines
In reply to [email protected]From: Abhishek Dubey <[email protected]>
The trampoline mechanism sets up its own stack frame and an additional dummy frame. We need to have additional JIT instructions handling tailcall dereferencing in the trampoline's context.
We don't add the two stack frames pointed above, rather add space for tail_call_info at bottom in trampoline frame for ppc64. This makes the trampoline's frame consistent with layout of all other frames wrt tail_call_info offset.
--- arch/powerpc/net/bpf_jit_comp.c | 83 ++++++++++++++++++++++----------- 1 file changed, 56 insertions(+), 27 deletions(-)
arch/powerpc/net/bpf_jit_comp.c +56 -27
@@ -606,33 +606,58 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_contextreturn 0 ; } -static void bpf_trampoline_setup_tail_call_cnt (u32 *image,struct codegen_context *ctx, -int func_frame_offset,int r4_off) -{ -if (IS_ENABLED (CONFIG_PPC64 )) { -/* See bpf_jit_stack_tailcallinfo_offset() */ -int tailcallcnt_offset =7 *8 ; - -EMIT (PPC_RAW_LL (_R3, _R1, func_frame_offset - tailcallcnt_offset)); -EMIT (PPC_RAW_STL (_R3, _R1, -tailcallcnt_offset)); - }else { -/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */ -EMIT (PPC_RAW_LL (_R4, _R1, r4_off)); - } -} +/* +* Refer the label 'Generated stack layout' in this file for actual stack +* layout during trampoline invocation. +* +* Refer __arch_prepare_bpf_trampoline() for stack component details. +* +* The tailcall count/reference is present in caller's stack frame. Its required +* to copy the content of tail_call_info before calling the actual function +* to which the trampoline is attached. +* +*/ -static void bpf_trampoline_restore_tail_call_cnt (u32 *image,struct codegen_context *ctx, -int func_frame_offset,int r4_off) +static void bpf_trampoline_setup_tail_call_info (u32 *image,struct codegen_context *ctx, +int func_frame_offset, +int bpf_dummy_frame_size,int r4_off) {if (IS_ENABLED (CONFIG_PPC64 )) {/* See bpf_jit_stack_tailcallinfo_offset() */ -int tailcallcnt_offset =7 *8 ; +int tailcallinfo_offset =BPF_PPC_TAILCALL ; +/* +* func_frame_offset = ...(1) +* bpf_dummy_frame_size + trampoline_frame_size +*/ +EMIT (PPC_RAW_LD (_R4, _R1, func_frame_offset)); +EMIT (PPC_RAW_LD (_R3, _R4, -tailcallinfo_offset)); + +/* +* Setting the tail_call_info in trampoline's frame +* depending on if previous frame had value or reference. +*/ +EMIT (PPC_RAW_CMPLWI (_R3,MAX_TAIL_CALL_CNT )); +PPC_COND_BRANCH (COND_GT ,CTX_NIA (ctx) +8 ); +EMIT (PPC_RAW_ADDI (_R3, _R4,bpf_jit_stack_tailcallinfo_offset (ctx))); +/* +* From ...(1) above: +* trampoline_frame_bottom = ...(2) +* func_frame_offset - bpf_dummy_frame_size +* +* Using ...(2) derived above: +* trampoline_tail_call_info_offset = ...(3) +* trampoline_frame_bottom - tailcallinfo_offset +* +* From ...(3): +* Use trampoline_tail_call_info_offset to write reference of main's +* tail_call_info in trampoline frame. +*/ +EMIT (PPC_RAW_STL (_R3, _R1, (func_frame_offset - bpf_dummy_frame_size) + - tailcallinfo_offset)); -EMIT (PPC_RAW_LL (_R3, _R1, -tailcallcnt_offset)); -EMIT (PPC_RAW_STL (_R3, _R1, func_frame_offset - tailcallcnt_offset)); }else {/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */ -EMIT (PPC_RAW_STL (_R4, _R1, r4_off)); +EMIT (PPC_RAW_LL (_R4, _R1, r4_off)); } }
@@ -720,6 +745,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im* LR save area [ r0 save (64-bit) ] | header * [ r0 save (32-bit) ] | * dummy frame for unwind [ back chain 1 ] -- +* [ tail_call_info ] non optional - 64-bit powerpc * [ padding ] align stack frame * r4_off [ r4 (tailcallcnt) ] optional - 32-bit powerpc * alt_lr_off [ real lr (ool stub)] optional - actual lr
@@ -801,8 +827,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im } } -/* Padding to align stack frame, if any */ - bpf_frame_size =round_up (bpf_frame_size,SZL *2 ); +if (!(bpf_frame_size % (2 *SZL ))) { +/* Stack is 16-byte aligned */ +/* Room for padding followed by 64-bit tail_call_info */ + bpf_frame_size +=SZL +BPF_PPC_TAILCALL ; + }else { +/* Room for 64-bit tail_call_info */ + bpf_frame_size +=BPF_PPC_TAILCALL ; + }/* Dummy frame size for proper unwind - includes 64-bytes red zone for 64-bit powerpc */ bpf_dummy_frame_size =STACK_FRAME_MIN_SIZE +64 ;
@@ -902,7 +934,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im/* Replicate tail_call_cnt before calling the original BPF prog */ if (flags &BPF_TRAMP_F_TAIL_CALL_CTX ) -bpf_trampoline_setup_tail_call_cnt (image, ctx, func_frame_offset, r4_off); +bpf_trampoline_setup_tail_call_info (image, ctx, func_frame_offset, + bpf_dummy_frame_size, r4_off);/* Restore args */ bpf_trampoline_restore_args_stack (image, ctx, func_frame_offset, nr_regs, regs_off);
@@ -917,10 +950,6 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im/* Store return value for bpf prog to access */ EMIT (PPC_RAW_STL (_R3, _R1, retval_off)); -/* Restore updated tail_call_cnt */ -if (flags &BPF_TRAMP_F_TAIL_CALL_CTX ) -bpf_trampoline_restore_tail_call_cnt (image, ctx, func_frame_offset, r4_off); -/* Reserve space to patch branch instruction to skip fexit progs */ if (ro_image)/* image is NULL for dummy pass */ im->ip_after_call = &((u32 *)ro_image)[ctx->idx];
Signature · 1 line
2.48.1
[email protected]
AI reviewRe: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines
Should this LD instruction be an ADDI instead? Looking at the stack frame setup, after the two STLU instructions in __arch_prepare_bpf_trampoline():
Quoted 4 lines
arch/powerpc/net/bpf_jit_comp.c +0
-0
[ ... ]
arch/powerpc/net/bpf_jit_comp.c static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context · line 633
Quoted 13 lines
arch/powerpc/net/bpf_jit_comp.c +13
-0+static void bpf_trampoline_setup_tail_call_info (u32 *image,struct codegen_context *ctx, +int func_frame_offset, +int bpf_dummy_frame_size,int r4_off) +{ +if (IS_ENABLED (CONFIG_PPC64 )) { +/* See bpf_jit_stack_tailcallinfo_offset() */ +int tailcallinfo_offset =BPF_PPC_TAILCALL ; +/* +* func_frame_offset = ...(1) +* bpf_dummy_frame_size + trampoline_frame_size +*/ +EMIT (PPC_RAW_LD (_R4, _R1, func_frame_offset)); +EMIT (PPC_RAW_LD (_R3, _R4, -tailcallinfo_offset));
^^^^
Should this LD instruction be an ADDI instead? Looking at the stack frame setup, after the two STLU instructions in __arch_prepare_bpf_trampoline():
STLU _R1, _R1, -bpf_dummy_frame_size -> stores original _R1 at dummy frame STLU _R1, _R1, -bpf_frame_size -> stores dummy frame ptr at tramp frame
From the trampoline frame, the back chain stored in the dummy frame at offset bpf_frame_size points to the caller's frame. But func_frame_offset equals (bpf_dummy_frame_size + bpf_frame_size), which points into the caller's frame itself. Loading from that address returns the caller's back chain (pointing to the grandparent frame), not a pointer to the caller's frame.
This appears to cause the subsequent load at [_R4 - 8] to read the grandparent's tail_call_info rather than the caller's. Would loading from [_R1 + bpf_frame_size] give the correct pointer to the caller's frame?
For comparison, the subprogram prologue in bpf_jit_build_prologue() uses [_R1 + 0] to get the back chain pointing to its caller, which seems like the right pattern.
arch/powerpc/net/bpf_jit_comp.c static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context · line 641
Quoted 8 lines
arch/powerpc/net/bpf_jit_comp.c +8
-0+ +/* +* Setting the tail_call_info in trampoline's frame +* depending on if previous frame had value or reference. +*/ +EMIT (PPC_RAW_CMPLWI (_R3,MAX_TAIL_CALL_CNT )); +PPC_COND_BRANCH (COND_GT ,CTX_NIA (ctx) +8 ); +EMIT (PPC_RAW_ADDI (_R3, _R4,bpf_jit_stack_tailcallinfo_offset (ctx)));
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/20993216190
kernel test robot
build botRe: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines
In reply to [email protected]kernel test robot noticed the following build errors:
Hi,
kernel test robot noticed the following build errors:
[auto build test ERROR on powerpc/next]
[also build test ERROR on powerpc/fixes linus/master v6.19-rc5 next-20260114] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/adubey-linux-ibm-com/powerpc64-bpf-Move-tail_call_cnt-to-bottom-of-stack-frame/20260114-195044 base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next patch link: https://lore.kernel.org/r/20260114114450.30405-4-adubey%40linux.ibm.com patch subject: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines config: powerpc-randconfig-002-20260114 (https://download.01.org/0day-ci/archive/20260115/[email protected]/config) compiler: powerpc-linux-gcc (GCC) 12.5.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260115/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
arch/powerpc/net/bpf_jit_comp.c: In function 'bpf_trampoline_setup_tail_call_info':
>> arch/powerpc/net/bpf_jit_comp.c:644:43: error: 'BPF_PPC_TAILCALL' undeclared (first use in this function); did you mean 'BPF_TAIL_CALL'?
644 | int tailcallinfo_offset = BPF_PPC_TAILCALL;
| ^~~~~~~~~~~~~~~~
| BPF_TAIL_CALL
arch/powerpc/net/bpf_jit_comp.c:644:43: note: each undeclared identifier is reported only once for each function it appears in
arch/powerpc/net/bpf_jit_comp.c: In function '__arch_prepare_bpf_trampoline':
arch/powerpc/net/bpf_jit_comp.c:850:41: error: 'BPF_PPC_TAILCALL' undeclared (first use in this function); did you mean 'BPF_TAIL_CALL'?
850 | bpf_frame_size += SZL + BPF_PPC_TAILCALL;
| ^~~~~~~~~~~~~~~~
| BPF_TAIL_CALLvim +644 arch/powerpc/net/bpf_jit_comp.c
625
626 /*
627 * Refer the label 'Generated stack layout' in this file for actual stack
628 * layout during trampoline invocation.
629 *
630 * Refer __arch_prepare_bpf_trampoline() for stack component details.
631 *
632 * The tailcall count/reference is present in caller's stack frame. Its required
633 * to copy the content of tail_call_info before calling the actual function
634 * to which the trampoline is attached.
635 *
636 */
637
638 static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx,
639 int func_frame_offset,
640 int bpf_dummy_frame_size, int r4_off)
641 {
642 if (IS_ENABLED(CONFIG_PPC64)) {
643 /* See bpf_jit_stack_tailcallinfo_offset() */
> 644 int tailcallinfo_offset = BPF_PPC_TAILCALL;
645 /*
646 * func_frame_offset = ...(1)
647 * bpf_dummy_frame_size + trampoline_frame_size
648 */
649 EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset));
650 EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset));
651
652 /*
653 * Setting the tail_call_info in trampoline's frame
654 * depending on if previous frame had value or reference.
655 */
656 EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT));
657 PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
658 EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx)));
659 /*
660 * From ...(1) above:
661 * trampoline_frame_bottom = ...(2)
662 * func_frame_offset - bpf_dummy_frame_size
663 *
664 * Using ...(2) derived above:
665 * trampoline_tail_call_info_offset = ...(3)
666 * trampoline_frame_bottom - tailcallinfo_offset
667 *
668 * From ...(3):
669 * Use trampoline_tail_call_info_offset to write reference of main's
670 * tail_call_info in trampoline frame.
671 */
672 EMIT(PPC_RAW_STL(_R3, _R1, (func_frame_offset - bpf_dummy_frame_size)
673 - tailcallinfo_offset));
674
675 } else {
676 /* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
677 EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
678 }
679 }
680
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Hari Bathini
maintainerRe: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines
In reply to [email protected]Quoted 52 lines · On 14/01/26 5:14 pm, [email protected] wrote:
From: Abhishek Dubey <[email protected]>
The trampoline mechanism sets up its own stack frame and an additional dummy frame. We need to have additional JIT instructions handling tailcall dereferencing in the trampoline's context.
We don't add the two stack frames pointed above, rather add space for
tail_call_infoat bottom in trampoline frame for ppc64. This makes the trampoline's frame consistent with layout of all other frames wrttail_call_infooffset.Signed-off-by Abhishek Dubey <[email protected]>--- arch/powerpc/net/bpf_jit_comp.c | 83 ++++++++++++++++++++++----------- 1 file changed, 56 insertions(+), 27 deletions(-)arch/powerpc/net/bpf_jit_comp.c +11
-15@@ -606,33 +606,58 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_contextreturn 0 ; } -static void bpf_trampoline_setup_tail_call_cnt (u32 *image,struct codegen_context *ctx, -int func_frame_offset,int r4_off) -{ -if (IS_ENABLED (CONFIG_PPC64 )) { -/* See bpf_jit_stack_tailcallinfo_offset() */ -int tailcallcnt_offset =7 *8 ; - -EMIT (PPC_RAW_LL (_R3, _R1, func_frame_offset - tailcallcnt_offset)); -EMIT (PPC_RAW_STL (_R3, _R1, -tailcallcnt_offset)); - }else { -/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */ -EMIT (PPC_RAW_LL (_R4, _R1, r4_off)); - } -} +/* +* Refer the label 'Generated stack layout' in this file for actual stack +* layout during trampoline invocation. +* +* Refer __arch_prepare_bpf_trampoline() for stack component details. +* +* The tailcall count/reference is present in caller's stack frame. Its required +* to copy the content of tail_call_info before calling the actual function +* to which the trampoline is attached. +* +*/ -static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_context *ctx,
arch/powerpc/net/bpf_jit_comp.c static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context · line 628
Quoted 10 lines
arch/powerpc/net/bpf_jit_comp.c +4
-2- int func_frame_offset, int r4_off) +static void bpf_trampoline_setup_tail_call_info (u32 *image,struct codegen_context *ctx, +int func_frame_offset, +int bpf_dummy_frame_size,int r4_off) {if (IS_ENABLED (CONFIG_PPC64 )) {/* See bpf_jit_stack_tailcallinfo_offset() */ -int tailcallcnt_offset =7 *8 ; +int tailcallinfo_offset =BPF_PPC_TAILCALL ;
This offset update should have been part of patch#1
Quoted 55 lines
arch/powerpc/net/bpf_jit_comp.c +32
-5+/* +* func_frame_offset = ...(1) +* bpf_dummy_frame_size + trampoline_frame_size +*/ +EMIT (PPC_RAW_LD (_R4, _R1, func_frame_offset)); +EMIT (PPC_RAW_LD (_R3, _R4, -tailcallinfo_offset)); + +/* +* Setting the tail_call_info in trampoline's frame +* depending on if previous frame had value or reference. +*/ +EMIT (PPC_RAW_CMPLWI (_R3,MAX_TAIL_CALL_CNT )); +PPC_COND_BRANCH (COND_GT ,CTX_NIA (ctx) +8 ); +EMIT (PPC_RAW_ADDI (_R3, _R4,bpf_jit_stack_tailcallinfo_offset (ctx))); +/* +* From ...(1) above: +* trampoline_frame_bottom = ...(2) +* func_frame_offset - bpf_dummy_frame_size +* +* Using ...(2) derived above: +* trampoline_tail_call_info_offset = ...(3) +* trampoline_frame_bottom - tailcallinfo_offset +* +* From ...(3): +* Use trampoline_tail_call_info_offset to write reference of main's +* tail_call_info in trampoline frame. +*/ +EMIT (PPC_RAW_STL (_R3, _R1, (func_frame_offset - bpf_dummy_frame_size) + - tailcallinfo_offset)); -EMIT (PPC_RAW_LL (_R3, _R1, -tailcallcnt_offset)); -EMIT (PPC_RAW_STL (_R3, _R1, func_frame_offset - tailcallcnt_offset)); }else {/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */ -EMIT (PPC_RAW_STL (_R4, _R1, r4_off)); +EMIT (PPC_RAW_LL (_R4, _R1, r4_off)); } }@@ -720,6 +745,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im* LR save area [ r0 save (64-bit) ] | header * [ r0 save (32-bit) ] | * dummy frame for unwind [ back chain 1 ] -- +* [ tail_call_info ] non optional - 64-bit powerpc * [ padding ] align stack frame * r4_off [ r4 (tailcallcnt) ] optional - 32-bit powerpc * alt_lr_off [ real lr (ool stub)] optional - actual lr @@ -801,8 +827,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im } } -/* Padding to align stack frame, if any */ - bpf_frame_size =round_up (bpf_frame_size,SZL *2 ); +if (!(bpf_frame_size % (2 *SZL ))) {
arch/powerpc/net/bpf_jit_comp.c static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · line 840
Quoted 10 lines
arch/powerpc/net/bpf_jit_comp.c +7
-0+/* Stack is 16-byte aligned */ +/* Room for padding followed by 64-bit tail_call_info */ + bpf_frame_size +=SZL +BPF_PPC_TAILCALL ; + }else { +/* Room for 64-bit tail_call_info */ + bpf_frame_size +=BPF_PPC_TAILCALL ; + }/* Dummy frame size for proper unwind - includes 64-bytes red zone for 64-bit powerpc */ bpf_dummy_frame_size =STACK_FRAME_MIN_SIZE +64 ;
This change assumes the size is at least 8-byte aligned which is true today but better skip that assumption by not touching the padding part. The above hunk could simply be:
arch/powerpc/net/bpf_jit_comp.c +4 -0
@@ -795,6 +795,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im } } +/* Save tailcall count pointer at the same offset on the stack where subprogs expect it */ +if ((flags &BPF_TRAMP_F_CALL_ORIG ) && (flags &BPF_TRAMP_F_TAIL_CALL_CTX )) + bpf_frame_size +=SZL ; +/* Padding to align stack frame, if any */ bpf_frame_size =round_up (bpf_frame_size,SZL *2 );
Patch#2 is not complete without this change. Please fold this patch into patch#2 itself.
- Hari
Hari Bathini
maintainerRe: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines
Quoted 69 lines · Hari Bathini
Quoted 65 lines · On 14/01/26 5:14 pm, [email protected] wrote:
From: Abhishek Dubey <[email protected]>
The trampoline mechanism sets up its own stack frame and an additional dummy frame. We need to have additional JIT instructions handling tailcall dereferencing in the trampoline's context.
We don't add the two stack frames pointed above, rather add space for
tail_call_infoat bottom in trampoline frame for ppc64. This makes the trampoline's frame consistent with layout of all other frames wrttail_call_infooffset.Signed-off-by Abhishek Dubey <[email protected]>---
arch/powerpc/net/bpf_jit_comp.c| 83 ++++++++++++++++++++++-----------1 file changed, 56 insertions(+), 27 deletions(-)arch/powerpc/net/bpf_jit_comp.c +2
-14@@ -606,33 +606,58 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context return 0; } -static void bpf_trampoline_setup_tail_call_cnt (u32 *image,struct codegen_context *ctx, - int func_frame_offset, int r4_off) -{ - if (IS_ENABLED(CONFIG_PPC64)) { - /* See bpf_jit_stack_tailcallinfo_offset() */ - int tailcallcnt_offset = 7 * 8; - - EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset)); - EMIT(PPC_RAW_STL(_R3, _R1, -tailcallcnt_offset)); - } else { - /* See bpf_jit_stack_offsetof() and BPF_PPC_TC */ - EMIT(PPC_RAW_LL(_R4, _R1, r4_off)); - } -} +/* +* Refer the label 'Generated stack layout' in this file for actual stack
+ * layout during trampoline invocation.
+ *
+ * Refer__arch_prepare_bpf_trampoline()for stack component details.
+ *
+ * The tailcall count/reference is present in caller's stack frame. Its required
+ * to copy the content oftail_call_infobefore calling the actual function
+ * to which the trampoline is attached.
+ *
+ */
-static void bpf_trampoline_restore_tail_call_cnt(u32 *image, structcodegen_context*ctx,
- intfunc_frame_offset, intr4_off) +static void bpf_trampoline_setup_tail_call_info(u32 *image, structcodegen_context*ctx,
+ intfunc_frame_offset,
+ intbpf_dummy_frame_size, intr4_off) {
if (IS_ENABLED(CONFIG_PPC64)) {
/* Seebpf_jit_stack_tailcallinfo_offset()*/
Quoted 54 lines
Quoted 2 lines
- int
tailcallcnt_offset= 7 * 8;
+ inttailcallinfo_offset=BPF_PPC_TAILCALL;This offset update should have been part of patch#1
Quoted 49 lines
+ /*
+ *func_frame_offset= ...(1) + *bpf_dummy_frame_size+trampoline_frame_size+ */
+EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset));
+EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset));
+
+ /*
+ * Setting thetail_call_infoin trampoline's frame + * depending on if previous frame had value or reference. + */
+EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT));
+PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx)));
+ /*
+ * From ...(1) above:
+ *trampoline_frame_bottom= ...(2) + *func_frame_offset-bpf_dummy_frame_size
+ *
+ * Using ...(2) derived above:
+ *trampoline_tail_call_info_offset= ...(3) + *trampoline_frame_bottom-tailcallinfo_offset+ *
+ * From ...(3):
+ * Usetrampoline_tail_call_info_offsetto write reference of main's
+ *tail_call_infoin trampoline frame.
+ */
+ EMIT(PPC_RAW_STL(_R3,_R1, (func_frame_offset-bpf_dummy_frame_size)
+ -tailcallinfo_offset));
-EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
-EMIT(PPC_RAW_STL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
} else {
/* Seebpf_jit_stack_offsetof()andBPF_PPC_TC*/ -EMIT(PPC_RAW_STL(_R4, _R1, r4_off));
+EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
}
}arch/powerpc/net/bpf_jit_comp.c +0
-0@@ -720,6 +745,7 @@ static int __arch_prepare_bpf_trampoline(struct
bpf_tramp_image*im, void*rw_im
* LR save area [ r0 save (64-bit) ] | header * [ r0 save (32-bit) ] |
* dummy frame for unwind [ back chain 1 ] --+ * [ tail_call_info ] non optional - 64-bit powerpc
Also, why non-optional? This can be optional on BPF_TRAMP_F_CALL_ORIG and BPF_TRAMP_F_TAIL_CALL_CTX flags?
Quoted 44 lines
Quoted 12 lines
* [ padding ] align stack frame * r4_off [ r4 (tailcallcnt) ] optional - 32-bit powerpc * alt_lr_off [ real lr (ool stub)] optional - actual lr
arch/powerpc/net/bpf_jit_comp.c +0
-0@@ -801,8 +827,14 @@ static int __arch_prepare_bpf_trampoline(struct
bpf_tramp_image*im, void*rw_im
}
}
- /* Padding to align stack frame, if any */ -bpf_frame_size=round_up(bpf_frame_size, SZL * 2); + if (!(bpf_frame_size % (2 * SZL))) {
+ /* Stack is 16-byte aligned */Quoted 8 lines
+ /* Room for padding followed by 64-bit
tail_call_info*/
+bpf_frame_size+= SZL +BPF_PPC_TAILCALL;
+ } else {
+ /* Room for 64-bittail_call_info*/
+bpf_frame_size+=BPF_PPC_TAILCALL;
+ }
/* Dummy frame size for proper unwind - includes 64-bytes red zone for 64-bit powerpc */bpf_dummy_frame_size=STACK_FRAME_MIN_SIZE+ 64;This change assumes the size is at least 8-byte aligned which is true today but better skip that assumption by not touching the padding part. The above hunk could simply be:
arch/powerpc/net/bpf_jit_comp.c +0
-0@@ -795,6 +795,10 @@ static int __arch_prepare_bpf_trampoline(struct
bpf_tramp_image*im, void*rw_im
}
}+ /* Save tailcall count pointer at the same offset on the stack where subprogs expect it */
+ if ((flags &BPF_TRAMP_F_CALL_ORIG) && (flags &BPF_TRAMP_F_TAIL_CALL_CTX))
+bpf_frame_size+= SZL;
+
/* Padding to align stack frame, if any */bpf_frame_size=round_up(bpf_frame_size, SZL * 2);
Quoted 3 lines
Patch#2 is not complete without this change. Please fold this patch into patch#2 itself.
- Hari
PATCH v2 4/6 powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT
In reply to [email protected]From: Abhishek Dubey <[email protected]>
This function is used by bpf_throw() to unwind the stack until frame of exception-boundary during BPF exception handling.
This function is necessary to support BPF exceptions on PowerPC.
--- arch/powerpc/net/bpf_jit_comp64.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
arch/powerpc/net/bpf_jit_comp64.c +28 -0
@@ -247,6 +247,34 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)bpf_jit_build_fentry_stubs (image, ctx); } +void arch_bpf_stack_walk (bool (*consume_fn)(void *,u64 ,u64 ,u64 ),void *cookie) +{ +// callback processing always in current context +unsigned long fp =current_stack_frame (); + +for (;;) { +unsigned long *frame = (unsigned long *) fp; +unsigned long ip; + +if (!validate_sp (fp, current)) +return ; + + ip = frame[STACK_FRAME_LR_SAVE ]; +if (!ip) +break ; + +/* +* consume_fn common code expects stack pointer(sp) in third +* argument. There is no sp in ppc64, rather pass frame +* pointer. +*/ +if (ip && !consume_fn (cookie, ip, fp, fp)) +break ; + + fp = frame[0 ]; + } +} +int bpf_jit_emit_func_call_rel (u32 *image,u32 *fimage,struct codegen_context *ctx,u64 func) {unsigned long func_addr = func ?ppc_function_entry ((void *)func) :0 ;
Signature · 1 line
2.48.1
Christophe Leroy (CS GROUP)
maintainerRe: [PATCH v2 4/6] powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT
Le 14/01/2026 à 12:44, [email protected] a écrit :
Quoted 40 lines
From: Abhishek Dubey <[email protected]>
This function is used by
bpf_throw()to unwind the stack until frame of exception-boundary during BPF exception handling.This function is necessary to support BPF exceptions on PowerPC.
Signed-off-by Abhishek Dubey <[email protected]>--- arch/powerpc/net/bpf_jit_comp64.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)arch/powerpc/net/bpf_jit_comp64.c +18
-0@@ -247,6 +247,34 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)bpf_jit_build_fentry_stubs (image, ctx); } +void arch_bpf_stack_walk (bool (*consume_fn)(void *,u64 ,u64 ,u64 ),void *cookie) +{ +// callback processing always in current context +unsigned long fp =current_stack_frame (); + +for (;;) { +unsigned long *frame = (unsigned long *) fp; +unsigned long ip; + +if (!validate_sp (fp, current)) +return ; + + ip = frame[STACK_FRAME_LR_SAVE ]; +if (!ip) +break ; + +/* +* consume_fn common code expects stack pointer(sp) in third
arch/powerpc/net/bpf_jit_comp64.c void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx) · line 277
Quoted 10 lines
arch/powerpc/net/bpf_jit_comp64.c +10
-0+* argument. There is no sp in ppc64, rather pass frame +* pointer. +*/ +if (ip && !consume_fn (cookie, ip, fp, fp)) +break ; + + fp = frame[0 ]; + } +} +
This fonction looks very close to arch_stack_walk(). Would it be possible to refactor and have a common part used by both functions, like ARM64 for instance ?
Quoted 3 lines
arch/powerpc/net/bpf_jit_comp64.c +0
-0int bpf_jit_emit_func_call_rel (u32 *image,u32 *fimage,struct codegen_context *ctx,u64 func) {unsigned long func_addr = func ?ppc_function_entry ((void *)func) :0 ;
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hari Bathini
maintainerRe: [PATCH v2 4/6] powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT
Quoted 56 lines · adubey
Quoted 47 lines · Christophe Leroy (CS GROUP)
Le 14/01/2026 à 12:44, [email protected] a écrit :
Quoted 46 lines
From: Abhishek Dubey <[email protected]>
This function is used by
bpf_throw()to unwind the stack until frame of exception-boundary during BPF exception handling.This function is necessary to support BPF exceptions on PowerPC.
Signed-off-by Abhishek Dubey <[email protected]>---
arch/powerpc/net/bpf_jit_comp64.c| 28 ++++++++++++++++++++++++++++1 file changed, 28 insertions(+)arch/powerpc/net/bpf_jit_comp64.c +23
-0@@ -247,6 +247,34 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx) bpf_jit_build_fentry_stubs(image, ctx); } +void arch_bpf_stack_walk(bool (*consume_fn)(void *, u64, u64, u64), void *cookie) +{ + // callback processing always in current context + unsigned long fp = current_stack_frame(); + + for (;;) { + unsigned long *frame = (unsigned long *) fp; + unsigned long ip; + + if (!validate_sp(fp, current)) + return; + + ip = frame[STACK_FRAME_LR_SAVE]; + if (!ip) + break; + + /* + * consume_fn common code expects stack pointer(sp) in third + * argument. There is no sp in ppc64, rather pass frame + * pointer. + */ + if (ip && !consume_fn(cookie, ip, fp, fp)) + break; +Quoted 8 lines
arch/powerpc/net/bpf_jit_comp64.c void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx) · line 277
Quoted 4 lines
arch/powerpc/net/bpf_jit_comp64.c +4
-0+ fp = frame[0]; + } +} +This fonction looks very close to
arch_stack_walk(). Would it be possible to refactor and have a common part used by both functions, like ARM64 for instance ?
Quoted 4 lines
Yes, its inspired from
arch_stack_walk().consume_entry()have different parameter count in both cases. If merged, it need additional handling to identify whichcall_backto invoke.
Also, we need to define arch-specific weak functionarch_bpf_stack_walk(), so renaming ofarch_stack_walkis needed on merge. Stack walker logic with "bpf" name might be confusing when used at other places. Thoughts?
Not sure what you mean by renaming of arch_stack_walk is needed on merge but refactoring does not have to change API signature or any common code for that matter..
- Hari
PATCH v2 5/6 powerpc64/bpf: Support exceptions
In reply to [email protected]From: Abhishek Dubey <[email protected]>
The modified prologue/epilogue generation code now enables exception-callback to use the stack frame of the program marked as exception boundary, where callee saved registers are stored.
As per ppc64 ABIv2 documentation[1], r14-r31 are callee saved registers. BPF programs on ppc64 already saves r26-r31 registers. Saving the remaining set of callee saved registers(r14-r25) is handled in the next patch.
[1] https://ftp.rtems.org/pub/rtems/people/sebh/ABI64BitOpenPOWERv1.1_16July2015_pub.pdf
--- arch/powerpc/net/bpf_jit.h | 2 ++ arch/powerpc/net/bpf_jit_comp.c | 7 ++++ arch/powerpc/net/bpf_jit_comp64.c | 53 +++++++++++++++++++++---------- 3 files changed, 45 insertions(+), 17 deletions(-)
arch/powerpc/net/bpf_jit.h +2 -0
@@ -179,6 +179,8 @@ struct codegen_context {u64 arena_vm_start;u64 user_vm_start;bool is_subprog; +bool exception_boundary; +bool exception_cb; };#define bpf_to_ppc (r) (ctx->b2p[r])
arch/powerpc/net/bpf_jit_comp.c +7 -0
@@ -207,6 +207,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) cgctx.arena_vm_start =bpf_arena_get_kern_vm_start (fp->aux->arena); cgctx.user_vm_start =bpf_arena_get_user_vm_start (fp->aux->arena); cgctx.is_subprog =bpf_is_subprog (fp); + cgctx.exception_boundary = fp->aux->exception_boundary; + cgctx.exception_cb = fp->aux->exception_cb;/* Scouting faux-generate pass 0 */ if (bpf_jit_build_body (fp,NULL ,NULL , &cgctx, addrs,0 , false)) {
@@ -436,6 +438,11 @@ void bpf_jit_free(struct bpf_prog *fp)bpf_prog_unlock_free (fp); } +bool bpf_jit_supports_exceptions (void ) +{ +return IS_ENABLED (CONFIG_PPC64 ); +} +bool bpf_jit_supports_subprog_tailcalls (void ) {return IS_ENABLED (CONFIG_PPC64 );
arch/powerpc/net/bpf_jit_comp64.c +36 -17
83 lines not on this page · raw message
Signature · 1 line
2.48.1
Hari Bathini
maintainerRe: [PATCH v2 5/6] powerpc64/bpf: Support exceptions
Quoted 132 lines · On 14/01/26 5:14 pm, [email protected] wrote:
From: Abhishek Dubey <[email protected]>
The modified prologue/epilogue generation code now enables exception-callback to use the stack frame of the program marked as exception boundary, where callee saved registers are stored.
As per ppc64 ABIv2 documentation[1], r14-r31 are callee saved registers. BPF programs on ppc64 already saves r26-r31 registers. Saving the remaining set of callee saved registers(r14-r25) is handled in the next patch.
[1] https://ftp.rtems.org/pub/rtems/people/sebh/ABI64BitOpenPOWERv1.1_16July2015_pub.pdf
Signed-off-by Abhishek Dubey <[email protected]>--- arch/powerpc/net/bpf_jit.h | 2 ++ arch/powerpc/net/bpf_jit_comp.c | 7 ++++ arch/powerpc/net/bpf_jit_comp64.c | 53 +++++++++++++++++++++---------- 3 files changed, 45 insertions(+), 17 deletions(-)arch/powerpc/net/bpf_jit.h +2
-0@@ -179,6 +179,8 @@ struct codegen_context {u64 arena_vm_start;u64 user_vm_start;bool is_subprog; +bool exception_boundary; +bool exception_cb; };#define bpf_to_ppc (r) (ctx->b2p[r])arch/powerpc/net/bpf_jit_comp.c +7
-0@@ -207,6 +207,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) cgctx.arena_vm_start =bpf_arena_get_kern_vm_start (fp->aux->arena); cgctx.user_vm_start =bpf_arena_get_user_vm_start (fp->aux->arena); cgctx.is_subprog =bpf_is_subprog (fp); + cgctx.exception_boundary = fp->aux->exception_boundary; + cgctx.exception_cb = fp->aux->exception_cb;/* Scouting faux-generate pass 0 */ if (bpf_jit_build_body (fp,NULL ,NULL , &cgctx, addrs,0 , false)) {@@ -436,6 +438,11 @@ void bpf_jit_free(struct bpf_prog *fp)bpf_prog_unlock_free (fp); } +bool bpf_jit_supports_exceptions (void ) +{ +return IS_ENABLED (CONFIG_PPC64 ); +} +bool bpf_jit_supports_subprog_tailcalls (void ) {return IS_ENABLED (CONFIG_PPC64 );arch/powerpc/net/bpf_jit_comp64.c +27
-16@@ -89,7 +89,9 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)* - the bpf program uses its stack area * The latter condition is deduced from the usage of BPF_REG_FP */ -return ctx->seen &SEEN_FUNC ||bpf_is_seen_register (ctx,bpf_to_ppc (BPF_REG_FP )); +return ctx->seen &SEEN_FUNC || +bpf_is_seen_register (ctx,bpf_to_ppc (BPF_REG_FP )) || + ctx->exception_cb; }/* @@ -190,23 +192,32 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)EMIT (PPC_RAW_STDU (_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size))); } -/* -* Back up non-volatile regs -- BPF registers 6-10 -* If we haven't created our own stack frame, we save these -* in the protected zone below the previous stack frame -*/ -for (i =BPF_REG_6 ; i <=BPF_REG_10 ; i++) -if (bpf_is_seen_register (ctx,bpf_to_ppc (i))) -EMIT (PPC_RAW_STD (bpf_to_ppc (i), _R1,bpf_jit_stack_offsetof (ctx,bpf_to_ppc (i)))); +if (!ctx->exception_cb) { +/* +* Back up non-volatile regs -- BPF registers 6-10 +* If we haven't created our own stack frame, we save these +* in the protected zone below the previous stack frame +*/ +for (i =BPF_REG_6 ; i <=BPF_REG_10 ; i++) +if (ctx->exception_boundary ||bpf_is_seen_register (ctx,bpf_to_ppc (i))) +EMIT (PPC_RAW_STD (bpf_to_ppc (i), _R1, +bpf_jit_stack_offsetof (ctx,bpf_to_ppc (i)))); -if (ctx->arena_vm_start) -EMIT (PPC_RAW_STD (bpf_to_ppc (ARENA_VM_START ), _R1, +if (ctx->exception_boundary || ctx->arena_vm_start) +EMIT (PPC_RAW_STD (bpf_to_ppc (ARENA_VM_START ), _R1,bpf_jit_stack_offsetof (ctx,bpf_to_ppc (ARENA_VM_START )))); -/* Setup frame pointer to point to the bpf stack area */ -if (bpf_is_seen_register (ctx,bpf_to_ppc (BPF_REG_FP ))) -EMIT (PPC_RAW_ADDI (bpf_to_ppc (BPF_REG_FP ), _R1, +/* Setup frame pointer to point to the bpf stack area */ +if (bpf_is_seen_register (ctx,bpf_to_ppc (BPF_REG_FP ))) +EMIT (PPC_RAW_ADDI (bpf_to_ppc (BPF_REG_FP ), _R1,STACK_FRAME_MIN_SIZE + ctx->stack_size)); + }else { +/* +* Exception callback receives Frame Pointer of main +* program as third arg +*/ +EMIT (PPC_RAW_MR (_R1, _R5)); + }if (ctx->arena_vm_start)PPC_LI64 (bpf_to_ppc (ARENA_VM_START ), ctx->arena_vm_start);@@ -218,17 +229,25 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx/* Restore NVRs */ for (i =BPF_REG_6 ; i <=BPF_REG_10 ; i++) -if (bpf_is_seen_register (ctx,bpf_to_ppc (i))) +if (ctx->exception_cb ||bpf_is_seen_register (ctx,bpf_to_ppc (i)))EMIT (PPC_RAW_LD (bpf_to_ppc (i), _R1,bpf_jit_stack_offsetof (ctx,bpf_to_ppc (i)))); -if (ctx->arena_vm_start) +if (ctx->exception_cb || ctx->arena_vm_start)EMIT (PPC_RAW_LD (bpf_to_ppc (ARENA_VM_START ), _R1,bpf_jit_stack_offsetof (ctx,bpf_to_ppc (ARENA_VM_START ))));
arch/powerpc/net/bpf_jit_comp64.c static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx · line 247
Quoted 10 lines
arch/powerpc/net/bpf_jit_comp64.c +8
-0+if (ctx->exception_cb) { +/* +* LR value from boundary-frame is received as second parameter +* in exception callback. +*/ +EMIT (PPC_RAW_MTLR (_R4)); + } +
No. Both second and third parameter of exception_cb() are stack pointer (or frame pointer, if you prefer that) and not LR.
The above hunk is wrong. It still worked because of mtlr instruction below that restored LR from PPC_LR_STKOFF offset on the stack. Please drop the above hunk.
Quoted 8 lines
arch/powerpc/net/bpf_jit_comp64.c +1
-1/* Tear down our stack frame */ if (bpf_has_stack_frame (ctx)) {EMIT (PPC_RAW_ADDI (_R1, _R1,BPF_PPC_STACKFRAME + ctx->stack_size)); -if (ctx->seen &SEEN_FUNC ) { +if (ctx->seen &SEEN_FUNC || ctx->exception_cb) {EMIT (PPC_RAW_LD (_R0, _R1,PPC_LR_STKOFF ));EMIT (PPC_RAW_MTLR (_R0));}
- Hari
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hari Bathini
maintainerRe: [PATCH v2 5/6] powerpc64/bpf: Support exceptions
Quoted 149 lines · adubey
Quoted 148 lines · adubey
Quoted 147 lines · Hari Bathini
Quoted 145 lines · On 14/01/26 5:14 pm, [email protected] wrote:
From: Abhishek Dubey <[email protected]>
The modified prologue/epilogue generation code now enables exception-callback to use the stack frame of the program marked as exception boundary, where callee saved registers are stored.
As per ppc64 ABIv2 documentation[1], r14-r31 are callee saved registers. BPF programs on ppc64 already saves r26-r31 registers. Saving the remaining set of callee saved registers(r14-r25) is handled in the next patch.
[1] https://ftp.rtems.org/pub/rtems/people/sebh/ ABI64BitOpenPOWERv1.1_16July2015_pub.pdf
Signed-off-by Abhishek Dubey <[email protected]>---
arch/powerpc/net/bpf_jit.h | 2 ++ arch/powerpc/net/bpf_jit_comp.c | 7 ++++arch/powerpc/net/bpf_jit_comp64.c| 53 ++++++++++++++++++++
+----------3 files changed, 45 insertions(+), 17 deletions(-)arch/powerpc/net/bpf_jit.h +2
-0@@ -179,6 +179,8 @@ struct codegen_context { u64 arena_vm_start; u64 user_vm_start; bool is_subprog; + bool exception_boundary; + bool exception_cb; }; #define bpf_to_ppc(r) (ctx->b2p[r])arch/powerpc/net/bpf_jit_comp.c +0
-0@@ -207,6 +207,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux-
Quoted 1 line
arena);
cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux-
Quoted 1 line
arena);
cgctx.is_subprog =
bpf_is_subprog(fp);
+ cgctx.exception_boundary =fp->aux->exception_boundary; + cgctx.exception_cb =fp->aux->exception_cb;
/* Scouting faux-generate pass 0 */
if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {arch/powerpc/net/bpf_jit_comp.c +4
-0@@ -436,6 +438,11 @@ void bpf_jit_free(struct bpf_prog *fp) bpf_prog_unlock_free(fp); } +bool bpf_jit_supports_exceptions(void) +{ + return IS_ENABLED(CONFIG_PPC64); +} + bool bpf_jit_supports_subprog_tailcalls(void) { return IS_ENABLED(CONFIG_PPC64);arch/powerpc/net/bpf_jit_comp64.c +3
-1@@ -89,7 +89,9 @@ static inline bool bpf_has_stack_frame(structcodegen_context *ctx) * - the bpf program uses its stack area * The latter condition is deduced from the usage of BPF_REG_FP */ - return ctx->seen & SEEN_FUNC || bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)); + return ctx->seen & SEEN_FUNC || + bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)) || + ctx->exception_cb; }/*
arch/powerpc/net/bpf_jit_comp64.c +0
-0@@ -190,23 +192,32 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)EMIT(PPC_RAW_STDU(_R1,
_R1, -(BPF_PPC_STACKFRAME + ctx-Quoted 1 line
stack_size)));}
- /*
- * Back up non-volatile regs -- BPF registers 6-10
- * If we haven't created our own stack frame, we save these - * in the protected zone below the previous stack frame
- */
- for (i =BPF_REG_6; i <=BPF_REG_10; i++)
- if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
-EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
+ if (!ctx->exception_cb) {
+ /*
+ * Back up non-volatile regs -- BPF registers 6-10
+ * If we haven't created our own stack frame, we save these + * in the protected zone below the previous stack frame + */
+ for (i =BPF_REG_6; i <=BPF_REG_10; i++)
+ if (ctx->exception_boundary||bpf_is_seen_register(ctx, bpf_to_ppc(i)))
+ EMIT(PPC_RAW_STD(bpf_to_ppc(i),_R1,
+bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i)))); - if (ctx->arena_vm_start)
- EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START),_R1,
+ if (ctx->exception_boundary||ctx->arena_vm_start)
+ EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START),_R1,bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START)))); - /* Setup frame pointer to point to the bpf stack area */
- if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
- EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP),_R1,
+ /* Setup frame pointer to point to the bpf stack area */ + if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
+EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1, STACK_FRAME_MIN_SIZE + ctx->stack_size));
+ } else {
+ /*
+ * Exception callback receives Frame Pointer of main
+ * program as third arg
+ */
+EMIT(PPC_RAW_MR(_R1, _R5));
+ }
if (ctx->arena_vm_start)PPC_LI64(bpf_to_ppc(ARENA_VM_START), ctx->arena_vm_start);arch/powerpc/net/bpf_jit_comp64.c +5
-1@@ -218,17 +229,25 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx /* Restore NVRs */ for (i = BPF_REG_6; i <= BPF_REG_10; i++) - if (bpf_is_seen_register(ctx, bpf_to_ppc(i))) + if (ctx->exception_cb || bpf_is_seen_register(ctx,bpf_to_ppc (i))) EMIT(PPC_RAW_LD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i)))); - if (ctx->arena_vm_start) + if (ctx->exception_cb || ctx->arena_vm_start) EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START)))); + if (ctx->exception_cb) { + /* +* LR value from boundary-frame is received as second parameter
+ * in exception callback.
+ */
+ EMIT(PPC_RAW_MTLR(_R4));
+ }
+
Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
No. Both second and third parameter of
exception_cb()are stack pointer (or frame pointer, if you prefer that) and not LR.
The above hunk is wrong. It still worked because of mtlr instruction below that restored LR fromPPC_LR_STKOFFoffset on the stack. Please drop the above hunk.Okay, but from patch 3/6:
My mistake, its patch 4/6.
Please note the consume_fn() is not exception_cb(). It is bpf_stack_walker() that helps with the stack walk to be precise. You can find the exception_cb() call in bpf_throw() kfunc as:
ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
Quoted 11 lines
Quoted 11 lines
ip = frame[STACK_FRAME_LR_SAVE];
...
if (ip && !consume_fn(cookie, ip, fp, fp))
we send cookie as first, LR as second parameter and fp as third & fourth both. Default callback expects 4 parameter, so we copy fp in both.Quoted 4 lines
Quoted 4 lines
/* Tear down our stack frame */
if (bpf_has_stack_frame(ctx)) { EMIT(PPC_RAW_ADDI(_R1,_R1,BPF_PPC_STACKFRAME+ ctx-
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
stack_size));- if (
ctx->seen&SEEN_FUNC) {
+ if (ctx->seen&SEEN_FUNC||ctx->exception_cb) {EMIT(PPC_RAW_LD(_R0, _R1, PPC_LR_STKOFF)); EMIT(PPC_RAW_MTLR(_R0));
}
- Hari
PATCH v2 6/6 powerpc64/bpf: Additional NVR handling for bpf_throw
In reply to [email protected]From: Abhishek Dubey <[email protected]>
The bpf_throw() function never returns, if it has clobbered any callee-saved register, those will remain clobbered. The prologue must take care of saving all callee-saved registers in the frame of exception boundary program. Later these additional non volatile registers R14-R25 along with other NVRs are restored back in the epilogue of exception callback.
To achieve above objective the frame size is determined dynamically to accommodate additional non volatile registers in exception boundary's frame. For non-exception boundary program, the frame size remains optimal. The additional instructions to save & restore r14-r25 registers are emitted only during exception boundary and exception callback respectively.
--- arch/powerpc/net/bpf_jit_comp64.c | 70 +++++++++++++++++++++++++++---- 1 file changed, 63 insertions(+), 7 deletions(-)
arch/powerpc/net/bpf_jit_comp64.c +63 -7
@@ -32,21 +32,37 @@* * [ prev sp ] <------------- * [ tail_call_info ] 8 | -* [ nv gpr save area ] 6*8 | +* [ nv gpr save area ] 6*8 + (12*8) | * [ local_tmp_var ] 24 | * fp (r31) --> [ ebpf stack space ] upto 512 | * [ frame header ] 32/112 | * sp (r1) ---> [ stack pointer ] -------------- +* +* Additional (12*8) in 'nv gpr save area' only in case of +* exception boundary. */ /* for bpf JIT code internal usage */ #define BPF_PPC_STACK_LOCALS 24 +/* +* for additional non volatile registers(r14-r25) to be saved +* at exception boundary +*/ +#define BPF_PPC_EXC_STACK_SAVE (12*8) +/* stack frame excluding BPF stack, ensure this is quadword aligned */ #define BPF_PPC_STACKFRAME (STACK_FRAME_MIN_SIZE + \ BPF_PPC_STACK_LOCALS + \ BPF_PPC_STACK_SAVE + \ BPF_PPC_TAILCALL) +/* +* same as BPF_PPC_STACKFRAME with save area for additional +* non volatile registers saved at exception boundary. +* This is quad-word aligned. +*/ +#define BPF_PPC_EXC_STACKFRAME (BPF_PPC_STACKFRAME + BPF_PPC_EXC_STACK_SAVE) +/* BPF register usage */ #define TMP_REG_1 (MAX_BPF_JIT_REG + 0)#define TMP_REG_2 (MAX_BPF_JIT_REG + 1)
@@ -103,9 +119,12 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)* [ ... ] | * sp (r1) ---> [ stack pointer ] -------------- * [ tail_call_info ] 8 -* [ nv gpr save area ] 6*8 +* [ nv gpr save area ] 6*8 + (12*8) * [ local_tmp_var ] 24 * [ unused red zone ] 224 +* +* Additional (12*8) in 'nv gpr save area' only in case of +* exception boundary. */ static int bpf_jit_stack_local (struct codegen_context *ctx) {
@@ -114,7 +133,11 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)return STACK_FRAME_MIN_SIZE + ctx->stack_size; }else {/* Stack layout 2 */ -return -(BPF_PPC_TAILCALL +BPF_PPC_STACK_SAVE +BPF_PPC_STACK_LOCALS ); +return -(BPF_PPC_TAILCALL + +BPF_PPC_STACK_SAVE + + (ctx->exception_boundary || ctx->exception_cb ? +BPF_PPC_EXC_STACK_SAVE :0 ) + +BPF_PPC_STACK_LOCALS ); } }
@@ -125,9 +148,19 @@ int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)static int bpf_jit_stack_offsetof (struct codegen_context *ctx,int reg) { -if (reg >=BPF_PPC_NVR_MIN && reg <32 ) +int min_valid_nvreg =BPF_PPC_NVR_MIN ; +/* Default frame size for all cases except exception boundary */ +int frame_nvr_size =BPF_PPC_STACKFRAME ; + +/* Consider all nv regs for handling exceptions */ +if (ctx->exception_boundary || ctx->exception_cb) { + min_valid_nvreg = _R14; + frame_nvr_size =BPF_PPC_EXC_STACKFRAME ; + } + +if (reg >= min_valid_nvreg && reg <32 )return (bpf_has_stack_frame (ctx) ? - (BPF_PPC_STACKFRAME + ctx->stack_size) :0 ) + (frame_nvr_size + ctx->stack_size) :0 ) - (8 * (32 - reg)) -BPF_PPC_TAILCALL ;pr_err ("BPF JIT is asking about unknown registers" );
@@ -189,7 +222,20 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)EMIT (PPC_RAW_STD (_R0, _R1,PPC_LR_STKOFF )); } -EMIT (PPC_RAW_STDU (_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size))); +int stack_expand = ctx->exception_boundary || ctx->exception_cb ? +BPF_PPC_EXC_STACKFRAME :BPF_PPC_STACKFRAME ; +EMIT (PPC_RAW_STDU (_R1, _R1, -(stack_expand + ctx->stack_size))); + } + +/* +* Program acting as exception boundary pushes R14..R25 in addition to +* BPF callee-saved non volatile registers. Exception callback uses +* the boundary program's stack frame, recover additionally saved +* registers in epilogue of exception callback. +*/ +if (ctx->exception_boundary) { +for (i = _R14; i <= _R25; i++) +EMIT (PPC_RAW_STD (i, _R1,bpf_jit_stack_offsetof (ctx, i))); }if (!ctx->exception_cb) {
@@ -237,6 +283,13 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));if (ctx->exception_cb) { +/* +* Recover additionally saved non volatile registers from stack +* frame of exception boundary program. +*/ +for (i = _R14; i <= _R25; i++) +EMIT (PPC_RAW_LD (i, _R1,bpf_jit_stack_offsetof (ctx, i))); +/* * LR value from boundary-frame is received as second parameter * in exception callback.
@@ -246,7 +299,10 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx/* Tear down our stack frame */ if (bpf_has_stack_frame (ctx)) { -EMIT (PPC_RAW_ADDI (_R1, _R1,BPF_PPC_STACKFRAME + ctx->stack_size)); +int stack_shrink = ctx->exception_cb || ctx->exception_boundary ? +BPF_PPC_EXC_STACKFRAME :BPF_PPC_STACKFRAME ; +EMIT (PPC_RAW_ADDI (_R1, _R1, stack_shrink + ctx->stack_size)); +if (ctx->seen &SEEN_FUNC || ctx->exception_cb) {EMIT (PPC_RAW_LD (_R0, _R1,PPC_LR_STKOFF ));EMIT (PPC_RAW_MTLR (_R0));
Signature · 1 line
2.48.1
[email protected]
AI reviewRe: [PATCH v2 6/6] powerpc64/bpf: Additional NVR handling for bpf_throw
Does this need to account for BPF_PPC_EXC_STACK_SAVE when exception_boundary or exception_cb is true?
Quoted 4 lines
arch/powerpc/net/bpf_jit_comp64.c +0
-0
[ ... ]
Quoted 15 lines
arch/powerpc/net/bpf_jit_comp64.c +4
-1@@ -103,9 +119,12 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)* [ ... ] | * sp (r1) ---> [ stack pointer ] -------------- * [ tail_call_info ] 8 -* [ nv gpr save area ] 6*8 +* [ nv gpr save area ] 6*8 + (12*8) * [ local_tmp_var ] 24 * [ unused red zone ] 224 +* +* Additional (12*8) in 'nv gpr save area' only in case of +* exception boundary. */ static int bpf_jit_stack_local (struct codegen_context *ctx) {
arch/powerpc/net/bpf_jit_comp64.c static int bpf_jit_stack_local(struct codegen_context *ctx) · line 133
Quoted 1 line
arch/powerpc/net/bpf_jit_comp64.c +0
-0@@ -114,7 +133,11 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)return STACK_FRAME_MIN_SIZE + ctx->stack_size;
Does this need to account for BPF_PPC_EXC_STACK_SAVE when exception_boundary or exception_cb is true?
In the else branch below, the exception case is handled, but the bpf_has_stack_frame() case returns the same value regardless of exception_boundary. However, bpf_jit_build_prologue() creates a larger frame using BPF_PPC_EXC_STACKFRAME when exception_boundary is true.
This inconsistency appears to cause bpf_jit_stack_tailcallinfo_offset() to return an incorrect offset when both bpf_has_stack_frame() returns true AND exception_boundary is true. The offset would be off by 96 bytes (BPF_PPC_EXC_STACK_SAVE).
For example, if a program uses bpf_throw (setting exception_boundary=true) and also calls helper functions (setting SEEN_FUNC, making bpf_has_stack_frame() return true), and uses tail calls, the tail call handling in bpf_jit_emit_tail_call() would access the wrong stack location.
Quoted 10 lines
arch/powerpc/net/bpf_jit_comp64.c +5
-1}else {/* Stack layout 2 */ -return -(BPF_PPC_TAILCALL +BPF_PPC_STACK_SAVE +BPF_PPC_STACK_LOCALS ); +return -(BPF_PPC_TAILCALL + +BPF_PPC_STACK_SAVE + + (ctx->exception_boundary || ctx->exception_cb ? +BPF_PPC_EXC_STACK_SAVE :0 ) + +BPF_PPC_STACK_LOCALS ); } }
[ ... ]
arch/powerpc/net/bpf_jit_comp64.c void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) · line 227
Quoted 8 lines
arch/powerpc/net/bpf_jit_comp64.c +3
-1@@ -189,7 +222,20 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)EMIT (PPC_RAW_STD (_R0, _R1,PPC_LR_STKOFF )); } -EMIT (PPC_RAW_STDU (_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size))); +int stack_expand = ctx->exception_boundary || ctx->exception_cb ? +BPF_PPC_EXC_STACKFRAME :BPF_PPC_STACKFRAME ; +EMIT (PPC_RAW_STDU (_R1, _R1, -(stack_expand + ctx->stack_size)));
The stack frame expansion here uses BPF_PPC_EXC_STACKFRAME for exception cases, but bpf_jit_stack_local() in the bpf_has_stack_frame() branch does not account for this larger frame size.
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/20993216190
Hari Bathini
maintainerRe: [PATCH v2 6/6] powerpc64/bpf: Additional NVR handling for bpf_throw
In reply to [email protected]Quoted 125 lines · On 14/01/26 5:14 pm, [email protected] wrote:
From: Abhishek Dubey <[email protected]>
The
bpf_throw()function never returns, if it has clobbered any callee-saved register, those will remain clobbered. The prologue must take care of saving all callee-saved registers in the frame of exception boundary program. Later these additional non volatile registers R14-R25 along with other NVRs are restored back in the epilogue of exception callback.To achieve above objective the frame size is determined dynamically to accommodate additional non volatile registers in exception boundary's frame. For non-exception boundary program, the frame size remains optimal. The additional instructions to save & restore r14-r25 registers are emitted only during exception boundary and exception callback respectively.
Signed-off-by Abhishek Dubey <[email protected]>--- arch/powerpc/net/bpf_jit_comp64.c | 70 +++++++++++++++++++++++++++---- 1 file changed, 63 insertions(+), 7 deletions(-)arch/powerpc/net/bpf_jit_comp64.c +41
-6@@ -32,21 +32,37 @@* * [ prev sp ] <------------- * [ tail_call_info ] 8 | -* [ nv gpr save area ] 6*8 | +* [ nv gpr save area ] 6*8 + (12*8) | * [ local_tmp_var ] 24 | * fp (r31) --> [ ebpf stack space ] upto 512 | * [ frame header ] 32/112 | * sp (r1) ---> [ stack pointer ] -------------- +* +* Additional (12*8) in 'nv gpr save area' only in case of +* exception boundary. */ /* for bpf JIT code internal usage */ #define BPF_PPC_STACK_LOCALS 24 +/* +* for additional non volatile registers(r14-r25) to be saved +* at exception boundary +*/ +#define BPF_PPC_EXC_STACK_SAVE (12*8) +/* stack frame excluding BPF stack, ensure this is quadword aligned */ #define BPF_PPC_STACKFRAME (STACK_FRAME_MIN_SIZE + \ BPF_PPC_STACK_LOCALS + \ BPF_PPC_STACK_SAVE + \ BPF_PPC_TAILCALL) +/* +* same as BPF_PPC_STACKFRAME with save area for additional +* non volatile registers saved at exception boundary. +* This is quad-word aligned. +*/ +#define BPF_PPC_EXC_STACKFRAME (BPF_PPC_STACKFRAME + BPF_PPC_EXC_STACK_SAVE) +/* BPF register usage */ #define TMP_REG_1 (MAX_BPF_JIT_REG + 0)#define TMP_REG_2 (MAX_BPF_JIT_REG + 1)@@ -103,9 +119,12 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)* [ ... ] | * sp (r1) ---> [ stack pointer ] -------------- * [ tail_call_info ] 8 -* [ nv gpr save area ] 6*8 +* [ nv gpr save area ] 6*8 + (12*8) * [ local_tmp_var ] 24 * [ unused red zone ] 224 +* +* Additional (12*8) in 'nv gpr save area' only in case of +* exception boundary. */ static int bpf_jit_stack_local (struct codegen_context *ctx) {@@ -114,7 +133,11 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)return STACK_FRAME_MIN_SIZE + ctx->stack_size; }else {/* Stack layout 2 */ -return -(BPF_PPC_TAILCALL +BPF_PPC_STACK_SAVE +BPF_PPC_STACK_LOCALS ); +return -(BPF_PPC_TAILCALL + +BPF_PPC_STACK_SAVE + + (ctx->exception_boundary || ctx->exception_cb ? +BPF_PPC_EXC_STACK_SAVE :0 ) + +BPF_PPC_STACK_LOCALS ); } }@@ -125,9 +148,19 @@ int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)static int bpf_jit_stack_offsetof (struct codegen_context *ctx,int reg) { -if (reg >=BPF_PPC_NVR_MIN && reg <32 ) +int min_valid_nvreg =BPF_PPC_NVR_MIN ; +/* Default frame size for all cases except exception boundary */ +int frame_nvr_size =BPF_PPC_STACKFRAME ; + +/* Consider all nv regs for handling exceptions */ +if (ctx->exception_boundary || ctx->exception_cb) { + min_valid_nvreg = _R14; + frame_nvr_size =BPF_PPC_EXC_STACKFRAME ; + } + +if (reg >= min_valid_nvreg && reg <32 )return (bpf_has_stack_frame (ctx) ? - (BPF_PPC_STACKFRAME + ctx->stack_size) :0 ) + (frame_nvr_size + ctx->stack_size) :0 ) - (8 * (32 - reg)) -BPF_PPC_TAILCALL ;pr_err ("BPF JIT is asking about unknown registers" );@@ -189,7 +222,20 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)EMIT (PPC_RAW_STD (_R0, _R1,PPC_LR_STKOFF )); } -EMIT (PPC_RAW_STDU (_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size))); +int stack_expand = ctx->exception_boundary || ctx->exception_cb ? +BPF_PPC_EXC_STACKFRAME :BPF_PPC_STACKFRAME ; +EMIT (PPC_RAW_STDU (_R1, _R1, -(stack_expand + ctx->stack_size)));
[...]
arch/powerpc/net/bpf_jit_comp64.c static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx · line 305
Quoted 5 lines
arch/powerpc/net/bpf_jit_comp64.c +4
-1-EMIT (PPC_RAW_ADDI (_R1, _R1,BPF_PPC_STACKFRAME + ctx->stack_size)); +int stack_shrink = ctx->exception_cb || ctx->exception_boundary ? +BPF_PPC_EXC_STACKFRAME :BPF_PPC_STACKFRAME ; +EMIT (PPC_RAW_ADDI (_R1, _R1, stack_shrink + ctx->stack_size)); +
An inline helper bpf_jit_stack_size() defined to return the stack size in both prologue and epilogue while setting up and tearing down the stack should be more elegant.
Also, IIUC, the JIT code to handle tailcall info is irrelevant for all subprogs of a BPF program with seen_exception. JIT code in the prologue for tailcall count handling can be skipped for exception_cb at least?
- Hari
Christophe Leroy (CS GROUP)
maintainerLe 14/01/2026 à 12:44, [email protected] a écrit :
Quoted 7 lines
From: Abhishek Dubey <[email protected]>
This patch series enables support for two BPF JIT features on powerpc64. The first three patches target support for tail calls with subprogram combinations. The first patch supports realignment of
tail_call_cntoffset in stack frame. Implementation details are provided in the commit messages.
Did you consider doing it also for powerpc32 ?
Christophe