powerpc/bpf: address missing verifier selftest coverage
Subsystems (6)
v9 cover powerpc/bpf: address missing verifier selftest coverage
From: Abhishek Dubey <[email protected]>
The verifier selftest validates JITed instructions by matching expected disassembly output. The first two patches fix issues in powerpc instruction disassembly that were causing test flow failures. The fix is common for 64-bit & 32-bit powerpc. Add support for the powerpc-specific "__powerpc64" architecture tag in the third patch, enabling proper test filtering in verifier test files. Introduce verifier testcases for tailcalls on powerpc64.
The first patch in series is fix patch, correcting memory alignment with 8-byte boundary for long branch address field. The subsequent patches enables verifier selftests on powerpc. The fifth patch in the series fixes incorrect comparator usage for comparing tailcall info with tailcall threshold. The last two patches fix JIT buffer overflow for large BPF progs and private stack memory leak (identified by bot during reviews).
Issue Details:
--------------
The Long branch stub in the trampoline implementation[1] provides
flexibility to handles short as well as long branch distance to
actual trampoline. Whereas, the 8 bytes long dummy_tramp_addr field
sitting before long branch stub leads to failure when enabling
verifier based seltest for ppc64.
The verifier selftests require disassembing the final jited image
to get native instructions. Later the disassembled instruction
sequence is matched against sequence of instructions provided in
test-file under __jited() wrapper. The final jited image contains
Out-of-line stub and Long branch stub as part of epilogue jitting
for a bpf program. The 8 bytes space for dummy_tramp is sandwiched
between both above mentioned stubs. These 8 bytes contain memory
address of dummy trampoline during trampoline invocation which don't
correspond to any powerpc instructions. So, disassembly fails
resulting in failure of verifier selftests.
The following code snippet shows the problem with current arrangement
made for dummy_tramp_addr.
/* Out-of-line stub */
mflr r0
[b|bl] tramp
mtlr r0 //only with OOL
b bpf_func + 4
/* Long branch stub */
.long <dummy_tramp_addr> <---Invalid bytes sequence, disassembly fails
mflr r11
bcl 20,31,$+4
mflr r12
ld r12, -8-SZL(r12)
mtctr r12
mtlr r11 //retain ftrace ABI
bctr
Consider test program binary of size 112 bytes:
0: 00000060 10004de8 00002039 f8ff21f9 81ff21f8 7000e1fb 3000e13b
28: 3000e13b 2a006038 f8ff7ff8 00000039 7000e1eb 80002138 7843037d
56: 2000804e a602087c 00000060 a603087c bcffff4b c0341d00 000000c0
84: a602687d 05009f42 a602887d f0ff8ce9 a603897d a603687d 2004804e
Disassembly output of above binary for ppc64le:
pc:0 left:112 00 00 00 60 : nop
pc:4 left:108 10 00 4d e8 : ld 2, 16(13)
pc:8 left:104 00 00 20 39 : li 9, 0
pc:12 left:100 f8 ff 21 f9 : std 9, -8(1)
pc:16 left:96 81 ff 21 f8 : stdu 1, -128(1)
pc:20 left:92 70 00 e1 fb : std 31, 112(1)
pc:24 left:88 30 00 e1 3b : addi 31, 1, 48
pc:28 left:84 30 00 e1 3b : addi 31, 1, 48
pc:32 left:80 2a 00 60 38 : li 3, 42
pc:36 left:76 f8 ff 7f f8 : std 3, -8(31)
pc:40 left:72 00 00 00 39 : li 8, 0
pc:44 left:68 70 00 e1 eb : ld 31, 112(1)
pc:48 left:64 80 00 21 38 : addi 1, 1, 128
pc:52 left:60 78 43 03 7d : mr 3, 8
pc:56 left:56 20 00 80 4e : blr
pc:60 left:52 a6 02 08 7c : mflr 0
pc:64 left:48 00 00 00 60 : nop
pc:68 left:44 a6 03 08 7c : mtlr 0
pc:72 left:40 bc ff ff 4b : b .-68
pc:76 left:36 c0 34 1d 00 :
...
Failure log:
Can't disasm instruction at offset 76: c0 34 1d 00 00 00 00 c0 a6 02 68 7d 05 00 9f 42
--------------------------------------
Observation:
Can't disasm instruction at offset 76 as this address has
".long <dummy_tramp_addr>" (0xc0341d00000000c0)
But valid instructions follow at offset 84 onwards.
Move the long branch address space to the bottom of the long
branch stub. This allows uninterrupted disassembly until the
last 8 bytes. Exclude these last bytes from the overall
program length to prevent failure in assembly generation.
Following is disassembler output for same test program with moved down
dummy_tramp_addr field:
.....
.....
pc:68 left:44 a6 03 08 7c : mtlr 0
pc:72 left:40 bc ff ff 4b : b .-68
pc:76 left:36 a6 02 68 7d : mflr 11
pc:80 left:32 05 00 9f 42 : bcl 20, 31, .+4
pc:84 left:28 a6 02 88 7d : mflr 12
pc:88 left:24 14 00 8c e9 : ld 12, 20(12)
pc:92 left:20 a6 03 89 7d : mtctr 12
pc:96 left:16 a6 03 68 7d : mtlr 11
pc:100 left:12 20 04 80 4e : bctr
pc:104 left:8 c0 34 1d 00 :
Failure log:
Can't disasm instruction at offset 104: c0 34 1d 00 00 00 00 c0
---------------------------------------
Disassembly logic can truncate at 104, ignoring last 8 bytes.
Update the dummy_tramp_addr field offset calculation from the end
of the program to reflect its new location, for bpf_arch_text_poke()
to update the actual trampoline's address in this field.
[1] https://lore.kernel.org/all/[email protected]v8->v9:
Dynamic pass handling until code keeps shrinking Fix private stack memory leak
v7->v8:
Fixed bot identified issues of alt_exit_addr and BPF_EXIT Fixed 32-bit ppc function signature mismatch
v6->v7:
Fixed JIT buffer overflow in case of large BPF progs Addressed remaining bot comments
v5->v6:
Changed alignment NOP emittion dependency on fimage layout Adjust tail truncate length for 32-bit ppc Addressed few minor bot comments
v4->v5:
Handled alignment NOP emit logic and corresponding stub offsets Handled image buffer overflow problem in last pass Above changes took care of other bot reviews
Included LLVMDisposeMessage() for graceful freeing Adjusted parameters in bpf_jit_build_fentry_stubs for ppc32 Adjusted expected JIT inst. in tailcall test forCONFIG_PPC_KERNEL_PCREL config
Added fix patch at last for inaccurate use of cmplwi inst.
v3->v4:
Changed logic for emitting alignment NOP
v2->v3:
Removed fixed NOP from bottom of long branch stub Rebased on top of bpf-next
v1->v2:
Added fix-patch to correct memory alignment in-place Moved the optional alignmnet NOP before OOL stub
[v1]: https://lore.kernel.org/bpf/[email protected]
[v2]: https://lore.kernel.org/bpf/[email protected]
[v3]: https://lore.kernel.org/bpf/[email protected]
[v4]: https://lore.kernel.org/bpf/[email protected]
[v5]: https://lore.kernel.org/bpf/[email protected]
[v6]: https://lore.kernel.org/bpf/[email protected]
[v7]: https://lore.kernel.org/bpf/[email protected]
[v8]: https://lore.kernel.org/bpf/[email protected]
Abhishek Dubey (8):
powerpc/bpf: fix alignment of long branch trampoline address powerpc/bpf: Move out dummy_tramp_addr after Long branch stub selftest/bpf: Fixing powerpc JIT disassembly failure selftest/bpf: Enable verifier selftest for powerpc64 powerpc64/bpf: fix compare instruction emitted for tailcall selftest/bpf: Add tailcall verifier selftest for powerpc64 powerpc/bpf: fix buffer overflow in JIT for large BPF programs powerpc64/bpf: fix percpu private stack leak on JIT failure
arch/powerpc/net/bpf_jit.h | 20 +++- arch/powerpc/net/bpf_jit_comp.c | 99 ++++++++++++++----- arch/powerpc/net/bpf_jit_comp32.c | 7 +- arch/powerpc/net/bpf_jit_comp64.c | 15 +-- .../selftests/bpf/jit_disasm_helpers.c | 27 ++++- tools/testing/selftests/bpf/progs/bpf_misc.h | 1 + .../bpf/progs/verifier_tailcall_jit.c | 69 +++++++++++++ tools/testing/selftests/bpf/test_loader.c | 5 + 8 files changed, 203 insertions(+), 40 deletions(-)
Signature · 1 line
2.52.0
PATCH v9 1/8 powerpc/bpf: fix alignment of long branch trampoline address
From: Abhishek Dubey <[email protected]>
Ensure the dummy trampoline address field present between the OOL stub and the long branch stub is 8-byte aligned, for memory compatibility when content loaded to a register.
--- arch/powerpc/net/bpf_jit.h | 4 ++-- arch/powerpc/net/bpf_jit_comp.c | 39 +++++++++++++++++++++++++++---- arch/powerpc/net/bpf_jit_comp32.c | 4 ++-- arch/powerpc/net/bpf_jit_comp64.c | 4 ++-- 4 files changed, 40 insertions(+), 11 deletions(-)
arch/powerpc/net/bpf_jit.h +2 -2
@@ -214,8 +214,8 @@ int bpf_jit_emit_func_call_rel(u32 *image, u32 *fimage, struct codegen_context *int bpf_jit_build_body (struct bpf_prog *fp,u32 *image,u32 *fimage,struct codegen_context *ctx,u32 *addrs,int pass,bool extra_pass);void bpf_jit_build_prologue (u32 *image,struct codegen_context *ctx); -void bpf_jit_build_epilogue (u32 *image,struct codegen_context *ctx); -void bpf_jit_build_fentry_stubs (u32 *image,struct codegen_context *ctx); +void bpf_jit_build_epilogue (u32 *image,u32 *fimage,struct codegen_context *ctx); +void bpf_jit_build_fentry_stubs (u32 *image,u32 *fimage,struct codegen_context *ctx);void bpf_jit_realloc_regs (struct codegen_context *ctx);int bpf_jit_emit_exit_insn (u32 *image,struct codegen_context *ctx,int tmp_reg,long exit_addr); void prepare_for_fsession_fentry(u32 *image, struct codegen_context *ctx, int cookie_cnt,
arch/powerpc/net/bpf_jit_comp.c +34 -5
@@ -49,11 +49,39 @@ asm (" .popsection ;" ); -void bpf_jit_build_fentry_stubs (u32 *image,struct codegen_context *ctx) +void bpf_jit_build_fentry_stubs (u32 *image,u32 *fimage,struct codegen_context *ctx) {int ool_stub_idx, long_branch_stub_idx; +int ool_stub_sz;/* +* In the final pass, align the mis-aligned dummy_tramp_addr field +* in the fimage. The alignment NOP must appear before OOL stub, +* to make ool_stub_idx & long_branch_stub_idx constant from end. +* +* dummy_tramp_addr must be 8-byte aligned for load-register +* compatibility. The fimage can be non 8-byte aligned, so final +* alignment depends on start of fimage and the stub's instruction +* count offset. The OOL stub size is 4 instructions (with +* CONFIG_PPC_FTRACE_OUT_OF_LINE) or 3 instructions (without) +* before dummy_tramp_addr. +* +* Emit a NOP here if (ctx->idx + ool_stub_sz) is odd, so that +* dummy_tramp_addr lands at an even instruction offset (== 8-byte +* aligned from an 8-byte aligned base). +* +* In pass=0 when image==NULL, conservatively account for space +* required to accommodate alignment NOP. In case final pass skips +* emitting alignment NOP, the image buffer have 4 spare bytes and +* jited_len signifies correct program size. +*/ + + ool_stub_sz =IS_ENABLED (CONFIG_PPC_FTRACE_OUT_OF_LINE ) ?16 :12 ; +if (!image || !IS_ALIGNED ((unsigned long )fimage + ctx->idx*4 + ool_stub_sz,SZL )) +EMIT (PPC_RAW_NOP ()); + +/* +* nop // optional, for alignment of dummy_tramp_addr * Out-of-line stub: * mflr r0 * [b|bl] tramp
@@ -70,7 +98,7 @@ void bpf_jit_build_fentry_stubs(u32 *image, struct codegen_context *ctx)/* * Long branch stub: -* .long <dummy_tramp_addr> +* .long <dummy_tramp_addr> // 8-byte aligned * mflr r11 * bcl 20,31,$+4 * mflr r12
@@ -81,6 +109,7 @@ void bpf_jit_build_fentry_stubs(u32 *image, struct codegen_context *ctx)*/ if (image) *((unsigned long *)&image[ctx->idx]) = (unsigned long )dummy_tramp; + ctx->idx +=SZL /4 ; long_branch_stub_idx = ctx->idx;EMIT (PPC_RAW_MFLR (_R11));
@@ -107,7 +136,7 @@ int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg,PPC_JMP (ctx->alt_exit_addr); }else { ctx->alt_exit_addr = ctx->idx *4 ; -bpf_jit_build_epilogue (image, ctx); +bpf_jit_build_epilogue (image,NULL , ctx); }return 0 ;
@@ -286,7 +315,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr*/ bpf_jit_build_prologue (NULL , &cgctx); addrs[fp->len] = cgctx.idx *4 ; -bpf_jit_build_epilogue (NULL , &cgctx); +bpf_jit_build_epilogue (NULL ,NULL , &cgctx); fixup_len = fp->aux->num_exentries *BPF_FIXUP_LEN *4 ; extable_len = fp->aux->num_exentries *sizeof (struct exception_table_entry );
@@ -318,7 +347,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prbpf_jit_binary_pack_free (fhdr, hdr); goto out_err; } -bpf_jit_build_epilogue (code_base, &cgctx); +bpf_jit_build_epilogue (code_base, fcode_base, &cgctx);if (bpf_jit_enable >1 ) pr_info("Pass %d: shrink = %d, seen = 0x%x\n", pass,
arch/powerpc/net/bpf_jit_comp32.c +2 -2
@@ -229,7 +229,7 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx } -void bpf_jit_build_epilogue (u32 *image,struct codegen_context *ctx) +void bpf_jit_build_epilogue (u32 *image,u32 *fimage,struct codegen_context *ctx) {EMIT (PPC_RAW_MR (_R3,bpf_to_ppc (BPF_REG_0 )));
@@ -237,7 +237,7 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)EMIT (PPC_RAW_BLR ()); -bpf_jit_build_fentry_stubs (image, ctx); +bpf_jit_build_fentry_stubs (image, fimage, ctx); }/* Relative offset needs to be calculated based on final image location */
arch/powerpc/net/bpf_jit_comp64.c +2 -2
@@ -398,7 +398,7 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx } } -void bpf_jit_build_epilogue (u32 *image,struct codegen_context *ctx) +void bpf_jit_build_epilogue (u32 *image,u32 *fimage,struct codegen_context *ctx) {bpf_jit_emit_common_epilogue (image, ctx);
@@ -407,7 +407,7 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)EMIT (PPC_RAW_BLR ()); -bpf_jit_build_fentry_stubs (image, ctx); +bpf_jit_build_fentry_stubs (image, fimage, ctx); }/*
Signature · 1 line
2.52.0
PATCH v9 2/8 powerpc/bpf: Move out dummy_tramp_addr after Long branch stub
In reply to [email protected]From: Abhishek Dubey <[email protected]>
Move the long branch address field to the bottom of the long branch stub. This allows uninterrupted disassembly until the last 8 bytes. The last bytes exclusion is logically necessary to prevent disassembly failure, otherwise the actual program layout is never altered. Hence no effect on overall program size. Also, align dummy_tramp_addr field with 8-byte boundary.
Following is disassembler output for test program with moved down dummy_tramp_addr field:
.....
.....
pc:68 left:44 a6 03 08 7c : mtlr 0 pc:72 left:40 bc ff ff 4b : b .-68 pc:76 left:36 a6 02 68 7d : mflr 11 pc:80 left:32 05 00 9f 42 : bcl 20, 31, .+4 pc:84 left:28 a6 02 88 7d : mflr 12 pc:88 left:24 14 00 8c e9 : ld 12, 20(12) pc:92 left:20 a6 03 89 7d : mtctr 12 pc:96 left:16 a6 03 68 7d : mtlr 11 pc:100 left:12 20 04 80 4e : bctr pc:104 left:8 c0 34 1d 00 :
Failure log:
Can't disasm instruction at offset 104: c0 34 1d 00 00 00 00 c0 Disassembly logic can truncate at 104, ignoring last 8 bytes.
Update the dummy_tramp_addr field offset calculation from the end of the program to reflect its new location, for bpf_arch_text_poke() to update the actual trampoline's address in this field.
All BPF trampoline selftests continue to pass with this patch applied.
--- arch/powerpc/net/bpf_jit.h | 3 +- arch/powerpc/net/bpf_jit_comp.c | 51 ++++++++++++++++--------------- arch/powerpc/net/bpf_jit_comp32.c | 3 +- arch/powerpc/net/bpf_jit_comp64.c | 3 +- 4 files changed, 33 insertions(+), 27 deletions(-)
arch/powerpc/net/bpf_jit.h +2 -1
@@ -217,7 +217,8 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);void bpf_jit_build_epilogue (u32 *image,u32 *fimage,struct codegen_context *ctx);void bpf_jit_build_fentry_stubs (u32 *image,u32 *fimage,struct codegen_context *ctx);void bpf_jit_realloc_regs (struct codegen_context *ctx); -int bpf_jit_emit_exit_insn (u32 *image,struct codegen_context *ctx,int tmp_reg,long exit_addr); +int bpf_jit_emit_exit_insn (u32 *image,u32 *fimage,struct codegen_context *ctx,int tmp_reg, +long exit_addr);void prepare_for_fsession_fentry (u32 *image,struct codegen_context *ctx,int cookie_cnt,int cookie_off,int retval_off);void store_func_meta (u32 *image,struct codegen_context *ctx,u64 func_meta,int func_meta_off);
arch/powerpc/net/bpf_jit_comp.c +27 -24
129 lines not on this page · raw message
arch/powerpc/net/bpf_jit_comp32.c +2 -1
@@ -1149,7 +1149,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, u32 *fimage, struct code* we'll just fall through to the epilogue. */ if (i != flen -1 ) { - ret =bpf_jit_emit_exit_insn (image, ctx, _R0, exit_addr); + ret =bpf_jit_emit_exit_insn (image, fimage, + ctx, _R0, exit_addr);if (ret)return ret; }
arch/powerpc/net/bpf_jit_comp64.c +2 -1
@@ -1726,7 +1726,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, u32 *fimage, struct code* we'll just fall through to the epilogue. */ if (i != flen -1 ) { - ret =bpf_jit_emit_exit_insn (image, ctx, tmp1_reg, exit_addr); + ret =bpf_jit_emit_exit_insn (image, fimage, ctx, + tmp1_reg, exit_addr);if (ret)return ret; }
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf v9 2/8] powerpc/bpf: Move out dummy_tramp_addr after Long branch stub
The commit message describes fixing a disassembly issue and updating offset calculations for bpf_arch_text_poke(). The change modifies how dummy_tramp_addr is positioned and how its offset is…
Quoted 6 lines
Move the long branch address field to the bottom of the long branch stub. This allows uninterrupted disassembly until the last 8 bytes. The last bytes exclusion is logically necessary to prevent disassembly failure, otherwise the actual program layout is never altered. Hence no effect on overall program size. Also, align
dummy_tramp_addrfield with 8-byte boundary.
The commit message describes fixing a disassembly issue and updating offset calculations for bpf_arch_text_poke(). The change modifies how dummy_tramp_addr is positioned and how its offset is calculated.
Looking at the diff, the offset calculation in bpf_arch_text_poke() changes from:
arch/powerpc/net/bpf_jit_comp.c int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, · line 1414
Quoted 2 lines
arch/powerpc/net/bpf_jit_comp.c +1
-1- ret = patch_ulong((void *)(bpf_func_end - bpf_jit_long_branch_stub - SZL), + ret = patch_ulong((void *)(bpf_func_end - SZL), /* SZL: dummy_tramp_addr offset */
This corrects a calculation that was introduced when long branch stubs were added. The original implementation placed dummy_tramp_addr at the top of the stub and calculated its location using the stub size. This commit moves it to the bottom and simplifies the calculation.
Should this include a Fixes tag for the commit that introduced the original placement and offset calculation?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28051760380
PATCH v9 3/8 selftest/bpf: Fixing powerpc JIT disassembly failure
In reply to [email protected]From: Abhishek Dubey <[email protected]>
Ensure that the trampoline stubs JITed at the tail of the epilogue do not expose the dummy trampoline address stored in the last 8 bytes(64-bit) and last 4 bytes(32-bit) to the disassembly flow. Prevent the disassembler from ingesting this memory address, as it may occasionally decode into a seemingly valid but incorrect instruction. Fix this issue by truncating the last 8/4 bytes from JITed buffers before supplying them for disassembly.
--- .../selftests/bpf/jit_disasm_helpers.c | 27 +++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/jit_disasm_helpers.c +25 -2
@@ -178,10 +178,12 @@ int get_jited_program_text(int fd, char *text, size_t text_sz) {struct bpf_prog_info info = {};__u32 info_len =sizeof (info); -__u32 jited_funcs, len, pc; +__u32 jited_funcs, len, pc, disasm_len; +__u32 trunc_len =0 ;__u32 *func_lens =NULL ;FILE *text_out =NULL ;uint8_t *image =NULL ; +char *triple =NULL ;int i, err =0 ;if (!llvm_initialized) {
@@ -225,9 +227,30 @@ int get_jited_program_text(int fd, char *text, size_t text_sz)if (!ASSERT_OK (err,"bpf_prog_get_info_by_fd #2" )) goto out; +/* +* last 8 bytes contains dummy_trampoline address in JIT +* output on 64-bit and last 4 bytes on 32-bit powerpc, +* which can't disassemble to a valid instruction. +*/ + triple =LLVMGetDefaultTargetTriple (); +if (triple) { +if (strstr (triple,"powerpc64" ) ||strstr (triple,"ppc64" )) + trunc_len =8 ; +else if (strstr (triple,"powerpc" ) ||strstr (triple,"ppc" )) + trunc_len =4 ; +LLVMDisposeMessage (triple); + } +for (pc =0 , i =0 ; i < jited_funcs; ++i) { +fprintf (text_out,"func #%d:\n" , i); -disasm_one_func (text_out, image + pc, func_lens[i]); +/* +* Disabled JIT have zero func_lens, hence underflow +*/ + disasm_len = func_lens[i] > trunc_len ? + func_lens[i] - trunc_len :0 ; +disasm_one_func (text_out, image + pc, disasm_len); +fprintf (text_out,"\n" ); pc += func_lens[i]; }
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf v9 3/8] selftest/bpf: Fixing powerpc JIT disassembly failure
This isn't a bug, but would it be clearer to follow the traditional C style by grouping related variables together? The disasm_len variable could be declared closer to where it's used along with…
Quoted 5 lines
tools/testing/selftests/bpf/jit_disasm_helpers.c +0
-0
tools/testing/selftests/bpf/jit_disasm_helpers.c int get_jited_program_text(int fd, char *text, size_t text_sz) · line 186
Quoted 10 lines
tools/testing/selftests/bpf/jit_disasm_helpers.c +3
-1@@ -178,10 +178,12 @@ int get_jited_program_text(int fd, char *text, size_t text_sz) {struct bpf_prog_info info = {};__u32 info_len =sizeof (info); -__u32 jited_funcs, len, pc; +__u32 jited_funcs, len, pc, disasm_len; +__u32 trunc_len =0 ;__u32 *func_lens =NULL ;FILE *text_out =NULL ;uint8_t *image =NULL ; +char *triple =NULL ;
This isn't a bug, but would it be clearer to follow the traditional C style by grouping related variables together? The disasm_len variable could be declared closer to where it's used along with trunc_len, and adding a blank line before the fprintf() statement in the loop might improve readability.
[ ... ]
Quoted 22 lines
tools/testing/selftests/bpf/jit_disasm_helpers.c +15
-1@@ -225,9 +227,30 @@ int get_jited_program_text(int fd, char *text, size_t text_sz)if (!ASSERT_OK (err,"bpf_prog_get_info_by_fd #2" )) goto out; +/* +* last 8 bytes contains dummy_trampoline address in JIT +* output on 64-bit and last 4 bytes on 32-bit powerpc, +* which can't disassemble to a valid instruction. +*/ + triple =LLVMGetDefaultTargetTriple (); +if (triple) { +if (strstr (triple,"powerpc64" ) ||strstr (triple,"ppc64" )) + trunc_len =8 ; +else if (strstr (triple,"powerpc" ) ||strstr (triple,"ppc" )) + trunc_len =4 ; +LLVMDisposeMessage (triple); + } +for (pc =0 , i =0 ; i < jited_funcs; ++i) { +fprintf (text_out,"func #%d:\n" , i); -disasm_one_func (text_out, image + pc, func_lens[i]);
tools/testing/selftests/bpf/jit_disasm_helpers.c int get_jited_program_text(int fd, char *text, size_t text_sz) · line 256
Quoted 10 lines
tools/testing/selftests/bpf/jit_disasm_helpers.c +7
-0+/* +* Disabled JIT have zero func_lens, hence underflow +*/ + disasm_len = func_lens[i] > trunc_len ? + func_lens[i] - trunc_len :0 ; +disasm_one_func (text_out, image + pc, disasm_len); +fprintf (text_out,"\n" ); pc += func_lens[i]; }
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28051760380
PATCH v9 4/8 selftest/bpf: Enable verifier selftest for powerpc64
In reply to [email protected]From: Abhishek Dubey <[email protected]>
This patch enables arch specifier "__powerpc64" in verifier selftest for ppc64. Power 32-bit would require separate handling. Changes tested for 64-bit only.
--- tools/testing/selftests/bpf/progs/bpf_misc.h | 1 + tools/testing/selftests/bpf/test_loader.c | 5 +++++ 2 files changed, 6 insertions(+)
tools/testing/selftests/bpf/progs/bpf_misc.h +1 -0
@@ -158,6 +158,7 @@#define __arch_arm64 __arch("ARM64")#define __arch_riscv64 __arch("RISCV64")#define __arch_s390x __arch("s390x") +#define __arch_powerpc64 __arch("POWERPC64")#define __caps_unpriv (caps) __test_tag("test_caps_unpriv=" EXPAND_QUOTE(caps))#define __load_if_JITed () __test_tag("load_mode=jited")#define __load_if_no_JITed () __test_tag("load_mode=no_jited")
tools/testing/selftests/bpf/test_loader.c +5 -0
@@ -377,6 +377,7 @@ enum arch {ARCH_ARM64 =0x4 ,ARCH_RISCV64 =0x8 ,ARCH_S390X =0x10 , + ARCH_POWERPC64 = 0x20, };static int get_current_arch (void )
@@ -389,6 +390,8 @@ static int get_current_arch(void)return ARCH_RISCV64 ;#elif defined(__s390x__)return ARCH_S390X ; +#elif defined(__powerpc64__) +return ARCH_POWERPC64 ;#endif return ARCH_UNKNOWN ; }
@@ -580,6 +583,8 @@ static int parse_test_spec(struct test_loader *tester, arch =ARCH_RISCV64 ; }else if (strcmp (val,"s390x" ) ==0 ) { arch =ARCH_S390X ; + }else if (strcmp (val,"POWERPC64" ) ==0 ) { + arch =ARCH_POWERPC64 ; }else {PRINT_FAIL ("bad arch spec: '%s'\n" , val); err = -EINVAL ;
Signature · 1 line
2.52.0
PATCH v9 5/8 powerpc64/bpf: fix compare instruction emitted for tailcall
In reply to [email protected]From: Abhishek Dubey <[email protected]>
The tail_call_info field can contain either a scalar counter value or a 64-bit pointer to the counter, using a 32-bit compare (cmplwi) only checks the lower 32 bits, which can lead to incorrect comparisions when location of counter is near 4GB boundary. Use instruction cmpldi/cmplwi for accurate comparision in corresponding cases.
--- arch/powerpc/net/bpf_jit.h | 6 ++++++ arch/powerpc/net/bpf_jit_comp.c | 2 +- arch/powerpc/net/bpf_jit_comp64.c | 8 ++++---- 3 files changed, 11 insertions(+), 5 deletions(-)
arch/powerpc/net/bpf_jit.h +6 -0
@@ -188,6 +188,12 @@ struct codegen_context {#define bpf_to_ppc (r) (ctx->b2p[r]) +#ifdef CONFIG_PPC64 +#define PPC_RAW_CMPLLI (a, i) PPC_RAW_CMPLDI(a, i) +#else +#define PPC_RAW_CMPLLI (a, i) PPC_RAW_CMPLWI(a, i) +#endif +#ifdef CONFIG_PPC32 #define BPF_FIXUP_LEN 3/* Three instructions => 12 bytes */ #else
arch/powerpc/net/bpf_jit_comp.c +1 -1
@@ -763,7 +763,7 @@ static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_conte* Setting the tail_call_info in trampoline's frame * depending on if previous frame had value or reference. */ -EMIT (PPC_RAW_CMPLWI (_R3,MAX_TAIL_CALL_CNT )); +EMIT (PPC_RAW_CMPLLI (_R3,MAX_TAIL_CALL_CNT ));PPC_BCC_CONST_SHORT (COND_GT ,8 );EMIT (PPC_RAW_ADDI (_R3, _R4, -BPF_PPC_TAILCALL ));
arch/powerpc/net/bpf_jit_comp64.c +4 -4
@@ -276,7 +276,7 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)*/ EMIT (PPC_RAW_LD (bpf_to_ppc (TMP_REG_2 ), _R1,0 ));EMIT (PPC_RAW_LD (bpf_to_ppc (TMP_REG_1 ),bpf_to_ppc (TMP_REG_2 ), -(BPF_PPC_TAILCALL ))); -EMIT (PPC_RAW_CMPLWI (bpf_to_ppc (TMP_REG_1 ),MAX_TAIL_CALL_CNT )); +EMIT (PPC_RAW_CMPLDI (bpf_to_ppc (TMP_REG_1 ),MAX_TAIL_CALL_CNT ));PPC_BCC_CONST_SHORT (COND_GT ,8 );EMIT (PPC_RAW_ADDI (bpf_to_ppc (TMP_REG_1 ),bpf_to_ppc (TMP_REG_2 ), -(BPF_PPC_TAILCALL )));
@@ -651,7 +651,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 oPPC_BCC_SHORT (COND_GE , out);EMIT (PPC_RAW_LD (bpf_to_ppc (TMP_REG_1 ), _R1,bpf_jit_stack_tailcallinfo_offset (ctx))); -EMIT (PPC_RAW_CMPLWI (bpf_to_ppc (TMP_REG_1 ),MAX_TAIL_CALL_CNT )); +EMIT (PPC_RAW_CMPLDI (bpf_to_ppc (TMP_REG_1 ),MAX_TAIL_CALL_CNT ));PPC_BCC_CONST_SHORT (COND_LE ,8 );/* dereference TMP_REG_1 */
@@ -661,7 +661,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o* if (tail_call_info == MAX_TAIL_CALL_CNT) * goto out; */ -EMIT (PPC_RAW_CMPLWI (bpf_to_ppc (TMP_REG_1 ),MAX_TAIL_CALL_CNT )); +EMIT (PPC_RAW_CMPLDI (bpf_to_ppc (TMP_REG_1 ),MAX_TAIL_CALL_CNT ));PPC_BCC_SHORT (COND_EQ , out);/*
@@ -696,7 +696,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o* tail_call_info. */ EMIT (PPC_RAW_LD (bpf_to_ppc (TMP_REG_2 ), _R1,bpf_jit_stack_tailcallinfo_offset (ctx))); -EMIT (PPC_RAW_CMPLWI (bpf_to_ppc (TMP_REG_2 ),MAX_TAIL_CALL_CNT )); +EMIT (PPC_RAW_CMPLDI (bpf_to_ppc (TMP_REG_2 ),MAX_TAIL_CALL_CNT ));PPC_BCC_CONST_SHORT (COND_GT ,8 );/* First get address of tail_call_info */
Signature · 1 line
2.52.0
PATCH v9 6/8 selftest/bpf: Add tailcall verifier selftest for powerpc64
In reply to [email protected]From: Abhishek Dubey <[email protected]>
Verifier testcase result for tailcalls:
# ./test_progs -t verifier_tailcall
#618/1 verifier_tailcall/invalid map type for tail call:OK
#618/2 verifier_tailcall/invalid map type for tail call @unpriv:OK
#618 verifier_tailcall:OK
#619/1 verifier_tailcall_jit/main:OK
#619 verifier_tailcall_jit:OK
Summary: 2/3 PASSED, 0 SKIPPED, 0 FAILED
--- .../bpf/progs/verifier_tailcall_jit.c | 69 +++++++++++++++++++ 1 file changed, 69 insertions(+)
tools/testing/selftests/bpf/progs/verifier_tailcall_jit.c +69 -0
@@ -91,6 +91,75 @@ __jited(" popq %rax") __jited(" jmp {{.*}}") /* jump to tail call tgt */ __jited("L0: leave") __jited(" {{(retq|jmp 0x)}}") /* return or jump to rethunk */ +__arch_powerpc64 +/* program entry for main(), regular function prologue */ +__jited(" nop") +__jited("...") /* ld 2, 16(13) absent with CONFIG_PPC_KERNEL_PCREL */ +__jited(" li 9, 0") +__jited(" std 9, -8(1)") +__jited(" mflr 0") +__jited(" std 0, 16(1)") +__jited (" stdu 1, {{.*}}(1)" ) +/* load address and call sub() via count register */ +__jited(" lis 12, {{.*}}") +__jited(" sldi 12, 12, 32") +__jited(" oris 12, 12, {{.*}}") +__jited(" ori 12, 12, {{.*}}") +__jited(" mtctr 12") +__jited(" bctrl") +__jited(" mr 8, 3") +__jited(" li 8, 0") +__jited(" addi 1, 1, {{.*}}") +__jited(" ld 0, 16(1)") +__jited(" mtlr 0") +__jited(" mr 3, 8") +__jited(" blr") +__jited("...") +__jited ("func #1" ) +/* subprogram entry for sub() */ +__jited(" nop") +__jited ("..." )/* ld 2, 16(13) absent with CONFIG_PPC_KERNEL_PCREL */ +/* tail call prologue for subprogram */ +__jited(" ld 10, 0(1)") +__jited(" ld 9, -8(10)") +__jited(" cmpldi 9, 33") +__jited(" bt {{.*}}, {{.*}}") +__jited(" addi 9, 10, -8") +__jited(" std 9, -8(1)") +__jited(" lis {{.*}}, {{.*}}") +__jited(" sldi {{.*}}, {{.*}}, 32") +__jited(" oris {{.*}}, {{.*}}, {{.*}}") +__jited(" ori {{.*}}, {{.*}}, {{.*}}") +__jited(" li {{.*}}, 0") +__jited(" lwz 9, {{.*}}({{.*}})") +__jited(" slwi {{.*}}, {{.*}}, 0") +__jited(" cmplw {{.*}}, 9") +__jited (" bf 0, {{.*}}" ) +/* bpf_tail_call implementation */ +__jited(" ld 9, -8(1)") +__jited(" cmpldi 9, 33") +__jited(" bf {{.*}}, {{.*}}") +__jited(" ld 9, 0(9)") +__jited(" cmpldi 9, 33") +__jited(" bt {{.*}}, {{.*}}") +__jited(" addi 9, 9, 1") +__jited(" mulli 10, {{.*}}, 8") +__jited(" add 10, 10, {{.*}}") +__jited(" ld 10, {{.*}}(10)") +__jited(" cmpldi 10, 0") +__jited(" bt {{.*}}, {{.*}}") +__jited(" ld 10, {{.*}}(10)") +__jited(" addi 10, 10, {{.*}}") /* offset depends on CONFIG_PPC_KERNEL_PCREL */ +__jited(" mtctr 10") +__jited(" ld 10, -8(1)") +__jited(" cmpldi 10, 33") +__jited(" bt {{.*}}, {{.*}}") +__jited(" addi 10, 1, -8") +__jited(" std 9, 0(10)") +__jited(" bctr") +__jited(" mr 3, 8") +__jited(" blr") + SEC("tc")__naked int main (void ) {
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf v9 6/8] selftest/bpf: Add tailcall verifier selftest for powerpc64
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] JIT disasm regex for loading 64-bit immediates strictly expects 'oris' and 'ori' instructions…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] JIT disasm regex for loading 64-bit immediates strictly expects 'oris' and 'ori' instructions, but the PowerPC JIT conditionally omits them.
- [Medium] JIT disasm regex for BPF-to-BPF tail calls fails on PowerPC kernels with CONFIG_PPC_KERNEL_PCREL enabled.
--
commit b4e7d4404702ca69adf00490d083d127f762e2ed
Author: Abhishek Dubey <[email protected]>
selftest/bpf: Add tailcall verifier selftest for powerpc64
This commit adds a tailcall verifier selftest targeting the powerpc64 architecture. It introduces __arch_powerpc64 JIT regex matches for both regular function calls and tail calls.
Quoted 12 lines
tools/testing/selftests/bpf/progs/verifier_tailcall_jit.c +4
-0@@ -91,6 +91,75 @@ __jited(" popq %rax") __jited(" jmp {{.*}}") /* jump to tail call tgt */ __jited("L0: leave") __jited(" {{(retq|jmp 0x)}}") /* return or jump to rethunk */ +__arch_powerpc64 +/* program entry for main(), regular function prologue */ +__jited(" nop") +__jited ("..." )/* ld 2, 16(13) absent with CONFIG_PPC_KERNEL_PCREL */
tools/testing/selftests/bpf/progs/verifier_tailcall_jit.c __jited(" popq %rax") · line 107
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_tailcall_jit.c +10
-0+__jited(" li 9, 0") +__jited(" std 9, -8(1)") +__jited(" mflr 0") +__jited(" std 0, 16(1)") +__jited (" stdu 1, {{.*}}(1)" ) +/* load address and call sub() via count register */ +__jited(" lis 12, {{.*}}") +__jited(" sldi 12, 12, 32") +__jited(" oris 12, 12, {{.*}}") +__jited (" ori 12, 12, {{.*}}" )
[Severity: Medium]
Will this strict absolute 64-bit load match fail on Power10 and other kernels with CONFIG_PPC_KERNEL_PCREL enabled?
It appears that when CONFIG_PPC_KERNEL_PCREL is enabled, bpf_jit_emit_func_call_rel() checks if the relative offset between the call site and the target is within 8GB. Since main() and sub() are BPF programs allocated close to each other in vmalloc space, the offset should fit within 8GB.
This means the JIT would emit a PC-relative load (pla r12, addr followed by paddi) instead of the absolute load sequence expected here. Does the test need a fallback or PCREL-specific wildcard block?
Quoted 2 lines
tools/testing/selftests/bpf/progs/verifier_tailcall_jit.c +2
-0+__jited(" mtctr 12") +__jited (" bctrl" )
[ ... ]
tools/testing/selftests/bpf/progs/verifier_tailcall_jit.c __jited(" popq %rax") · line 132
Quoted 11 lines
tools/testing/selftests/bpf/progs/verifier_tailcall_jit.c +11
-0+/* tail call prologue for subprogram */ +__jited(" ld 10, 0(1)") +__jited(" ld 9, -8(10)") +__jited(" cmpldi 9, 33") +__jited(" bt {{.*}}, {{.*}}") +__jited(" addi 9, 10, -8") +__jited(" std 9, -8(1)") +__jited(" lis {{.*}}, {{.*}}") +__jited(" sldi {{.*}}, {{.*}}, 32") +__jited(" oris {{.*}}, {{.*}}, {{.*}}") +__jited (" ori {{.*}}, {{.*}}, {{.*}}" )
[Severity: Medium]
Can this strict expectation for oris and ori cause test failures depending on the page size?
The PPC_LI64 macro optimizes 64-bit immediate loads by omitting the ori instruction if the lowest 16 bits are zero, and oris if bits 16-31 are zero.
Because the test loads the address of the jmp_table map, which is allocated via vmalloc() and is page-aligned, on PowerPC64 systems with a 64KB page size the lowest 16 bits will always be zero. This would cause the ori instruction to be omitted entirely, resulting in a regex mismatch.
Quoted 2 lines
tools/testing/selftests/bpf/progs/verifier_tailcall_jit.c +2
-0+__jited(" li {{.*}}, 0") +__jited (" lwz 9, {{.*}}({{.*}})" )
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
PATCH v9 7/8 powerpc/bpf: fix buffer overflow in JIT for large BPF programs
In reply to [email protected]From: Abhishek Dubey <[email protected]>
During size calculation in pass-0, exit_addr is 0 since addrs[fp->len] is not yet populated. bpf_jit_emit_exit_insn() treats a zero exit_addr as in-range and skips bpf_jit_build_epilogue(), so the alternate inline epilogue instructions are not counted in alloclen.
In later passes, if the real exit_addr falls outside the 32MB branch range, the full inline epilogue is emitted into the already-allocated buffer, writing past its end and corrupting adjacent memory.
Fix by ensuring exit_addr is non-zero before treating it as in-range, so pass-0 always falls through to bpf_jit_build_epilogue() and conservatively accounts for all epilogue instructions in alloclen. Also range check alt_exit_addr directly in the else-if condition.
Since exit_addr handling now falls through to the epilogue, two related issues in bpf_int_jit_compile() must also be addressed:
1. Reset cgctx.alt_exit_addr before the second size-calculation pass. Without this, a stale alt_exit_addr from the first pass causes the second pass to emit a single jump instead of the full epilogue, undercounting alloclen and reintroducing the overflow.
2. Recompute addrs[fp->len] at the end of each code-generation pass. The larger pass-0 body can shrink in later passes as out-of-range exits settle into in-range jumps; a stale addrs[fp->len] would leave exit branches targeting past the real (shrunken) epilogue.
Because shrinkage in a later pass can move the epilogue offset, the fixed two-pass loop is no longer sufficient: an exit that was out of range in an earlier pass may fall in range once the epilogue offset shrinks, shrinking the body further and overwriting the start of the epilogue. Convert the code-generation loop to iterate until the program size converges, bounded by CODEGEN_MAX_PASSES, and fail the JIT if it does not converge.
--- arch/powerpc/net/bpf_jit.h | 7 +++++++ arch/powerpc/net/bpf_jit_comp.c | 31 ++++++++++++++++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-)
arch/powerpc/net/bpf_jit.h +7 -0
@@ -14,6 +14,13 @@#include <asm/ppc-opcode.h> #include <linux/build_bug.h> +/* +* We need at least 2 passes for proper code generation, and may need +* additional passes if code size changes between passes. +*/ +#define CODEGEN_MIN_PASSES 2 +#define CODEGEN_MAX_PASSES 3 +#ifdef CONFIG_PPC64_ELF_ABI_V1 #define FUNCTION_DESCR_SIZE 24#else
arch/powerpc/net/bpf_jit_comp.c +24 -7
@@ -128,11 +128,10 @@ void bpf_jit_build_fentry_stubs(u32 *image, u32 *fimage, struct codegen_contextint bpf_jit_emit_exit_insn (u32 *image,u32 *fimage,struct codegen_context *ctx,int tmp_reg,long exit_addr) { -if (!exit_addr ||is_offset_in_branch_range (exit_addr - (ctx->idx *4 ))) { +if (exit_addr &&is_offset_in_branch_range (exit_addr - (long )(ctx->idx *4 ))) {PPC_JMP (exit_addr); - }else if (ctx->alt_exit_addr) { -if (WARN_ON (!is_offset_in_branch_range ((long )ctx->alt_exit_addr - (ctx->idx *4 )))) -return -1 ; + }else if (ctx->alt_exit_addr &&is_offset_in_branch_range ( + (long )(ctx->alt_exit_addr) - (long )(ctx->idx *4 ))) {PPC_JMP (ctx->alt_exit_addr); }else { ctx->alt_exit_addr = ctx->idx *4 ;
@@ -303,6 +302,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr*/ if (cgctx.seen &SEEN_TAILCALL || !is_offset_in_branch_range ((long )cgctx.idx *4 )) { cgctx.idx =0 ; + cgctx.alt_exit_addr =0 ;if (bpf_jit_build_body (fp,NULL ,NULL , &cgctx, addrs,0 , false)) goto out_err; }
@@ -335,8 +335,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr code_base = (u32 *)(image +FUNCTION_DESCR_SIZE ); fcode_base = (u32 *)(fimage +FUNCTION_DESCR_SIZE ); -/* Code generation passes 1-2 */ -for (pass =1 ; pass <3 ; pass++) { +/* Code generation passes 1-2+, loop until program size converges. */ +for (pass =1 ; pass <=CODEGEN_MAX_PASSES ; pass++) { +u32 prev_proglen = proglen; +/* Now build the prologue, body code & epilogue for real. */ cgctx.idx =0 ; cgctx.alt_exit_addr =0 ;
@@ -347,11 +349,26 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prbpf_jit_binary_pack_free (fhdr, hdr); goto out_err; } + addrs[fp->len] = cgctx.idx *4 ;bpf_jit_build_epilogue (code_base, fcode_base, &cgctx); + proglen = cgctx.idx *4 ; +if (bpf_jit_enable >1 )pr_info ("Pass %d: shrink = %d, seen = 0x%x\n" , pass, - proglen - (cgctx.idx *4 ), cgctx.seen); + prev_proglen - proglen, cgctx.seen); + +/* Check if program size has converged, but ensure minimum passes */ +if (pass >=CODEGEN_MIN_PASSES && proglen == prev_proglen) +break ; + +if (pass ==CODEGEN_MAX_PASSES && proglen != prev_proglen) { +pr_err ("BPF JIT: Program did not converge after %d passes\n" , +CODEGEN_MAX_PASSES ); +bpf_arch_text_copy (&fhdr->size, &hdr->size,sizeof (hdr->size)); +bpf_jit_binary_pack_free (fhdr, hdr); + goto out_err; + } }if (bpf_jit_enable >1 )
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf v9 7/8] powerpc/bpf: fix buffer overflow in JIT for large BPF programs
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Stale exception table due to missing updates during additional JIT passes, leading to kernel panic…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Stale exception table due to missing updates during additional JIT passes, leading to kernel panic on BPF_PROBE_MEM.
- [Medium] Unratelimited error log on JIT convergence failure enables log spam (Denial of Service).
--
commit b6894df59a3f875e5bd3f27ed0af4e32b5c00461
Author: Abhishek Dubey <[email protected]>
powerpc/bpf: fix buffer overflow in JIT for large BPF programs
This commit fixes a buffer overflow in the PowerPC BPF JIT by ensuring the pass-0 size calculation falls through to the epilogue when the exit address is out of range. It converts the fixed two-pass code generation into a loop that iterates up to CODEGEN_MAX_PASSES until the program size converges.
Quoted 4 lines
arch/powerpc/net/bpf_jit_comp.c +0
-0
[ ... ]
arch/powerpc/net/bpf_jit_comp.c struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · line 344
Quoted 13 lines
arch/powerpc/net/bpf_jit_comp.c +4
-2@@ -335,8 +335,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr code_base = (u32 *)(image +FUNCTION_DESCR_SIZE ); fcode_base = (u32 *)(fimage +FUNCTION_DESCR_SIZE ); -/* Code generation passes 1-2 */ -for (pass =1 ; pass <3 ; pass++) { +/* Code generation passes 1-2+, loop until program size converges. */ +for (pass =1 ; pass <=CODEGEN_MAX_PASSES ; pass++) { +u32 prev_proglen = proglen; +/* Now build the prologue, body code & epilogue for real. */ cgctx.idx =0 ; cgctx.alt_exit_addr =0 ;
[Severity: High]
Does this loop need to reset cgctx.exentry_idx along with cgctx.idx?
Since CODEGEN_MAX_PASSES is now 3, the exception table could be populated with stale instruction offsets if the JIT requires 3 passes.
Looking at bpf_add_extable_entry(), there is a hardcoded pass check:
bpf_add_extable_entry() {
/* Populate extable entries only in the last pass */
if (pass != 2)
return 0;
...
}If the program takes 3 passes to converge, this would leave the exception table populated exclusively during pass 2. During pass 3, instruction sizes shrink and absolute offsets shift, leaving the exception table with stale relative offsets.
When a page fault occurs on a valid BPF_PROBE_MEM, could the kernel fail to find the matching exception table entry and panic?
Quoted 14 lines
arch/powerpc/net/bpf_jit_comp.c +3
-0bpf_jit_build_prologue (code_base, &cgctx);if (bpf_jit_build_body (fp, code_base, fcode_base, &cgctx, addrs, pass, extra_pass)) {bpf_arch_text_copy (&fhdr->size, &hdr->size,sizeof (hdr->size));bpf_jit_binary_pack_free (fhdr, hdr); goto out_err; } + addrs[fp->len] = cgctx.idx *4 ;bpf_jit_build_epilogue (code_base, fcode_base, &cgctx); + proglen = cgctx.idx *4 ; +if (bpf_jit_enable >1 ) pr_info("Pass %d: shrink = %d, seen = 0x%x\n", pass,
arch/powerpc/net/bpf_jit_comp.c struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · line 367
Quoted 10 lines
arch/powerpc/net/bpf_jit_comp.c +9
-1- proglen - (cgctx.idx * 4), cgctx.seen); + prev_proglen - proglen, cgctx.seen); + +/* Check if program size has converged, but ensure minimum passes */ +if (pass >=CODEGEN_MIN_PASSES && proglen == prev_proglen) +break ; + +if (pass ==CODEGEN_MAX_PASSES && proglen != prev_proglen) { +pr_err ("BPF JIT: Program did not converge after %d passes\n" , +CODEGEN_MAX_PASSES );
[Severity: Medium]
Could this unconditional pr_err() cause log spam?
If a user loads a pathological BPF program that fails to converge, they could repeatedly trigger this compilation failure and flood the kernel logs.
Should this use pr_err_ratelimited() instead, similar to other user-triggerable JIT failures?
Quoted 7 lines
arch/powerpc/net/bpf_jit_comp.c +4
-0+bpf_arch_text_copy (&fhdr->size, &hdr->size,sizeof (hdr->size)); +bpf_jit_binary_pack_free (fhdr, hdr); + goto out_err; + } }if (bpf_jit_enable >1 )
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
PATCH v9 8/8 powerpc64/bpf: fix percpu private stack leak on JIT failure
In reply to [email protected]From: Abhishek Dubey <[email protected]>
The existing conditional statement in bpf_int_jit_compile() frees the percpu private stack at out_addrs only when the image buffer was never allocated.
If bpf_jit_build_body() fails during a code-generation pass, the image buffer has already been allocated, so !image is false and the percpu stack is not freed.
Because JIT compilation failed, fp->jited remains at 0. The subsequent bpf_jit_free() path only frees priv_stack_ptr when fp->jited is set, so freeing is skipped here too, leaking the percpu allocation.
Fix implements freeing the private stack whenever fp->jited was not set, i.e. compilation did not succeed, instead of keying off !image. !fp->jited already covers the !image case, since image is only NULL on early-failure paths where fp->jited is likewise 0.
--- arch/powerpc/net/bpf_jit_comp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/net/bpf_jit_comp.c +1 -1
@@ -403,7 +403,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr (void *)fimage + FUNCTION_DESCR_SIZE); out_addrs: -if (!image && priv_stack_ptr) { +if (!fp->jited && priv_stack_ptr) { fp->aux->priv_stack_ptr =NULL ;free_percpu (priv_stack_ptr); }
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf v9 8/8] powerpc64/bpf: fix percpu private stack leak on JIT failure
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Pre-existing issues:
- [High] bpf_jit_free() ignores the return value of bpf_jit_binary_pack_finalize(), leading to a double free, use-after-free, and modmem underflow if finalization fails.
- [Medium] The exact same percpu memory leak fixed in powerpc by this patch is also present in the arm64 BPF JIT compiler.
--
commit bd7becf49ebb21e1322fb30107d94244f94e35a2
Author: Abhishek Dubey <[email protected]>
powerpc64/bpf: fix percpu private stack leak on JIT failure
This patch fixes a percpu private stack leak on JIT failure in the powerpc64 BPF JIT compiler. It ensures the private stack is freed whenever JIT compilation fails by checking !fp->jited instead of !image.
arch/powerpc/net/bpf_jit_comp.c struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · line 406
Quoted 10 lines
arch/powerpc/net/bpf_jit_comp.c +1
-1@@ -403,7 +403,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr (void *)fimage + FUNCTION_DESCR_SIZE); out_addrs: -if (!image && priv_stack_ptr) { +if (!fp->jited && priv_stack_ptr) {
[Severity: Medium]
This is a pre-existing issue, but the exact same percpu memory leak fixed here in powerpc is also present in the arm64 BPF JIT compiler.
In arm64's bpf_int_jit_compile(), if JIT fails after ro_header is allocated:
arch/arm64/net/bpf_jit_comp.c:bpf_int_jit_compile() {
...
if (header) {
bpf_arch_text_copy(&ro_header->size, &header->size,
sizeof(header->size));
bpf_jit_binary_pack_free(ro_header, header);
}
goto out_off;
...
out_off:
if (!ro_header && priv_stack_ptr) {
free_percpu(priv_stack_ptr);
prog->aux->priv_stack_ptr = NULL;
}
...
}Because ro_header is freed but not set to NULL, it bypasses the cleanup at out_off, leaking the private stack allocation. Should a similar fix be applied to arm64?
[Severity: High]
This is a pre-existing issue, but there is a double-free vulnerability in bpf_jit_free() that affects powerpc, arm64, and x86 when finalization fails.
In arch/powerpc/net/bpf_jit_comp.c:bpf_jit_free():
if (jit_data) {
bpf_jit_binary_pack_finalize(jit_data->fhdr, jit_data->hdr);
...
}
hdr = bpf_jit_binary_pack_hdr(fp);
bpf_jit_binary_pack_free(hdr, NULL);If bpf_jit_binary_pack_finalize() fails (e.g., due to memory permission errors), it explicitly frees the RO header internally. However, its return value is ignored here. The code proceeds to extract the same header and call bpf_jit_binary_pack_free() again, leading to a double-free and use-after-free (dereferencing the freed size).
The exact same vulnerability exists in the arm64 and x86 JIT compilers. Does the return value of bpf_jit_binary_pack_finalize() need to be checked to prevent this double free?
Quoted 3 lines
arch/powerpc/net/bpf_jit_comp.c +0
-0fp->aux->priv_stack_ptr =NULL ;free_percpu (priv_stack_ptr); }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Hi Abhishek,
I have tested this patch series on PowerPC64 (ppc64le) and can confirm all 8 patches work correctly.
## Test Environment
- System: PowerPC64 (ppc64le)
- Kernel: 7.1.0-rc3 with all 8 patches applied
- BTF Support: Enabled (CONFIG_DEBUG_INFO_BTF=y)
- Test Suite: tools/testing/selftests/bpf/test_progs
## Test Results
### WITHOUT Patches (Baseline):
- Kernel: 7.1.0-rc3-00362-g6916d5703ddf
- Summary: 111/2135 PASSED, 35 SKIPPED, 5 FAILED
- verifier_tailcall_jit: **FAIL**
- Error: "Can't disasm instruction at offset 96: 60 80 1c 00 00 00 00 c0"
### WITH All 8 Patches:
- Kernel: 7.1.0-rc3-00370-g202ecf282b6e
- Summary: 112/2136 PASSED, 35 SKIPPED, 4 FAILED
- verifier_tailcall_jit: **OK**
The critical PowerPC64-specific test verifier_tailcall_jit now passes:
#635/1 verifier_tailcall_jit/main:OK
#635 verifier_tailcall_jit:OK
This test was previously failing due to JIT disassembly issues caused by the dummy_tramp_addr placement. The patches successfully:
1. Fix alignment of long branch trampoline address
2. Relocate dummy_tramp_addr to bottom of stub
3. Fix JIT disassembly truncation
4. Enable PowerPC64 arch support in verifier tests
5. Fix compare instruction (cmpldi vs cmplwi) for tailcalls
6. Add PowerPC64 tailcall verifier test
7. Fix JIT buffer overflow for large programs
8. Fix percpu private stack leak on JIT failure
The 4 remaining failures are all in verifier_arena* tests with -ENOMEM errors, which are unrelated to these patches and appear to be platform-specific memory allocation issues with the arena feature.
Please add below tag.
Regards,
Yeswanth
Quoted 190 lines · On 24/06/26 4:44 am, [email protected] wrote:
From: Abhishek Dubey <[email protected]>
The verifier selftest validates JITed instructions by matching expected disassembly output. The first two patches fix issues in powerpc instruction disassembly that were causing test flow failures. The fix is common for 64-bit & 32-bit powerpc. Add support for the powerpc-specific "
__powerpc64" architecture tag in the third patch, enabling proper test filtering in verifier test files. Introduce verifier testcases for tailcalls on powerpc64.The first patch in series is fix patch, correcting memory alignment with 8-byte boundary for long branch address field. The subsequent patches enables verifier selftests on powerpc. The fifth patch in the series fixes incorrect comparator usage for comparing tailcall info with tailcall threshold. The last two patches fix JIT buffer overflow for large BPF progs and private stack memory leak (identified by bot during reviews).
Issue Details:
--------------The Long branch stub in the trampoline implementation[1] provides flexibility to handles short as well as long branch distance to actual trampoline. Whereas, the 8 bytes long dummy_tramp_addr field sitting before long branch stub leads to failure when enabling verifier based seltest for ppc64. The verifier selftests require disassembing the final jited image to get native instructions. Later the disassembled instruction sequence is matched against sequence of instructions provided in test-file under __jited() wrapper. The final jited image contains Out-of-line stub and Long branch stub as part of epilogue jitting for a bpf program. The 8 bytes space for dummy_tramp is sandwiched between both above mentioned stubs. These 8 bytes contain memory address of dummy trampoline during trampoline invocation which don't correspond to any powerpc instructions. So, disassembly fails resulting in failure of verifier selftests. The following code snippet shows the problem with current arrangement made for dummy_tramp_addr. /* Out-of-line stub */ mflr r0 [b|bl] tramp mtlr r0 //only with OOL b bpf_func + 4 /* Long branch stub */ .long <dummy_tramp_addr> <---Invalid bytes sequence, disassembly fails mflr r11 bcl 20,31,$+4 mflr r12 ld r12, -8-SZL(r12) mtctr r12 mtlr r11 //retain ftrace ABI bctr Consider test program binary of size 112 bytes: 0: 00000060 10004de8 00002039 f8ff21f9 81ff21f8 7000e1fb 3000e13b 28: 3000e13b 2a006038 f8ff7ff8 00000039 7000e1eb 80002138 7843037d 56: 2000804e a602087c 00000060 a603087c bcffff4b c0341d00 000000c0 84: a602687d 05009f42 a602887d f0ff8ce9 a603897d a603687d 2004804e Disassembly output of above binary for ppc64le: pc:0 left:112 00 00 00 60 : nop pc:4 left:108 10 00 4d e8 : ld 2, 16(13) pc:8 left:104 00 00 20 39 : li 9, 0 pc:12 left:100 f8 ff 21 f9 : std 9, -8(1) pc:16 left:96 81 ff 21 f8 : stdu 1, -128(1) pc:20 left:92 70 00 e1 fb : std 31, 112(1) pc:24 left:88 30 00 e1 3b : addi 31, 1, 48 pc:28 left:84 30 00 e1 3b : addi 31, 1, 48 pc:32 left:80 2a 00 60 38 : li 3, 42 pc:36 left:76 f8 ff 7f f8 : std 3, -8(31) pc:40 left:72 00 00 00 39 : li 8, 0 pc:44 left:68 70 00 e1 eb : ld 31, 112(1) pc:48 left:64 80 00 21 38 : addi 1, 1, 128 pc:52 left:60 78 43 03 7d : mr 3, 8 pc:56 left:56 20 00 80 4e : blr pc:60 left:52 a6 02 08 7c : mflr 0 pc:64 left:48 00 00 00 60 : nop pc:68 left:44 a6 03 08 7c : mtlr 0 pc:72 left:40 bc ff ff 4b : b .-68 pc:76 left:36 c0 34 1d 00 : ... Failure log: Can't disasm instruction at offset 76: c0 34 1d 00 00 00 00 c0 a6 02 68 7d 05 00 9f 42 -------------------------------------- Observation: Can't disasm instruction at offset 76 as this address has ".long <dummy_tramp_addr>" (0xc0341d00000000c0) But valid instructions follow at offset 84 onwards. Move the long branch address space to the bottom of the long branch stub. This allows uninterrupted disassembly until the last 8 bytes. Exclude these last bytes from the overall program length to prevent failure in assembly generation. Following is disassembler output for same test program with moved down dummy_tramp_addr field: ..... ..... pc:68 left:44 a6 03 08 7c : mtlr 0 pc:72 left:40 bc ff ff 4b : b .-68 pc:76 left:36 a6 02 68 7d : mflr 11 pc:80 left:32 05 00 9f 42 : bcl 20, 31, .+4 pc:84 left:28 a6 02 88 7d : mflr 12 pc:88 left:24 14 00 8c e9 : ld 12, 20(12) pc:92 left:20 a6 03 89 7d : mtctr 12 pc:96 left:16 a6 03 68 7d : mtlr 11 pc:100 left:12 20 04 80 4e : bctr pc:104 left:8 c0 34 1d 00 : Failure log: Can't disasm instruction at offset 104: c0 34 1d 00 00 00 00 c0 --------------------------------------- Disassembly logic can truncate at 104, ignoring last 8 bytes. Update the dummy_tramp_addr field offset calculation from the end of the program to reflect its new location, for bpf_arch_text_poke() to update the actual trampoline's address in this field. [1] https://lore.kernel.org/all/[email protected]
v8->v9:
Dynamic pass handling until code keeps shrinking Fix private stack memory leak
v7->v8:
Fixed bot identified issues ofalt_exit_addrandBPF_EXITFixed 32-bit ppc function signature mismatch
v6->v7:
Fixed JIT buffer overflow in case of large BPF progs Addressed remaining bot comments
v5->v6:
Changed alignment NOP emittion dependency on fimage layout Adjust tail truncate length for 32-bit ppc Addressed few minor bot comments
v4->v5:
Handled alignment NOP emit logic and corresponding stub offsets Handled image buffer overflow problem in last pass Above changes took care of other bot reviews IncludedLLVMDisposeMessage()for graceful freeing Adjusted parameters inbpf_jit_build_fentry_stubsfor ppc32 Adjusted expected JIT inst. in tailcall test forCONFIG_PPC_KERNEL_PCRELconfig
Added fix patch at last for inaccurate use of cmplwi inst.
v3->v4:
Changed logic for emitting alignment NOP
v2->v3:
Removed fixed NOP from bottom of long branch stub Rebased on top of bpf-next
v1->v2:
Added fix-patch to correct memory alignment in-place Moved the optional alignmnet NOP before OOL stub[v1]: https://lore.kernel.org/bpf/[email protected]
[v2]: https://lore.kernel.org/bpf/[email protected]
[v3]: https://lore.kernel.org/bpf/[email protected]
[v4]: https://lore.kernel.org/bpf/[email protected]
[v5]: https://lore.kernel.org/bpf/[email protected]
[v6]: https://lore.kernel.org/bpf/[email protected]
[v7]: https://lore.kernel.org/bpf/[email protected]
[v8]: https://lore.kernel.org/bpf/[email protected]Abhishek Dubey (8):
powerpc/bpf: fix alignment of long branch trampoline address powerpc/bpf: Move outdummy_tramp_addrafter Long branch stub selftest/bpf: Fixing powerpc JIT disassembly failure selftest/bpf: Enable verifier selftest for powerpc64 powerpc64/bpf: fix compare instruction emitted for tailcall selftest/bpf: Add tailcall verifier selftest for powerpc64 powerpc/bpf: fix buffer overflow in JIT for large BPF programs powerpc64/bpf: fix percpu private stack leak on JIT failurearch/powerpc/net/bpf_jit.h | 20 +++- arch/powerpc/net/bpf_jit_comp.c | 99 ++++++++++++++----- arch/powerpc/net/bpf_jit_comp32.c | 7 +- arch/powerpc/net/bpf_jit_comp64.c | 15 +-- .../selftests/bpf/jit_disasm_helpers.c | 27 ++++- tools/testing/selftests/bpf/progs/bpf_misc.h | 1 + .../bpf/progs/verifier_tailcall_jit.c | 69 +++++++++++++ tools/testing/selftests/bpf/test_loader.c | 5 + 8 files changed, 203 insertions(+), 40 deletions(-)