bpf: Run exception cleanup landing pads when bpf_throw() unwinds
Yonghong Song
reviewerv2 cover bpf: Run exception cleanup landing pads when bpf_throw() unwinds
bpf_throw() walks the BPF call stack to the exception boundary and discards every frame in between. A frame that owns something -- an RCU read lock, a preemption-disabled section, a referenced kptr -- never gets to give it back, so the verifier refuses to let such a frame throw at all. That is the whole reason a Rust program cannot use bpf_throw() as its panic path right now: Rust's Drop glue *is* that give-back, and there is nowhere to run it.
LLVM 23 added the compiler half ([1]). A Rust function that owns a value across a call that can unwind
fn foo() {
let _guard = RcuReadGuard::new(); /* bpf_rcu_read_lock() */
may_throw(); /* extern "C-unwind" */
} /* Drop: rcu_read_unlock */lowers to an invoke with a cleanup landing pad holding the Drop call, and the BPF backend writes one record per invoke region into a .bpf_cleanup
section: a flat table of 12-byte (begin, end, landing_pad) triples, each field a byte offset into the code section. The rule is "if a frame unwinds with its return address in [begin, end), run landing_pad before discarding it". A pad ends with a call to _Unwind_Resume(), which the kernel provides as the bpf_unwind_resume() kfunc.
This series is the kernel half: take that table at BPF_PROG_LOAD, teach the verifier that a covered call can also go to its landing pad, and have bpf_throw() run the pads as it walks.
C has no unwinding, so the selftests spell out by hand what a frontend emits -- a call site bracketed by two labels, a landing pad, and a record tying them together. The frame above, written that way:
"call bpf_rcu_read_lock;"
"1:" "call foo3;" /* cleanup region */
"2:"
... normal path, ends in bpf_rcu_read_unlock ...
"6:" /* landing pad */
"call bpf_rcu_read_unlock;"
"call bpf_unwind_resume;" CLEANUP_REC("1b", "2b", "6b")
Right now that program does not load: the throw inside foo3 is reported as "bpf_throw cannot be used inside bpf_rcu_read_lock-ed region", because as far as the verifier is concerned nothing will ever unlock. With the series the verifier walks the unwind the same way the run time will -- into the pad, which unlocks, then on to the next frame -- and the program both loads and releases the lock when it throws.
Design
======
A pad is run, not lowered. bpf_throw() already walks the frames with arch_bpf_stack_walk(); it now looks each frame's return address up in that (sub)program's table and calls the pad as a subroutine of the walker, with the unwinding frame's frame pointer and its callee-saved registers restored from the spill its callee's prologue left. The pad therefore sees its own frame but runs on the walker's stack, far below it, so nothing it calls can disturb the frame it is cleaning up after. The JIT turns its bpf_unwind_resume() into the way back to the walker.
The verifier walks the same thing, step for step, so the resource rules are unchanged: whatever a pad releases is released in the verifier state too, and check_resource_leak() simply moves from "a throw was seen" to the end of the walk.
1-2 uapi: cleanup_info in BPF_PROG_LOAD, struct bpf_cleanup_info,
and the bpf_unwind_resume() kfunc
3-9 verifier: mark the covered call sites, give them an edge to the
pad, walk the unwind, and refuse the shapes that cannot be dispatched
10 bpf_throw(): dispatch pads while walking 11-12 x86-64 and arm64 JITs 13-17 libbpf: collect .bpf_cleanup, pass it to the kernel, resolve
_Unwind_Resume, carry it through the light skeleton and the static linker
18-20 selftests
Limitations
===========
- Cleanup pads only. A catch pad -- one that ends in a plain exit rather than a resume, which is what Rust's catch_unwind would need -- is
refused: the walker calls a pad as a subroutine and cannot hand a frame back its own execution. LLVM refuses type-specific catches and filters on its side as well.
- A JIT that can dispatch pads is required: x86-64 (with CONFIG_UNWINDER_ORC, which bpf_throw() already needs there) and arm64. Anywhere else the load fails with -EOPNOTSUPP rather than silently doing nothing.
- No offloaded programs, no private stack, and no combining a table with an exception callback.
- In a pad body: no tail call, no indirect jump, and no on-stack call arguments -- all of them read or write a stack the pad does not own.
- The Rust toolchain does not properly support BPF exception handling yet. The tables the selftests use are hand-written inline asm, which the assembler turns into the same relocations the BPF AsmPrinter emits, so libbpf and the kernel see an object indistinguishable from a compiler-generated one.
[1] https://github.com/llvm/llvm-project/pull/192164
llvm commit 9d51c891b719 ("[BPF] Add exception handling support with .bpf_cleanup section")
Changelog
=========
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Consolidate all usages of kern_extern_name() in a single patch in libbpf.
- Avoid compiler warning and add proper cleanup_info_cnt guard in libbpf when collecting .bpf_cleanup records.
- Add cleanup_info_cnt condition for emit_rel_store() with cleanup_info.
Yonghong Song (20):
bpf: Accept the compiler's exception cleanup table at program load
bpf: Add the bpf_unwind_resume() kfunc
bpf: Add lookups for exception cleanup resumes and landing pads
bpf: Mark the call sites an exception cleanup table covers
bpf: Make exception landing pads reachable in the CFG
bpf: Explore the landing pads no call site reaches
bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch
bpf: Walk the exception unwind in the verifier
bpf: Refuse a private stack for a program with an exception cleanup
table
bpf: Dispatch exception cleanup pads from bpf_throw()
bpf, x86: Dispatch exception cleanup pads at run time
bpf, arm64: Dispatch exception cleanup pads at run time
libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
libbpf: Add cleanup_info to bpf_prog_load_opts
libbpf: Collect .bpf_cleanup records and pass them to the kernel
libbpf: Carry the exception cleanup table through the light skeleton
libbpf: Let the static linker carry .bpf_cleanup relocations selftests/bpf: Add an end-to-end .bpf_cleanup exception test selftests/bpf: Cover the exception cleanup shapes the chain does not
reach
selftests/bpf: Load an exception cleanup program from a light skeleton
arch/arm64/net/Makefile | 2 +- arch/arm64/net/bpf_cleanup_pad.S | 95 ++ arch/arm64/net/bpf_jit_comp.c | 97 +- arch/x86/net/Makefile | 2 +- arch/x86/net/bpf_cleanup_pad.S | 74 ++ arch/x86/net/bpf_jit_comp.c | 78 +- include/linux/bpf.h | 75 ++ include/linux/bpf_cleanup_abi.h | 16 + include/linux/bpf_verifier.h | 11 + include/linux/filter.h | 2 + include/uapi/linux/bpf.h | 9 + kernel/bpf/Makefile | 2 +- kernel/bpf/cfg.c | 64 +- kernel/bpf/check_btf.c | 147 +++ kernel/bpf/core.c | 35 +- kernel/bpf/exception.c | 654 +++++++++++++ kernel/bpf/exception.h | 22 + kernel/bpf/fixups.c | 143 +++ kernel/bpf/helpers.c | 46 + kernel/bpf/liveness.c | 20 + kernel/bpf/states.c | 3 + kernel/bpf/syscall.c | 2 +- kernel/bpf/verifier.c | 125 ++- tools/include/uapi/linux/bpf.h | 9 + tools/lib/bpf/bpf.c | 6 +- tools/lib/bpf/bpf.h | 7 +- tools/lib/bpf/gen_loader.c | 29 +- tools/lib/bpf/libbpf.c | 312 ++++++- tools/lib/bpf/libbpf_internal.h | 10 + tools/lib/bpf/linker.c | 19 +- tools/testing/selftests/bpf/Makefile | 2 +- .../selftests/bpf/exceptions_cleanup.h | 52 ++ .../bpf/prog_tests/exceptions_cleanup.c | 421 +++++++++ .../selftests/bpf/progs/exceptions_cleanup.c | 152 +++ .../bpf/progs/exceptions_cleanup_ext_table.c | 48 + .../bpf/progs/exceptions_cleanup_fail.c | 600 ++++++++++++ .../bpf/progs/exceptions_cleanup_freplace.c | 17 + .../bpf/progs/exceptions_cleanup_light.c | 39 + .../progs/exceptions_cleanup_pad_freplace.c | 17 + .../bpf/progs/exceptions_cleanup_shapes.c | 863 ++++++++++++++++++ 40 files changed, 4263 insertions(+), 64 deletions(-) create mode 100644 arch/arm64/net/bpf_cleanup_pad.S create mode 100644 arch/x86/net/bpf_cleanup_pad.S create mode 100644 include/linux/bpf_cleanup_abi.h create mode 100644 kernel/bpf/exception.c create mode 100644 kernel/bpf/exception.h create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v2 1/20 bpf: Accept the compiler's exception cleanup table at program load
LLVM 23 added exception handling support for BPF with the .bpf_cleanup section ([1]). Rust code compiled with panic=unwind runs cleanup code (Drop glue) when bpf_throw() fires, and the LLVM BPF backend emits that section from the landing pads the frontend produced. Plain C cannot generate .bpf_cleanup unless inline asm is used. The Rust compiler does not *properly* support BPF exception handling yet, but the kernel can support the table today, and inline assembly is enough to test it.
Add the UAPI to carry the .bpf_cleanup table into the kernel. BPF_PROG_LOAD grows cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, and struct bpf_cleanup_info describes one record as a triple of instruction indices:
the half-open call-site range [begin_off, end_off) and the landing_pad_off the frame resumes at. check_cleanup_info() validates the table a program is loaded with, so the rest of the kernel can rely on it.
[1] https://github.com/llvm/llvm-project/pull/192164
--- include/linux/bpf_verifier.h | 2 + include/uapi/linux/bpf.h | 9 ++ kernel/bpf/check_btf.c | 147 +++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 2 +- kernel/bpf/verifier.c | 1 + tools/include/uapi/linux/bpf.h | 9 ++ 6 files changed, 169 insertions(+), 1 deletion(-)
include/linux/bpf_verifier.h +2 -0
@@ -987,6 +987,8 @@ struct bpf_verifier_env {struct arg_track **callsite_at_stack;u32 pass_cnt;/* number of times do_check() was called */ u32 subprog_cnt; +struct bpf_cleanup_info *cleanup_info; +u32 cleanup_info_cnt;/* number of instructions analyzed by the verifier */ u32 prev_insn_processed, insn_processed;/* number of jmps, calls, exits analyzed so far */
include/uapi/linux/bpf.h +9 -0
@@ -1669,6 +1669,9 @@ union bpf_attr {* verification. */ __s32 keyring_id; +__aligned_u64 cleanup_info;/* exception cleanup table */ +__u32 cleanup_info_rec_size;/* userspace bpf_cleanup_info size */ +__u32 cleanup_info_cnt;/* number of bpf_cleanup_info records */ };struct {/* anonymous struct used by BPF_OBJ_* commands */
@@ -7588,6 +7591,12 @@ struct bpf_line_info {__u32 line_col; }; +struct bpf_cleanup_info { +__u32 begin_off; +__u32 end_off; +__u32 landing_pad_off; +}; +struct bpf_spin_lock {__u32 val; };
kernel/bpf/check_btf.c +147 -0
159 lines not on this page · raw message
kernel/bpf/syscall.c +1 -1
@@ -2912,7 +2912,7 @@ int __init __used bpf_multi_func(void) { return 0; }BTF_ID_LIST_GLOBAL_SINGLE (bpf_multi_func_btf_id, func, bpf_multi_func) /* last field in 'union bpf_attr' used by this command */ -#define BPF_PROG_LOAD_LAST_FIELD keyring_id +#define BPF_PROG_LOAD_LAST_FIELD cleanup_info_cntstatic int bpf_prog_load (union bpf_attr *attr,bpfptr_t uattr,struct bpf_log_attr *attr_log) {
kernel/bpf/verifier.c +1 -0
@@ -21845,6 +21845,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,kvfree (env->succ);kvfree (env->gotox_tmp_buf);bpf_diag_free (env); +kvfree (env->cleanup_info);kvfree (env);return ret; }
tools/include/uapi/linux/bpf.h +9 -0
@@ -1669,6 +1669,9 @@ union bpf_attr {* verification. */ __s32 keyring_id; +__aligned_u64 cleanup_info;/* exception cleanup table */ +__u32 cleanup_info_rec_size;/* userspace bpf_cleanup_info size */ +__u32 cleanup_info_cnt;/* number of bpf_cleanup_info records */ };struct {/* anonymous struct used by BPF_OBJ_* commands */
@@ -7588,6 +7591,12 @@ struct bpf_line_info {__u32 line_col; }; +struct bpf_cleanup_info { +__u32 begin_off; +__u32 end_off; +__u32 landing_pad_off; +}; +struct bpf_spin_lock {__u32 val; };
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v2 2/20 bpf: Add the bpf_unwind_resume() kfunc
In reply to Yonghong SongA compiler-emitted cleanup landing pad ends with a call to _Unwind_Resume(), which carries the unwind on once that frame's cleanups have run. The kernel provides the same terminator as a kfunc, named bpf_unwind_resume() to keep the 'bpf_' prefix kfunc convention; a later libbpf patch resolves the compiler's name to it.
The body never runs. A JIT emits the way back out of a landing pad in its place.
--- kernel/bpf/helpers.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/bpf/helpers.c +12 -0
@@ -3407,6 +3407,17 @@ __bpf_kfunc void bpf_throw(u64 cookie)WARN (1 ,"A call to BPF exception callback should never return\n" ); } +/* +* Terminator of a compiler-emitted cleanup landing pad. The compiler names +* this _Unwind_Resume, the base unwind ABI's entry point for carrying an +* unwind on once a frame's cleanups have run. To match kernel kfunc +* convention, the kernel calls it bpf_unwind_resume and libbpf maps the +* compiler's name onto it. +*/ +__bpf_kfunc void bpf_unwind_resume(void) +{ +} + __bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, unsigned int flags) {struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
@@ -4853,6 +4864,7 @@ BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_throw) +BTF_ID_FLAGS(func, bpf_unwind_resume)#ifdef CONFIG_BPF_EVENTS BTF_ID_FLAGS(func, bpf_send_signal_task)#endif
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v2 3/20 bpf: Add lookups for exception cleanup resumes and landing pads
In reply to Yonghong SongAdd two new files, exception.h and exception.c, to host the exception handling code. Only a few helpers so far: recognising a call to bpf_unwind_resume(), and asking which landing pad, if any, a call site unwinds to.
The pad of a call site is kept in insn_aux_data, so the two places that move instructions around -- bpf_patch_insn_data() and verifier_remove_insns() -- learn to keep it in step. Nothing sets the mark yet; the next patch does.
--- include/linux/bpf_verifier.h | 6 ++++++ kernel/bpf/Makefile | 2 +- kernel/bpf/exception.c | 36 ++++++++++++++++++++++++++++++++++++ kernel/bpf/exception.h | 12 ++++++++++++ kernel/bpf/fixups.c | 17 +++++++++++++++++ 5 files changed, 72 insertions(+), 1 deletion(-) create mode 100644 kernel/bpf/exception.c create mode 100644 kernel/bpf/exception.h
include/linux/bpf_verifier.h +6 -0
@@ -681,6 +681,11 @@ struct bpf_insn_aux_data {bool needs_zext;/* alu op needs to clear upper bits */ bool non_sleepable;/* helper/kfunc may be called from non-sleepable context */ bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ +/* +* 1 + the instruction index of the exception cleanup landing pad this +* call site unwinds to, or 0 for none. +*/ +u32 cleanup_pad;bool call_with_percpu_alloc_ptr;/* {this,per}_cpu_ptr() with prog percpu alloc */ u8 alu_state;/* used in combination with alu_limit */ /* true if STX or LDX instruction is a part of a spill/fill
@@ -1518,6 +1523,7 @@ u32 btf_func_arg_align(const struct btf *btf, const struct btf_type *t);int bpf_find_subprog (struct bpf_verifier_env *env,int off);bool bpf_is_throw_kfunc (struct bpf_insn *insn); +bool bpf_is_unwind_resume_kfunc (const struct bpf_insn *insn);int bpf_compute_const_regs (struct bpf_verifier_env *env);int bpf_prune_dead_branches (struct bpf_verifier_env *env);int bpf_check_cfg (struct bpf_verifier_env *env);
kernel/bpf/Makefile +1 -1
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_ obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o -obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o +obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o exception.o obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o obj-$(CONFIG_BPF_JIT) += trampoline.o
kernel/bpf/exception.c new file +36 -0
@@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <linux/bpf.h> +#include <linux/bpf_verifier.h> +#include <linux/btf.h> +#include <linux/btf_ids.h> +#include <linux/filter.h> +#include <linux/slab.h> +#include "exception.h" + +#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args) + +enum exc_kfunc { + EXC_KF_bpf_unwind_resume, +}; + +BTF_ID_LIST (exc_kfunc_list) +BTF_ID (func, bpf_unwind_resume) + +static bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf) +{ + return bpf_pseudo_kfunc_call(insn) && insn->off == 0 && + insn->imm == exc_kfunc_list[kf]; +} + +bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn) +{ +return insn_is_exc_kfunc (insn, EXC_KF_bpf_unwind_resume); +} + +int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx) +{ +u32 pad = env->insn_aux_data[idx].cleanup_pad; + +return pad ? (int )pad -1 :-1 ; +}
kernel/bpf/exception.h new file +12 -0
@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#ifndef _LINUX_BPF_EXCEPTION_H +#define _LINUX_BPF_EXCEPTION_H + +#include <linux/types.h> + +struct bpf_verifier_env ; + +int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx); + +#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +17 -0
@@ -261,6 +261,11 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env, } } +if (env->cleanup_info_cnt) +for (i =0 ; i < prog_len; i++) +if (data[i].cleanup_pad > off +1 ) + data[i].cleanup_pad += cnt -1 ; +/* * Last slot instruction could be a newly generated * BPF_ST/BPF_LDX/BPF_STX, systematically mark it for non-stack access
@@ -549,6 +554,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)struct bpf_insn_aux_data *aux_data = env->insn_aux_data;unsigned int orig_prog_len = env->prog->len;int err; +u32 i;if (bpf_prog_is_offloaded (env->prog->aux))bpf_prog_offload_remove_insns (env, off, cnt);
@@ -573,6 +579,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) sizeof(*aux_data) * (orig_prog_len - off - cnt)); env->insn_aux_data_len -= cnt; +if (env->cleanup_info_cnt) { +for (i =0 ; i < env->insn_aux_data_len; i++) { +u32 pad = aux_data[i].cleanup_pad; + +if (pad > off + cnt) + aux_data[i].cleanup_pad = pad - cnt; +else if (pad > off) + aux_data[i].cleanup_pad =0 ; + } + } +return 0 ; }
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 03/20] bpf: Add lookups for exception cleanup resumes and landing pads
Can this logic handle NOP removal correctly? When bpf_opt_remove_nops() calls verifier_remove_insns(env, i, 1) to remove a single fall-through goto +0, if a cleanup record names that instruction as…
Quoted 15 lines
include/linux/bpf_verifier.h +5
-0@@ -681,6 +681,11 @@ struct bpf_insn_aux_data {bool needs_zext;/* alu op needs to clear upper bits */ bool non_sleepable;/* helper/kfunc may be called from non-sleepable context */ bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ +/* +* 1 + the instruction index of the exception cleanup landing pad this +* call site unwinds to, or 0 for none. +*/ +u32 cleanup_pad;bool call_with_percpu_alloc_ptr;/* {this,per}_cpu_ptr() with prog percpu alloc */ u8 alu_state;/* used in combination with alu_limit */
[ ... ]
Quoted 13 lines
kernel/bpf/fixups.c +5
-0@@ -261,6 +261,11 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env, } } +if (env->cleanup_info_cnt) +for (i =0 ; i < prog_len; i++) +if (data[i].cleanup_pad > off +1 ) + data[i].cleanup_pad += cnt -1 ; +
[ ... ]
Quoted 15 lines
kernel/bpf/fixups.c +4
-0@@ -549,6 +554,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)struct bpf_insn_aux_data *aux_data = env->insn_aux_data;unsigned int orig_prog_len = env->prog->len;int err; +u32 i;if (bpf_prog_is_offloaded (env->prog->aux))bpf_prog_offload_remove_insns (env, off, cnt);@@ -573,6 +579,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) sizeof(*aux_data) * (orig_prog_len - off - cnt)); env->insn_aux_data_len -= cnt; +if (env->cleanup_info_cnt) { +for (i =0 ; i < env->insn_aux_data_len; i++) { +u32 pad = aux_data[i].cleanup_pad;
kernel/bpf/fixups.c static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) · line 594
Quoted 10 lines
kernel/bpf/fixups.c +8
-0+ +if (pad > off + cnt) + aux_data[i].cleanup_pad = pad - cnt; +else if (pad > off) + aux_data[i].cleanup_pad =0 ; + } + } +return 0 ; }
Can this logic handle NOP removal correctly? When bpf_opt_remove_nops() calls verifier_remove_insns(env, i, 1) to remove a single fall-through goto +0, if a cleanup record names that instruction as its landing pad, then pad == off + 1, so pad > off is true and cleanup_pad becomes 0.
But removing the NOP leaves the pad's next instruction at the same index, so the correct landing pad index stays unchanged rather than being zeroed. For example:
off: some_call with cleanup_pad = off+1
off+1: goto +0 (NOP landing pad)
off+2: actual cleanup codeAfter removing the NOP at off+1:
off: some_call (cleanup_pad should still be off+1)
off+1: actual cleanup code (was at off+2)
The else if (pad > off) arm erases the record instead of preserving it. Nothing rejects NOP landing pads during validation - a goto +0 has code 0x05, not 0x00, so check_cleanup_info() accepts it.
The ordering makes the loss escape validation: bpf_check_cleanup_exceptions() runs before bpf_opt_remove_nops(), so the verifier proves the cleanup path releases acquired references, but then the NOP removal zeroes cleanup_pad, and later cleanup_info_for_subprog() sees if (!pad) continue; and omits the record from the runtime table. The program is accepted but bpf_throw() finds no pad for that call site and skips the cleanup, leaking whatever the pad was supposed to release.
Does the else if (pad > off) arm need to distinguish removals that consume the landing pad instruction itself versus removals that happen before it?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Re: [PATCH bpf-next v2 03/20] bpf: Add lookups for exception cleanup resumes and landing pads
Quoted 90 lines · bot+bpf-ci
Quoted 15 lines
include/linux/bpf_verifier.h +5
-0@@ -681,6 +681,11 @@ struct bpf_insn_aux_data {bool needs_zext;/* alu op needs to clear upper bits */ bool non_sleepable;/* helper/kfunc may be called from non-sleepable context */ bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ +/* +* 1 + the instruction index of the exception cleanup landing pad this +* call site unwinds to, or 0 for none. +*/ +u32 cleanup_pad;bool call_with_percpu_alloc_ptr;/* {this,per}_cpu_ptr() with prog percpu alloc */ u8 alu_state;/* used in combination with alu_limit */ [ ... ]
Quoted 13 lines
kernel/bpf/fixups.c +5
-0@@ -261,6 +261,11 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env, } } +if (env->cleanup_info_cnt) +for (i =0 ; i < prog_len; i++) +if (data[i].cleanup_pad > off +1 ) + data[i].cleanup_pad += cnt -1 ; +[ ... ]
Quoted 15 lines
kernel/bpf/fixups.c +4
-0@@ -549,6 +554,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)struct bpf_insn_aux_data *aux_data = env->insn_aux_data;unsigned int orig_prog_len = env->prog->len;int err; +u32 i;if (bpf_prog_is_offloaded (env->prog->aux))bpf_prog_offload_remove_insns (env, off, cnt);@@ -573,6 +579,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) sizeof(*aux_data) * (orig_prog_len - off - cnt)); env->insn_aux_data_len -= cnt; +if (env->cleanup_info_cnt) { +for (i =0 ; i < env->insn_aux_data_len; i++) { +u32 pad = aux_data[i].cleanup_pad;kernel/bpf/fixups.c static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) · line 594
Quoted 10 lines
kernel/bpf/fixups.c +8
-0+ +if (pad > off + cnt) + aux_data[i].cleanup_pad = pad - cnt; +else if (pad > off) + aux_data[i].cleanup_pad =0 ; + } + } +return 0 ; }Can this logic handle NOP removal correctly? When
bpf_opt_remove_nops()callsverifier_remove_insns(env, i, 1)to remove a single fall-through goto +0, if a cleanup record names that instruction as its landing pad, then pad == off + 1, so pad > off is true andcleanup_padbecomes 0.But removing the NOP leaves the pad's next instruction at the same index, so the correct landing pad index stays unchanged rather than being zeroed. For example:
off: some_call with cleanup_pad = off+1 off+1: goto +0 (NOP landing pad) off+2: actual cleanup codeAfter removing the NOP at off+1:
off:
some_call(cleanup_padshould still be off+1)
off+1: actual cleanup code (was at off+2)The
else if (pad > off)arm erases the record instead of preserving it. Nothing rejects NOP landing pads during validation - a goto +0 has code0x05, not0x00, socheck_cleanup_info()accepts it.The ordering makes the loss escape validation:
bpf_check_cleanup_exceptions()runs beforebpf_opt_remove_nops(), so the verifier proves the cleanup path releases acquired references, but then the NOP removal zeroescleanup_pad, and latercleanup_info_for_subprog()seesif (!pad) continue;and omits the record from the runtime table. The program is accepted butbpf_throw()finds no pad for that call site and skips the cleanup, leaking whatever the pad was supposed to release.
Quoted 3 lines
Does the
else if (pad > off)arm need to distinguish removals that consume the landing pad instruction itself versus removals that happen before it?
bot is correct here.
pls fix
pw-bot: cr
Yonghong Song
reviewerRe: [PATCH bpf-next v2 03/20] bpf: Add lookups for exception cleanup resumes and landing pads
Quoted 88 lines · Alexei Starovoitov
Quoted 87 lines · bot+bpf-ci
Quoted 15 lines
include/linux/bpf_verifier.h +5
-0@@ -681,6 +681,11 @@ struct bpf_insn_aux_data {bool needs_zext;/* alu op needs to clear upper bits */ bool non_sleepable;/* helper/kfunc may be called from non-sleepable context */ bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ +/* +* 1 + the instruction index of the exception cleanup landing pad this +* call site unwinds to, or 0 for none. +*/ +u32 cleanup_pad;bool call_with_percpu_alloc_ptr;/* {this,per}_cpu_ptr() with prog percpu alloc */ u8 alu_state;/* used in combination with alu_limit */ [ ... ]
Quoted 13 lines
kernel/bpf/fixups.c +5
-0@@ -261,6 +261,11 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env, } } +if (env->cleanup_info_cnt) +for (i =0 ; i < prog_len; i++) +if (data[i].cleanup_pad > off +1 ) + data[i].cleanup_pad += cnt -1 ; +[ ... ]
Quoted 15 lines
kernel/bpf/fixups.c +4
-0@@ -549,6 +554,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)struct bpf_insn_aux_data *aux_data = env->insn_aux_data;unsigned int orig_prog_len = env->prog->len;int err; +u32 i;if (bpf_prog_is_offloaded (env->prog->aux))bpf_prog_offload_remove_insns (env, off, cnt);@@ -573,6 +579,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) sizeof(*aux_data) * (orig_prog_len - off - cnt)); env->insn_aux_data_len -= cnt; +if (env->cleanup_info_cnt) { +for (i =0 ; i < env->insn_aux_data_len; i++) { +u32 pad = aux_data[i].cleanup_pad;kernel/bpf/fixups.c static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) · line 594
Quoted 10 lines
kernel/bpf/fixups.c +8
-0+ +if (pad > off + cnt) + aux_data[i].cleanup_pad = pad - cnt; +else if (pad > off) + aux_data[i].cleanup_pad =0 ; + } + } +return 0 ; }Can this logic handle NOP removal correctly? When
bpf_opt_remove_nops()callsverifier_remove_insns(env, i, 1)to remove a single fall-through goto +0, if a cleanup record names that instruction as its landing pad, then pad == off + 1, so pad > off is true andcleanup_padbecomes 0.But removing the NOP leaves the pad's next instruction at the same index, so the correct landing pad index stays unchanged rather than being zeroed. For example:
off: some_call with cleanup_pad = off+1 off+1: goto +0 (NOP landing pad) off+2: actual cleanup codeAfter removing the NOP at off+1:
off:
some_call(cleanup_padshould still be off+1)
off+1: actual cleanup code (was at off+2)The
else if (pad > off)arm erases the record instead of preserving it. Nothing rejects NOP landing pads during validation - a goto +0 has code0x05, not0x00, socheck_cleanup_info()accepts it.The ordering makes the loss escape validation:
bpf_check_cleanup_exceptions()runs beforebpf_opt_remove_nops(), so the verifier proves the cleanup path releases acquired references, but then the NOP removal zeroescleanup_pad, and latercleanup_info_for_subprog()seesif (!pad) continue;and omits the record from the runtime table. The program is accepted butbpf_throw()finds no pad for that call site and skips the cleanup, leaking whatever the pad was supposed to release.
Quoted 5 lines
Quoted 3 lines
Does the
else if (pad > off)arm need to distinguish removals that consume the landing pad instruction itself versus removals that happen before it?bot is correct here.
pls fix
Agree with bot about for two different cases. Will fix.
Quoted 2 lines
pw-bot: cr
Yonghong Song
reviewerPATCH v2 4/20 bpf: Mark the call sites an exception cleanup table covers
In reply to Yonghong SongSome plumbing work is done before bpf_check_cfg(). More specifically, insn_aux_data records cleanup_throw_site for every bpf_throw(), and cleanup_pad, the landing pad a frame resumes at, for every call within the [begin_off, end_off) range of a cleanup record. Subsequent commits consume both.
bpf_prepare_cleanup_exceptions() runs before bpf_check_cfg(), because what it produces is what the CFG walk consumes. It refuses a table on an offloaded program, on a program whose JIT cannot dispatch landing pads or which the JIT was not asked to compile, and on a program that also installs an exception callback -- two different answers to what runs on the way out. bpf_jit_supports_cleanup_pads() is weak here and says no; the arch patches provide the real ones.
--- include/linux/bpf_verifier.h | 1 + include/linux/filter.h | 1 + kernel/bpf/core.c | 5 ++++ kernel/bpf/exception.c | 55 ++++++++++++++++++++++++++++++++++++ kernel/bpf/exception.h | 1 + kernel/bpf/verifier.c | 6 ++++ 6 files changed, 69 insertions(+)
include/linux/bpf_verifier.h +1 -0
@@ -681,6 +681,7 @@ struct bpf_insn_aux_data {bool needs_zext;/* alu op needs to clear upper bits */ bool non_sleepable;/* helper/kfunc may be called from non-sleepable context */ bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ +bool cleanup_throw_site;/* call to bpf_throw() */ /* * 1 + the instruction index of the exception cleanup landing pad this * call site unwinds to, or 0 for none.
include/linux/filter.h +1 -0
@@ -1242,6 +1242,7 @@ bool bpf_jit_supports_stack_args(void);bool bpf_jit_supports_arena_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void ); +bool bpf_jit_supports_cleanup_pads (void );bool bpf_jit_supports_ptr_xchg (void );bool bpf_jit_supports_arena (void );bool bpf_jit_supports_insn (struct bpf_insn *insn,bool in_arena);
kernel/bpf/core.c +5 -0
@@ -3470,6 +3470,11 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, { } +bool __weak bpf_jit_supports_cleanup_pads (void ) +{ +return false; +} +bool __weak bpf_jit_supports_timed_may_goto (void ) {return false;
kernel/bpf/exception.c +55 -0
61 lines not on this page · raw message
kernel/bpf/exception.h +1 -0
@@ -7,6 +7,7 @@struct bpf_verifier_env ; +int bpf_prepare_cleanup_exceptions (struct bpf_verifier_env *env);int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx);#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/verifier.c +6 -0
@@ -37,6 +37,7 @@#include "diagnostics.h" #include "disasm.h" +#include "exception.h" static const struct bpf_verifier_ops *const bpf_verifier_ops[] = { #define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
@@ -21638,6 +21639,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,if (ret <0 ) goto skip_full_check; +/* The CFG needs an edge from a call in a cleanup range to its pad. */ + ret =bpf_prepare_cleanup_exceptions (env); +if (ret <0 ) + goto skip_full_check; +/* Validate instructions and resolve the program's referenced resources. */ ret =check_and_resolve_insns (env);if (ret <0 )
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v2 5/20 bpf: Make exception landing pads reachable in the CFG
In reply to Yonghong SongBoth the CFG walk and liveness are involved. A bpf_throw() or a bpf2bpf call inside the [begin_off, end_off) range of a cleanup record can reach that record's landing pad, so both grow that edge; and a bpf_unwind_resume() reaches nothing after it, so it has no successors.
Liveness needs one more thing. bpf_stack_slot_alive() decides whether an outer frame's stack slot is still read after the call the frame is suspended at by looking at the instruction after the call. A slot whose only remaining reader is the landing pad -- which is every slot a compiler-generated pad reloads, since only four registers survive a call -- is dead by that measure, so clean_verifier_state() poisons it while the callee runs and the pad is then rejected for reading it. Ask about the pad as well when the call site names one.
--- kernel/bpf/cfg.c | 47 ++++++++++++++++++++++++++++++++++++++++--- kernel/bpf/liveness.c | 20 ++++++++++++++++++ 2 files changed, 64 insertions(+), 3 deletions(-)
kernel/bpf/cfg.c +44 -3
@@ -6,6 +6,7 @@#include <linux/sort.h> #include "diagnostics.h" +#include "exception.h" #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@@ -158,17 +159,57 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)return DONE_EXPLORING ; } +static int visit_cleanup_pad_edge (int t,struct bpf_verifier_env *env) +{ +int *insn_stack = env->cfg.insn_stack; +int *insn_state = env->cfg.insn_state; +int w; + +if (!env->cleanup_info_cnt) +return DONE_EXPLORING ; + w =bpf_cleanup_pad_of_call (env, t); +if (w <0 ) +return DONE_EXPLORING ; + +mark_prune_point (env, t); +mark_jmp_point (env, w); +mark_jump_target (env, w); + +if (insn_state[w]) +return DONE_EXPLORING ; +if (env->cfg.cur_stack >= env->prog->len) +return -E2BIG ; + insn_stack[env->cfg.cur_stack++] = w; + insn_state[w] |=DISCOVERED ; +return KEEP_EXPLORING ; +} + +static int merge_visit_ret (int a,int b) +{ +if (a <0 ) +return a; +if (b <0 ) +return b; +if (a ==KEEP_EXPLORING || b ==KEEP_EXPLORING ) +return KEEP_EXPLORING ; +return DONE_EXPLORING ; +} +static int visit_func_call_insn (int t,struct bpf_insn *insns,struct bpf_verifier_env *env,bool visit_callee) { -int ret, insn_sz; +int ret, insn_sz, pad_ret;int w; + pad_ret =visit_cleanup_pad_edge (t, env); +if (pad_ret <0 ) +return pad_ret; + insn_sz =bpf_is_ldimm64 (&insns[t]) ?2 :1 ; ret =push_insn (t, t + insn_sz,FALLTHROUGH , env);if (ret) -return ret; +return merge_visit_ret (pad_ret, ret);mark_prune_point (env, t + insn_sz);/* when we exit from subprog, we need to record non-linear history */
@@ -180,7 +221,7 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,merge_callee_effects (env, t, w); ret =push_insn (t, w,BRANCH , env); } -return ret; +return merge_visit_ret (pad_ret, ret); }struct bpf_iarray *bpf_iarray_realloc (struct bpf_iarray *old,size_t n_elem)
kernel/bpf/liveness.c +20 -0
@@ -8,6 +8,8 @@#include <linux/slab.h> #include <linux/sort.h> +#include "exception.h" + #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)struct per_frame_masks {
@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) succ = env->succ; succ->cnt =0 ; +if (unlikely (bpf_is_unwind_resume_kfunc (insn))) +return succ; + opcode_info = &opcode_info_tbl[BPF_CLASS (insn->code) |BPF_OP (insn->code)]; insn_sz =bpf_is_ldimm64 (insn) ?2 :1 ;if (opcode_info->can_fallthrough)
@@ -264,6 +269,13 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)if (opcode_info->can_jump) succ->items[succ->cnt++] = idx +bpf_jmp_offset (insn) +1 ; +if (unlikely (env->cleanup_info_cnt)) { +int pad =bpf_cleanup_pad_of_call (env, idx); + +if (pad >=0 ) + succ->items[succ->cnt++] = pad; + } +return succ; }
@@ -397,6 +409,14 @@ bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp alive =bpf_calls_callback (env, callsite) ?is_live_before (instance, callsite, rel, half_spi) :is_live_before (instance, callsite +1 , rel, half_spi); + +/* Control may also go to the landing pad. */ +if (!alive &&unlikely (env->cleanup_info_cnt)) { +int pad =bpf_cleanup_pad_of_call (env, callsite); + +if (pad >=0 ) + alive =is_live_before (instance, pad, rel, half_spi); + }if (alive)return true; }
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 05/20] bpf: Make exception landing pads reachable in the CFG
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] bpf: missing prune point at exception landing pad target --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: missing prune point at exception landing pad target
--
commit 4fbd0e36862ba7e3ee8480e535068a6821d5cad0
Author: Yonghong Song <[email protected]>
bpf: Make exception landing pads reachable in the CFG
This commit updates the verifier's control flow graph and liveness analysis to properly handle exception landing pads. It ensures that exception throwing calls can reach cleanup records, and prevents outer frame stack slots from being read incorrectly when control is transferred to landing pads.
Quoted 4 lines
kernel/bpf/cfg.c +0
-0
[ ... ]
Quoted 9 lines
kernel/bpf/cfg.c +5
-0@@ -158,17 +159,57 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)return DONE_EXPLORING ; } +static int visit_cleanup_pad_edge (int t,struct bpf_verifier_env *env) +{ +int *insn_stack = env->cfg.insn_stack; +int *insn_state = env->cfg.insn_state; +int w;
kernel/bpf/cfg.c static int push_insn(int t, int w, int e, struct bpf_verifier_env *env) · line 176
Quoted 10 lines
kernel/bpf/cfg.c +10
-0+ +if (!env->cleanup_info_cnt) +return DONE_EXPLORING ; + w =bpf_cleanup_pad_of_call (env, t); +if (w <0 ) +return DONE_EXPLORING ; + +mark_prune_point (env, t); +mark_jmp_point (env, w); +mark_jump_target (env, w);
[Severity: High]
Should mark_prune_point() be applied to the landing pad target w instead of the call site t?
Because w is a branch target where control flow merges, it needs to be marked as a prune point so the verifier knows to check for state convergence. By marking t instead of w, the verifier will evaluate the landing pad and all subsequent cleanup instructions independently for every exception-throwing call that branches to w.
Can this lead to combinatorial path explosion and result in the verifier rejecting the program with -E2BIG?
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Yonghong Song
reviewerRe: [PATCH bpf-next v2 05/20] bpf: Make exception landing pads reachable in the CFG
Quoted 43 lines · On 9/17/26 9:59 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: missing prune point at exception landing pad target
--commit
4fbd0e36862ba7e3ee8480e535068a6821d5cad0
Author: Yonghong Song <[email protected]>bpf: Make exception landing pads reachable in the CFG
This commit updates the verifier's control flow graph and liveness analysis to properly handle exception landing pads. It ensures that exception throwing calls can reach cleanup records, and prevents outer frame stack slots from being read incorrectly when control is transferred to landing pads.
Quoted 4 lines
kernel/bpf/cfg.c +0
-0[ ... ]
Quoted 9 lines
kernel/bpf/cfg.c +5
-0@@ -158,17 +159,57 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)return DONE_EXPLORING ; } +static int visit_cleanup_pad_edge (int t,struct bpf_verifier_env *env) +{ +int *insn_stack = env->cfg.insn_stack; +int *insn_state = env->cfg.insn_state; +int w;kernel/bpf/cfg.c static int push_insn(int t, int w, int e, struct bpf_verifier_env *env) · line 176
Quoted 10 lines
kernel/bpf/cfg.c +10
-0+ +if (!env->cleanup_info_cnt) +return DONE_EXPLORING ; + w =bpf_cleanup_pad_of_call (env, t); +if (w <0 ) +return DONE_EXPLORING ; + +mark_prune_point (env, t); +mark_jmp_point (env, w); +mark_jump_target (env, w);[Severity: High]
Shouldmark_prune_point()be applied to the landing pad target w instead of the call site t?
Quoted 8 lines
Because w is a branch target where control flow merges, it needs to be marked as a prune point so the verifier knows to check for state convergence. By marking t instead of w, the verifier will evaluate the landing pad and all subsequent cleanup instructions independently for every exception-throwing call that branches to w.
Can this lead to combinatorial path explosion and result in the verifier rejecting the program with
-E2BIG?
We can add mark_prune_point(env, w) as well, esp. there are cases there are multiple cleanup regions but many (or all) of them may pointing into the same landing pad.
Quoted 3 lines
[ ... ]
[email protected]
AI reviewRe: [PATCH bpf-next v2 05/20] bpf: Make exception landing pads reachable in the CFG
In reply to Yonghong SongShould this early return be gated on env->cleanup_info_cnt?
Quoted 14 lines
kernel/bpf/liveness.c +2
-0@@ -8,6 +8,8 @@#include <linux/slab.h> #include <linux/sort.h> +#include "exception.h" + #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)struct per_frame_masks {
kernel/bpf/liveness.c bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) · line 263
Quoted 6 lines
kernel/bpf/liveness.c +3
-0@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) succ = env->succ; succ->cnt =0 ; +if (unlikely (bpf_is_unwind_resume_kfunc (insn))) +return succ; +
Should this early return be gated on env->cleanup_info_cnt?
bpf_unwind_resume() is an unconditionally registered generic kfunc with no KF_ flag restricting it to programs that carry a cleanup table. Its body is empty, so at run time the call simply returns and execution continues at the next instruction.
visit_insn() in cfg.c still gives the call a fallthrough edge, and at this commit nothing in do_check() rejects a bpf_unwind_resume() call that is not the tail of a cleanup landing pad. So any CAP_BPF program with no cleanup table can call bpf_unwind_resume() and keep executing after it.
For such a program, bpf_insn_successors() reports zero successors, so bpf_compute_live_registers() treats everything after the call as dead. The register-cleaning side fails closed (a poisoned slot is rejected), but the 32-bit zero-extension side fails open:
insn_aux[i].zext_dst for an insn preceding the call is only set when the hi half of the defined register is live-after, so it stays false, and bpf_opt_subreg_zext_lo32_rnd_hi32() (kernel/bpf/fixups.c:706) then does 'if (!aux[adj_idx].zext_dst) { if (!rnd_hi32) continue; ... }' and emits no BPF_ZEXT_REG.
On architectures where bpf_jit_needs_zext() is true and kfunc calls are supported (s390, riscv64, powerpc64, parisc, x86-32), the upper 32 bits of a register the verifier proved to be 32-bit bounded are left with stale JIT-defined contents, which defeats the range checks the verifier derived for it.
Compare with the pad lookup ten lines below:
kernel/bpf/liveness.c bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) · line 277
Quoted 13 lines
kernel/bpf/liveness.c +6
-0opcode_info = &opcode_info_tbl[BPF_CLASS (insn->code) |BPF_OP (insn->code)]; insn_sz =bpf_is_ldimm64 (insn) ?2 :1 ;if (opcode_info->can_fallthrough)@@ -264,6 +269,13 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)if (opcode_info->can_jump) succ->items[succ->cnt++] = idx +bpf_jmp_offset (insn) +1 ; +if (unlikely (env->cleanup_info_cnt)) { +int pad =bpf_cleanup_pad_of_call (env, idx); + +if (pad >=0 ) + succ->items[succ->cnt++] = pad; + }
which gates the pad lookup on cleanup_info_cnt, and with visit_cleanup_pad_edge() in kernel/bpf/cfg.c which returns DONE_EXPLORING when !env->cleanup_info_cnt.
Would gating the early return on env->cleanup_info_cnt fix this? That would be sound because a program that carries a cleanup table is additionally constrained (bpf_jit_supports_cleanup_pads() plus, from commit fa6890df61539 onward, the requirement that a resume sit at the end of a landing pad), whereas a program with cleanup_info_cnt == 0 must keep the ordinary fallthrough successor of the call.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerRe: [PATCH bpf-next v2 05/20] bpf: Make exception landing pads reachable in the CFG
Quoted 22 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 14 lines
kernel/bpf/liveness.c +2
-0@@ -8,6 +8,8 @@#include <linux/slab.h> #include <linux/sort.h> +#include "exception.h" + #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)struct per_frame_masks {kernel/bpf/liveness.c bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) · line 263
Quoted 6 lines
kernel/bpf/liveness.c +3
-0@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) succ = env->succ; succ->cnt =0 ; +if (unlikely (bpf_is_unwind_resume_kfunc (insn))) +return succ; +Should this early return be gated on
env->cleanup_info_cnt?
Quoted 4 lines
bpf_unwind_resume()is an unconditionally registered generic kfunc with noKF_flag restricting it to programs that carry a cleanup table. Its body is empty, so at run time the call simply returns and execution continues at the next instruction.
This should be okay without env->cleanup_info_cnt guard. Eventually if bpf_unwind_resume() is not in cleanup region, a verifier error message will be in the log.
Quoted 42 lines
visit_insn()incfg.cstill gives the call a fallthrough edge, and at this commit nothing indo_check()rejects abpf_unwind_resume()call that is not the tail of a cleanup landing pad. So anyCAP_BPFprogram with no cleanup table can callbpf_unwind_resume()and keep executing after it.For such a program,
bpf_insn_successors()reports zero successors, sobpf_compute_live_registers()treats everything after the call as dead. The register-cleaning side fails closed (a poisoned slot is rejected), but the 32-bit zero-extension side fails open:insn_aux[i].zext_dst for an insn preceding the call is only set when the hi half of the defined register is live-after, so it stays false, and
bpf_opt_subreg_zext_lo32_rnd_hi32()(kernel/bpf/fixups.c:706) then does 'if (!aux[adj_idx].zext_dst) { if (!rnd_hi32) continue; ... }' and emits noBPF_ZEXT_REG.On architectures where
bpf_jit_needs_zext()is true and kfunc calls are supported (s390, riscv64, powerpc64, parisc, x86-32), the upper 32 bits of a register the verifier proved to be 32-bit bounded are left with stale JIT-defined contents, which defeats the range checks the verifier derived for it.Compare with the pad lookup ten lines below:
kernel/bpf/liveness.c bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) · line 277
Quoted 13 lines
kernel/bpf/liveness.c +6
-0opcode_info = &opcode_info_tbl[BPF_CLASS (insn->code) |BPF_OP (insn->code)]; insn_sz =bpf_is_ldimm64 (insn) ?2 :1 ;if (opcode_info->can_fallthrough)@@ -264,6 +269,13 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)if (opcode_info->can_jump) succ->items[succ->cnt++] = idx +bpf_jmp_offset (insn) +1 ; +if (unlikely (env->cleanup_info_cnt)) { +int pad =bpf_cleanup_pad_of_call (env, idx); + +if (pad >=0 ) + succ->items[succ->cnt++] = pad; + }which gates the pad lookup on
cleanup_info_cnt, and withvisit_cleanup_pad_edge()inkernel/bpf/cfg.cwhich returnsDONE_EXPLORINGwhen !env->cleanup_info_cnt.
Quoted 6 lines
Would gating the early return on
env->cleanup_info_cntfix this? That would be sound because a program that carries a cleanup table is additionally constrained (bpf_jit_supports_cleanup_pads()plus, from commitfa6890df61539onward, the requirement that a resume sit at the end of a landing pad), whereas a program withcleanup_info_cnt== 0 must keep the ordinary fallthrough successor of the call.
As I mentioned, previous gating with "env->cleanup_info_cnt" is not necessary.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerPATCH v2 6/20 bpf: Explore the landing pads no call site reaches
In reply to Yonghong SongA cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving bpf_check_cfg() to refuse the program over code its own frontend had no way not to emit:
0: call bpf_preempt_disable
1: call bpf_preempt_enable record = { begin = 1, end = 2, pad = 4 }
2: r0 = 0
3: exit
4: r1 = pads_ran ll landing pad
6: r2 = *(u64 *)(r1 + 0)
7: r2 |= RAN_NOUNWIND_REC
8: *(u64 *)(r1 + 0) = r2
9: call bpf_unwind_resume
10: exit
The range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it. cleanup_mark_call_sites() marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.
Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other: do_check() never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.
--- kernel/bpf/cfg.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
kernel/bpf/cfg.c +17 -0
@@ -633,6 +633,7 @@ int bpf_check_cfg(struct bpf_verifier_env *env)int insn_cnt = env->prog->len;int *insn_stack, *insn_state;int ex_insn_beg, i, ret =0 ; +u32 pad_idx =0 ; insn_state = env->cfg.insn_state =kvzalloc_objs (int, insn_cnt,GFP_KERNEL_ACCOUNT );
@@ -688,6 +689,22 @@ int bpf_check_cfg(struct bpf_verifier_env *env) goto walk_cfg; } +/* +* A landing pad no call site was marked with -- a record whose range +* holds no call an exception can unwind out of -- is reached by +* nothing. Walk it from here, and let the dead code sweep remove it. +*/ +while (pad_idx < env->cleanup_info_cnt) { +u32 pad = env->cleanup_info[pad_idx++].landing_pad_off; + +if (insn_state[pad] !=EXPLORED ) { + insn_state[pad] =DISCOVERED ; + insn_stack[0 ] = pad; + env->cfg.cur_stack =1 ; + goto walk_cfg; + } + } +for (i =0 ; i < insn_cnt; i++) {struct bpf_insn *insn = &env->prog->insnsi[i];
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 06/20] bpf: Explore the landing pads no call site reaches
This looks like a bug fix for the earlier commit that made exception landing pads reachable in the CFG. Should this include:
Quoted 28 lines
commit
c93117f24a1a9cc39bfeaef79c42dbff02d8c455
Author: Yonghong Song <[email protected]>bpf: Explore the landing pads no call site reaches
A cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving
bpf_check_cfg()to refuse the program over code its own frontend had no way not to emit:0: call
bpf_preempt_disable
1: callbpf_preempt_enablerecord = { begin = 1, end = 2, pad = 4 }
2: r0 = 0
3: exit
4: r1 =pads_ranll landing pad
6: r2 = *(u64 *)(r1 + 0)
7: r2 |=RAN_NOUNWIND_REC
8: *(u64 *)(r1 + 0) = r2
9: callbpf_unwind_resume
10: exitThe range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it.
cleanup_mark_call_sites()marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.
Quoted 7 lines
Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other:
do_check()never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.Signed-off-by Yonghong Song <[email protected]>
This looks like a bug fix for the earlier commit that made exception landing pads reachable in the CFG. Should this include:
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerRe: [PATCH bpf-next v2 06/20] bpf: Explore the landing pads no call site reaches
Quoted 34 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 34 lines
commit
c93117f24a1a9cc39bfeaef79c42dbff02d8c455
Author: Yonghong Song <[email protected]>bpf: Explore the landing pads no call site reaches
A cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving
bpf_check_cfg()to refuse the program over code its own frontend had no way not to emit:0: call
bpf_preempt_disable
1: callbpf_preempt_enablerecord = { begin = 1, end = 2, pad = 4 }
2: r0 = 0
3: exit
4: r1 =pads_ranll landing pad
6: r2 = *(u64 *)(r1 + 0)
7: r2 |=RAN_NOUNWIND_REC
8: *(u64 *)(r1 + 0) = r2
9: callbpf_unwind_resume
10: exitThe range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it.
cleanup_mark_call_sites()marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other:
do_check()never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.
Quoted 5 lines
Quoted 1 line
Signed-off-by Yonghong Song <[email protected]>This looks like a bug fix for the earlier commit that made exception landing pads reachable in the CFG. Should this include:
Fixes a67011bbbca1ba95 ("bpf: Make exception landing pads reachable in the CFG")
No need for this Fixes tag.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerPATCH v2 7/20 bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch
In reply to Yonghong SongThe table is on the instructions and the landing pads are in the control flow graph. What is left before bpf_throw() can be taught to dispatch them is to work out what that walk will need, and refuse the shapes it could not handle.
bpf_check_cleanup_exceptions() runs after bpf_check_cfg(). Everything it needs is control flow, so it reads what that walk has already worked out rather than working it out again:
subprog_info.might_throw which subprograms an exception may leave,
closed over the call graph by merge_callee_effects() as the walk pops each callee
cleanup_reachability() what each instruction can reach -- a resume, a
plain exit, a throw, an indirect jump
cleanup_mark_pad_bodies() which instructions only ever run with an
exception already in flight
What it refuses:
- a pad that reaches both a resume and a plain exit, or neither: nothing says whether it is a cleanup pad or a catch pad
- a catch pad, which ends in a plain exit: the walker calls a pad as a subroutine and cannot hand a frame back its own execution
- a throw in a pad, or a call from a pad to a subprogram that can throw:
a second unwind over frames the first is still discarding
- a bpf_unwind_resume() outside a pad body
- a subprogram that may unwind used as a helper callback: the helper's own kernel frame would end the walk before it found a boundary
- a tail call, an indirect jump, or an outgoing on-stack call argument in a pad body, all of which touch a stack the pad does not own
--- include/linux/bpf_verifier.h | 1 + kernel/bpf/exception.c | 397 +++++++++++++++++++++++++++++++++++ kernel/bpf/exception.h | 2 + kernel/bpf/fixups.c | 1 + kernel/bpf/verifier.c | 8 + 5 files changed, 409 insertions(+)
include/linux/bpf_verifier.h +1 -0
@@ -682,6 +682,7 @@ struct bpf_insn_aux_data {bool non_sleepable;/* helper/kfunc may be called from non-sleepable context */ bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ bool cleanup_throw_site;/* call to bpf_throw() */ +bool in_cleanup_pad;/* only runs with an exception in flight */ /* * 1 + the instruction index of the exception cleanup landing pad this * call site unwinds to, or 0 for none.
kernel/bpf/exception.c +397 -0
421 lines not on this page · raw message
kernel/bpf/exception.h +2 -0
@@ -8,6 +8,8 @@struct bpf_verifier_env ;int bpf_prepare_cleanup_exceptions (struct bpf_verifier_env *env); +int bpf_check_cleanup_exceptions (struct bpf_verifier_env *env); +int bpf_cleanup_check_callback (struct bpf_verifier_env *env,int subprog);int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx);#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +1 -0
@@ -252,6 +252,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,/* Expand insni[off]'s seen count to the patched range. */ data[i].seen = old_seen; data[i].zext_dst =bpf_insn_def32 (new_prog, insn + i) >=0 ; + data[i].in_cleanup_pad = data[off + cnt -1 ].in_cleanup_pad;if (!memcmp (insn + i, original_insn,sizeof (struct bpf_insn ))) { data[i].non_stack_access = data[off + cnt -1 ].non_stack_access;
kernel/bpf/verifier.c +8 -0
@@ -10485,6 +10485,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins* callbacks */ env->subprog_info[subprog].is_cb = true; + err =bpf_cleanup_check_callback (env, subprog); +if (err) +return err; +if (bpf_pseudo_kfunc_call (insn) && !is_callback_calling_kfunc (insn->imm)) { verifier_bug(env, "kfunc %s#%d not marked as callback-calling",
@@ -21664,6 +21668,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,if (ret <0 ) goto skip_full_check; + ret =bpf_check_cleanup_exceptions (env); +if (ret <0 ) + goto skip_full_check; + ret =bpf_compute_postorder (env);if (ret <0 ) goto skip_full_check;
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v2 8/20 bpf: Walk the exception unwind in the verifier
In reply to Yonghong Songbpf_throw() is about to stop discarding the frames it unwinds and start running their landing pads instead. Have the verifier walk the same thing, step for step:
at a throw, with the frame chain in env->cur_state->frame[]
if a record covers the call control is at
run that record's landing pad in this frame on reaching its resume, pop the frame and carry on
else
pop the frame and carry on
at the boundary, deliver
Doing it this way is what keeps the resource rules unchanged. Whatever a pad releases is released in the verifier state too, so by the time the walk reaches the boundary the state says exactly what the program will really hold there -- and check_resource_leak(), which used to fire the moment a throw was seen, simply moves to the end of the walk. A frame that no record covers contributes nothing, so anything it held is still held when the walk ends, and that is what gets reported.
--- include/linux/bpf_cleanup_abi.h | 16 +++++ include/linux/bpf_verifier.h | 1 + kernel/bpf/states.c | 3 + kernel/bpf/verifier.c | 102 +++++++++++++++++++++++++------- 4 files changed, 99 insertions(+), 23 deletions(-) create mode 100644 include/linux/bpf_cleanup_abi.h
include/linux/bpf_cleanup_abi.h new file +16 -0
@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#ifndef _LINUX_BPF_CLEANUP_ABI_H +#define _LINUX_BPF_CLEANUP_ABI_H + +/* +* Value arch_bpf_run_cleanup_pad() leaves in r0 on the way into a landing pad. +* It has to be a constant the verifier knows: LLVM names r0 as both the +* exception pointer and the exception selector register, so every pad reads it +* before anything else and is free to store what it read. Kept on its own +* because the verifier and the arch dispatchers, which are assembly, have to +* agree on it. +*/ +#define BPF_PAD_ENTRY_R0 1 + +#endif /* _LINUX_BPF_CLEANUP_ABI_H */
include/linux/bpf_verifier.h +1 -0
@@ -509,6 +509,7 @@ struct bpf_verifier_state {bool speculative;bool in_sleepable; +bool unwinding;/* first and last insn idx of this verifier state */ u32 first_insn_idx;
kernel/bpf/states.c +3 -0
@@ -996,6 +996,9 @@ static bool states_equal(struct bpf_verifier_env *env,if (old->in_sleepable != cur->in_sleepable)return false; +if (old->unwinding != cur->unwinding) +return false; +if (!refsafe (old, cur, &env->idmap_scratch))return false;
kernel/bpf/verifier.c +79 -23
160 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v2 9/20 bpf: Refuse a private stack for a program with an exception cleanup table
In reply to Yonghong SongA landing pad is about to start running outside its own frame, on the bpf_throw() walker's stack, with that frame's registers put back from a spill area whose contents arch_bpf_run_cleanup_pad() knows how to read. The x86-64 JIT addresses a private-stack program's frame through a scratch register that it recomputes after each call rather than through rbp, and no spill area holds that register, so a pad there would address its frame through whatever the kernel left behind.
Refuse the combination in check_max_stack_depth(), where the choice is made. Everywhere rather than arch-conditionally: arm64 keeps its private stack pointer in x27, which is in the prologue spill and so survives, but a rule that holds on one arch and not the other is not worth the second code path when nothing is lost but an optimization.
--- kernel/bpf/verifier.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/bpf/verifier.c +8 -0
@@ -5591,6 +5591,14 @@ static int check_max_stack_depth(struct bpf_verifier_env *env) } } +/* +* A pad rebuilds its frame from a spill area, and on x86-64 a private +* stack's frame pointer is in no spill area. Refused on every arch +* rather than just that one. +*/ +if (env->cleanup_info_cnt) + priv_stack_mode =NO_PRIV_STACK ; +if (priv_stack_mode ==PRIV_STACK_UNKNOWN ) priv_stack_mode =bpf_enable_priv_stack (env->prog);
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 09/20] bpf: Refuse a private stack for a program with an exception cleanup table
The description says the x86-64 JIT 'recomputes after each call'. Does the JIT actually recompute it?
Quoted 7 lines
A landing pad is about to start running outside its own frame, on the
bpf_throw()walker's stack, with that frame's registers put back from a spill area whose contentsarch_bpf_run_cleanup_pad()knows how to read. The x86-64 JIT addresses a private-stack program's frame through a scratch register that it recomputes after each call rather than through rbp, and no spill area holds that register, so a pad there would address its frame through whatever the kernel left behind.
The description says the x86-64 JIT 'recomputes after each call'. Does the JIT actually recompute it?
Looking at the x86 JIT, emit_priv_frame_ptr() (arch/x86/net/bpf_jit_comp.c) is emitted once in the prologue, and the value in X86_REG_R9 is carried across calls by push_r9()/pop_r9() around each call site; BPF_REG_FP accesses are rewritten to X86_REG_R9.
The load-bearing half of the argument is still correct - arch_bpf_run_cleanup_pad() (added later in the series, arch/x86/net/bpf_cleanup_pad.S) restores only r15/r14/r13/rbx/r12 and rbp from the spill area and never restores r9, so a pad in a private-stack program would address its frame through a stale register.
Only the described mechanism (recompute vs save/restore around calls) is wrong, which makes the justification harder to check against the JIT.
Quoted 4 lines
Everywhere rather than arch-conditionally: arm64 keeps its private stack pointer in x27, which is in the prologue spill and so survives, but a rule that holds on one arch and not the other is not worth the second code path when nothing is lost but an optimization.
Does 'nothing is lost but an optimization' understate the effect?
The private-stack mode also changes the verifier's stack-depth accounting, not just JIT codegen. In check_max_stack_depth_subprog() a PRIV_STACK_ADAPTIVE subprog is checked against MAX_BPF_STACK on its own ('if (subprog_depth > MAX_BPF_STACK)'), while a NO_PRIV_STACK subprog accumulates into 'depth' and the whole bpf2bpf chain must fit in MAX_BPF_STACK ('combined stack size of %d calls is %d. Too large').
So for the prog types bpf_enable_priv_stack() would have returned PRIV_STACK_ADAPTIVE for (kprobe/tracepoint/perf_event/raw_tp, and tracing/lsm/struct_ops with a recursion or an explicit request), a program carrying a cleanup table whose nested subprog depths sum above 512 bytes now fails to load rather than losing performance.
Worth stating in the changelog, since it is a load-time visible restriction on the new feature rather than an optimization.
Related: kernel/sched/ext/ext.c sets prog->aux->priv_stack_requested for sched_ext dispatch/sub_caps_updated, and verifier.c rejects such a request when the JIT cannot honour it; here the request is silently ignored instead.
This looks like a bug fix for private stack support. Should this include:
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerRe: [PATCH bpf-next v2 09/20] bpf: Refuse a private stack for a program with an exception cleanup table
Quoted 37 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 7 lines
A landing pad is about to start running outside its own frame, on the
bpf_throw()walker's stack, with that frame's registers put back from a spill area whose contentsarch_bpf_run_cleanup_pad()knows how to read. The x86-64 JIT addresses a private-stack program's frame through a scratch register that it recomputes after each call rather than through rbp, and no spill area holds that register, so a pad there would address its frame through whatever the kernel left behind.The description says the x86-64 JIT 'recomputes after each call'. Does the JIT actually recompute it?
Looking at the x86 JIT,
emit_priv_frame_ptr()(arch/x86/net/bpf_jit_comp.c) is emitted once in the prologue, and the value inX86_REG_R9is carried across calls by push_r9()/pop_r9() around each call site;BPF_REG_FPaccesses are rewritten toX86_REG_R9.The load-bearing half of the argument is still correct -
arch_bpf_run_cleanup_pad()(added later in the series,arch/x86/net/bpf_cleanup_pad.S) restores only r15/r14/r13/rbx/r12 and rbp from the spill area and never restores r9, so a pad in a private-stack program would address its frame through a stale register.Only the described mechanism (recompute vs save/restore around calls) is wrong, which makes the justification harder to check against the JIT.
Quoted 4 lines
Everywhere rather than arch-conditionally: arm64 keeps its private stack pointer in x27, which is in the prologue spill and so survives, but a rule that holds on one arch and not the other is not worth the second code path when nothing is lost but an optimization.
Does 'nothing is lost but an optimization' understate the effect?
The private-stack mode also changes the verifier's stack-depth accounting, not just JIT codegen. In
check_max_stack_depth_subprog()aPRIV_STACK_ADAPTIVEsubprog is checked againstMAX_BPF_STACKon its own ('if (subprog_depth>MAX_BPF_STACK)'), while aNO_PRIV_STACKsubprog accumulates into 'depth' and the whole bpf2bpf chain must fit inMAX_BPF_STACK('combined stack size of %d calls is %d. Too large').
Quoted 8 lines
So for the prog types
bpf_enable_priv_stack()would have returnedPRIV_STACK_ADAPTIVEfor (kprobe/tracepoint/perf_event/raw_tp, and tracing/lsm/struct_ops with a recursion or an explicit request), a program carrying a cleanup table whose nested subprog depths sum above 512 bytes now fails to load rather than losing performance.Worth stating in the changelog, since it is a load-time visible restriction on the new feature rather than an optimization.
Okay, comments and commit message may have some stale things or something like (when nothing is lost but an optimization) is not correct. Will fix.
Quoted 8 lines
Related:
kernel/sched/ext/ext.csetsprog->aux->priv_stack_requestedforsched_extdispatch/sub_caps_updated, andverifier.crejects such a request when the JIT cannot honour it; here the request is silently ignored instead.This looks like a bug fix for private stack support. Should this include:
Fixes a76ab5731e32d ("bpf: Find eligible subprogs for private stack support")
Fixes tag is unnecessary.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerPATCH v2 10/20 bpf: Dispatch exception cleanup pads from bpf_throw()
In reply to Yonghong SongStop discarding the frames an exception unwinds through. bpf_throw() already walks the BPF call stack with arch_bpf_stack_walk() to find the exception boundary; have it look each frame's return address up in that (sub)program's cleanup table on the way and run the landing pad a matching record names.
A pad is run as a subroutine of the walker, not jumped to. It executes with the unwinding frame's frame pointer and BPF callee-saved registers, so everything it reads is that frame's, but on the current stack far below it, so nothing it calls can disturb the frame it is cleaning up after. It ends in what the JIT emits for its bpf_unwind_resume(), which hands control back to the walker rather than to the frame's caller. That is exactly the procedure the LLVM commit emitting the section describes.
Restoring the frame's r6-r9 is what makes this work, and it is only possible because the callee about to be discarded spilled them in its own prologue. bpf_cleanup_force_spill() tells a JIT to make that spill unconditional and of a known shape for every subprogram of a program carrying a cleanup table, and aux->exc->spill_off records where it starts, so the walker needs no per-frame metadata. The frame that called bpf_throw() has no callee to have spilled anything and never runs its own epilogue, so the JIT spills that frame's registers at the throw site instead, in the area aux->exc->throw_spill_off names.
The main program needs the table handed to it rather than built for it. jit_subprogs() compiles it as func[0], but the ksym covering that image is the one bpf_prog_load() registers for the outer bpf_prog, and that is what the walker finds -- so without the handover a landing pad in the main program's own frame is never dispatched, silently, since the exception still reaches the boundary and the cookie still comes back.
All of it hangs off bpf_prog_aux by a single pointer. bpf_prog_aux is on every BPF program and this is a niche feature, so struct bpf_exception_info exists only for a program that carries a table, and aux->exc being set is what says the program carries one at all.
--- include/linux/bpf.h | 75 +++++++++++++++++++ include/linux/filter.h | 1 + kernel/bpf/core.c | 30 +++++++- kernel/bpf/exception.c | 166 +++++++++++++++++++++++++++++++++++++++++ kernel/bpf/exception.h | 7 ++ kernel/bpf/fixups.c | 125 +++++++++++++++++++++++++++++++ kernel/bpf/helpers.c | 34 +++++++++ 7 files changed, 435 insertions(+), 3 deletions(-)
include/linux/bpf.h +75 -0
@@ -1770,6 +1770,80 @@ enum bpf_sig_keyring { BPF_SIG_KEYRING_BPF, }; +/* +* One cleanup region of a JITed (sub)program: @pad is the landing pad to run +* for a return address in (begin, end], the native code of its call sites. +*/ +struct bpf_cleanup_range { +u64 begin; +u64 end; +u64 pad; +}; + +struct bpf_exception_info { +struct bpf_cleanup_info *info; +struct bpf_cleanup_range *ranges; +/* Landing pad instruction indices, sorted and deduplicated. */ +u32 *pad_at; +/* bpf_throw() call instruction indices, sorted. */ +u32 *throw_at; +/* One bit per instruction that only runs while unwinding. */ +unsigned long *pad_body; +u32 nr_info; +u32 nr_ranges; +u32 nr_pad_at; +u32 nr_throw_at; +u32 nr_pad_body; +/* Offset from a frame's FP to the caller's spilled r6-r9. */ +s32 spill_off; +/* Likewise, to the registers a frame spills before calling bpf_throw(). */ +s32 throw_spill_off; +}; + +#ifdef CONFIG_BPF_SYSCALL +bool bpf_cleanup_force_spill (const struct bpf_prog *prog); +bool bpf_cleanup_insn_is_pad (const struct bpf_prog *prog,u32 idx); +bool bpf_cleanup_insn_in_pad (const struct bpf_prog *prog,u32 idx); +bool bpf_cleanup_insn_is_throw (const struct bpf_prog *prog,u32 idx); +int bpf_cleanup_attach_main_prog (struct bpf_verifier_env *env,struct bpf_prog *prog); +void bpf_cleanup_fill_native_ranges (struct bpf_prog *prog,u32 *addrs,void *image); +void bpf_cleanup_free_info (struct bpf_prog_aux *aux); +#else +static inline bool bpf_cleanup_force_spill (const struct bpf_prog *prog) +{ +return false; +} + +static inline bool bpf_cleanup_insn_is_pad (const struct bpf_prog *prog,u32 idx) +{ +return false; +} + +static inline bool bpf_cleanup_insn_in_pad (const struct bpf_prog *prog,u32 idx) +{ +return false; +} + +static inline bool bpf_cleanup_insn_is_throw (const struct bpf_prog *prog,u32 idx) +{ +return false; +} + +static inline int bpf_cleanup_attach_main_prog (struct bpf_verifier_env *env, +struct bpf_prog *prog) +{ +return 0 ; +} + +static inline void bpf_cleanup_fill_native_ranges (struct bpf_prog *prog,u32 *addrs,void *image) +{ +} + +static inline void bpf_cleanup_free_info (struct bpf_prog_aux *aux) +{ +} +#endif +struct bpf_prog_aux {atomic64_t refcnt;u32 used_map_cnt;
@@ -1850,6 +1924,7 @@ struct bpf_prog_aux {char name[BPF_OBJ_NAME_LEN ];u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 ,u64 );u16 stack_arg_sp_adjust; +struct bpf_exception_info *exc;#ifdef CONFIG_SECURITY void *security;#endif
include/linux/filter.h +1 -0
@@ -1243,6 +1243,7 @@ bool bpf_jit_supports_arena_args(void);bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );bool bpf_jit_supports_cleanup_pads (void ); +void arch_bpf_run_cleanup_pad (u64 pad,u64 frame_fp,u64 spill_base);bool bpf_jit_supports_ptr_xchg (void );bool bpf_jit_supports_arena (void );bool bpf_jit_supports_insn (struct bpf_insn *insn,bool in_arena);
kernel/bpf/core.c +27 -3
@@ -292,6 +292,7 @@ void __bpf_prog_free(struct bpf_prog *fp)mutex_destroy (&fp->aux->dst_mutex);mutex_destroy (&fp->aux->st_ops_assoc_mutex);kfree (fp->aux->poke_tab); +bpf_cleanup_free_info (fp->aux);kfree (fp->aux); }free_percpu (fp->stats);
@@ -2625,13 +2626,21 @@ static bool bpf_prog_select_interpreter(struct bpf_prog *fp)return select_interpreter; } -static struct bpf_prog *bpf_prog_jit_compile (struct bpf_verifier_env *env,struct bpf_prog *prog) +static struct bpf_prog *bpf_prog_jit_compile (struct bpf_verifier_env *env,struct bpf_prog *prog, +int *err) {#ifdef CONFIG_BPF_JIT struct bpf_prog *orig_prog; +int ret; -if (!bpf_prog_need_blind (prog)) +if (!bpf_prog_need_blind (prog)) { + ret =bpf_cleanup_attach_main_prog (env, prog); +if (ret) { + *err = ret; +return prog; + }return bpf_int_jit_compile (env, prog); + } orig_prog = prog; prog =bpf_jit_blind_constants (env, prog);
@@ -2642,6 +2651,13 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, strucif (IS_ERR (prog)) goto out_restore; + ret =bpf_cleanup_attach_main_prog (env, prog); +if (ret) { + *err = ret; +bpf_jit_prog_release_other (orig_prog, prog); + goto out_restore; + } + prog =bpf_int_jit_compile (env, prog);if (prog->jited) {bpf_jit_prog_release_other (prog, orig_prog);
@@ -2681,8 +2697,10 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, structif (*err)return fp; - fp =bpf_prog_jit_compile (env, fp); + fp =bpf_prog_jit_compile (env, fp, err);bpf_prog_jit_attempt_done (fp); +if (*err) +return fp;if (!fp->jited && jit_needed) { *err = -ENOTSUPP ;return fp;
@@ -3475,6 +3493,12 @@ bool __weak bpf_jit_supports_cleanup_pads(void)return false; } +/* Call @pad with the frame pointer @frame_fp and r6-r9 spilled at @spill_base. */ +void __weak arch_bpf_run_cleanup_pad (u64 pad,u64 frame_fp,u64 spill_base) +{ +WARN_ON_ONCE (1 ); +} +bool __weak bpf_jit_supports_timed_may_goto (void ) {return false;
kernel/bpf/exception.c +166 -0
174 lines not on this page · raw message
kernel/bpf/exception.h +7 -0
@@ -5,11 +5,18 @@#include <linux/types.h> +struct bpf_cleanup_info ; +struct bpf_cleanup_range ; +struct bpf_prog ; +struct bpf_prog_aux ;struct bpf_verifier_env ;int bpf_prepare_cleanup_exceptions (struct bpf_verifier_env *env);int bpf_check_cleanup_exceptions (struct bpf_verifier_env *env);int bpf_cleanup_check_callback (struct bpf_verifier_env *env,int subprog);int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx); +int bpf_cleanup_alloc_info (struct bpf_prog_aux *aux); +int bpf_cleanup_attach_info (struct bpf_prog_aux *aux,struct bpf_cleanup_info *recs,u32 cnt); +const struct bpf_cleanup_range *bpf_cleanup_pad_for_ip (const struct bpf_prog *prog,u64 ip);#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +125 -0
166 lines not on this page · raw message
kernel/bpf/helpers.c +34 -0
@@ -31,6 +31,7 @@#include <linux/buildid.h> #include "../../lib/kstrtox.h" +#include "exception.h" /* If kernel subsystem is allowing eBPF programs to call this function, * inside its own verifier_ops->get_func_proto() callback it should return
@@ -3360,8 +3361,36 @@ struct bpf_throw_ctx {u64 sp;u64 bp;int cnt; +const struct bpf_prog *callee; +u64 callee_fp; }; +static void bpf_run_cleanup_pad (struct bpf_throw_ctx *ctx,const struct bpf_prog *prog, +u64 ip,u64 fp) +{ +const struct bpf_exception_info *exc = prog->aux->exc; +const struct bpf_cleanup_range *rec; +u64 spill_base; + +if (!exc || !exc->nr_ranges) +return ; + rec =bpf_cleanup_pad_for_ip (prog, ip); +if (!rec) +return ; + +/* +* The callee is always another subprogram of this program -- the walk +* ends at any frame that is not one -- so its prologue spilled these +* registers and its exc is there to say where. +*/ +if (ctx->callee) + spill_base = ctx->callee_fp + ctx->callee->aux->exc->spill_off; +else + spill_base = fp + exc->throw_spill_off; + +arch_bpf_run_cleanup_pad (rec->pad, fp, spill_base); +} +static bool bpf_stack_walker (void *cookie,u64 ip,u64 sp,u64 bp) {struct bpf_throw_ctx *ctx = cookie;
@@ -3378,6 +3407,11 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)if (!prog)return !ctx->cnt; ctx->cnt++; + +bpf_run_cleanup_pad (ctx, prog, ip, bp); + ctx->callee = prog; + ctx->callee_fp = bp; +if (bpf_is_subprog (prog))return true; ctx->aux = prog->aux;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 10/20] bpf: Dispatch exception cleanup pads from bpf_throw()
This isn't a bug, but could insn_idx_in() just use bsearch() from <linux/bsearch.h>, reusing the cmp_u32() already defined above for the sort()? kernel/bpf/fixups.c in this series uses bsearch() for…
Quoted 4 lines
kernel/bpf/exception.c +0
-0
[ ... ]
Quoted 6 lines
kernel/bpf/exception.c +6
-0+static int cmp_u32 (const void *a,const void *b) +{ +u32 x = *(const u32 *)a, y = *(const u32 *)b; + +return x < y ?-1 : x > y; +}
[ ... ]
Quoted 7 lines
kernel/bpf/exception.c +7
-0+/* Is @idx in the sorted array @at of @n instruction indices? */ +static bool insn_idx_in (const u32 *at,u32 n,u32 idx) +{ +u32 l =0 , r = n; + +while (l < r) { +u32 m = l + (r - l) /2 ;
kernel/bpf/exception.c int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx) · line 632
Quoted 10 lines
kernel/bpf/exception.c +10
-0+ +if (idx < at[m]) + r = m; +else if (idx > at[m]) + l = m +1 ; +else +return true; + } +return false; +}
This isn't a bug, but could insn_idx_in() just use bsearch() from <linux/bsearch.h>, reusing the cmp_u32() already defined above for the sort()? kernel/bpf/fixups.c in this series uses bsearch() for similar lookups.
Quoted 4 lines
kernel/bpf/fixups.c +0
-0
[ ... ]
Quoted 55 lines
kernel/bpf/fixups.c +51
-0@@ -1113,6 +1120,116 @@ static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_s env->subprog_info[env->subprog_cnt].start = env->prog->len; } +static int cleanup_info_for_subprog (struct bpf_verifier_env *env,struct bpf_prog *sub, +u32 start,u32 end) +{ +struct bpf_cleanup_info *recs; +u32 i, cnt =0 ; +int err; + +if (!env->cleanup_info_cnt) +return 0 ; + + err =bpf_cleanup_alloc_info (sub->aux); +if (err) +return err; + + err =cleanup_throw_sites_for_subprog (env, sub, start, end); +if (err) +return err; + + err =cleanup_pad_body_for_subprog (env, sub, start, end); +if (err) +return err; + +for (i = start; i < end; i++) +if (env->insn_aux_data[i].cleanup_pad) + cnt++; +if (!cnt) +return 0 ; + + recs =kvmalloc_array (cnt,sizeof (*recs),GFP_KERNEL_ACCOUNT | __GFP_NOWARN); +if (!recs) +return -ENOMEM ; + +for (i = start, cnt =0 ; i < end; i++) { +u32 pad = env->insn_aux_data[i].cleanup_pad; + +if (!pad) +continue ; + pad--; +if (verifier_bug_if (pad < start || pad >= end, env, +"insn %u is covered by a landing pad at %u outside its subprog [%u, %u)" , + i, pad, start, end)) { +kvfree (recs); +return -EFAULT ; + } + recs[cnt].begin_off = i - start; + recs[cnt].end_off = i - start +1 ; + recs[cnt].landing_pad_off = pad - start; + cnt++; + } +return bpf_cleanup_attach_info (sub->aux, recs, cnt); +}
[ ... ]
Quoted 3 lines
static int
jit_subprogs(struct bpf_verifier_env *env){
structbpf_prog*prog =env->prog, **func, *tmp;
[ ... ]
kernel/bpf/fixups.c static int jit_subprogs(struct bpf_verifier_env *env) · line 1371
Quoted 6 lines
kernel/bpf/fixups.c +2
-0@@ -1250,6 +1367,10 @@ static int jit_subprogs(struct bpf_verifier_env *env) func[i]->aux->token = prog->aux->token;if (!i) func[i]->aux->exception_boundary = env->seen_exception; + err =cleanup_info_for_subprog (env, func[i], subprog_start, + env->subprog_info[i +1 ].start);
This isn't a bug, but would passing subprog_end (already computed a few lines above from env->subprog_info[i + 1].start) be clearer here? It keeps the cleanup table's end bound explicitly tied to the same value jit_subprogs() uses for func[i]->len.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerRe: [PATCH bpf-next v2 10/20] bpf: Dispatch exception cleanup pads from bpf_throw()
Quoted 27 lines · On 9/17/26 10:58 PM, [email protected] wrote:
Quoted 4 lines
kernel/bpf/exception.c +0
-0[ ... ]
Quoted 6 lines
kernel/bpf/exception.c +6
-0+static int cmp_u32 (const void *a,const void *b) +{ +u32 x = *(const u32 *)a, y = *(const u32 *)b; + +return x < y ?-1 : x > y; +}[ ... ]
Quoted 13 lines
kernel/bpf/exception.c +13
-0+/* Is @idx in the sorted array @at of @n instruction indices? */ +static bool insn_idx_in (const u32 *at,u32 n,u32 idx) +{ +u32 l =0 , r = n; + +while (l < r) { +u32 m = l + (r - l) /2 ; + +if (idx < at[m]) + r = m; +else if (idx > at[m]) + l = m +1 ; +else
Quoted 8 lines
kernel/bpf/exception.c int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx) · line 632
Quoted 4 lines
kernel/bpf/exception.c +4
-0+return true; + } +return false; +}This isn't a bug, but could
insn_idx_in()just usebsearch()from <linux/bsearch.h>, reusing thecmp_u32()already defined above for thesort()?kernel/bpf/fixups.cin this series usesbsearch()for similar lookups.
Good point. will do.
Quoted 69 lines
Quoted 5 lines
kernel/bpf/fixups.c +0
-0[ ... ]
Quoted 55 lines
kernel/bpf/fixups.c +51
-0@@ -1113,6 +1120,116 @@ static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_s env->subprog_info[env->subprog_cnt].start = env->prog->len; } +static int cleanup_info_for_subprog (struct bpf_verifier_env *env,struct bpf_prog *sub, +u32 start,u32 end) +{ +struct bpf_cleanup_info *recs; +u32 i, cnt =0 ; +int err; + +if (!env->cleanup_info_cnt) +return 0 ; + + err =bpf_cleanup_alloc_info (sub->aux); +if (err) +return err; + + err =cleanup_throw_sites_for_subprog (env, sub, start, end); +if (err) +return err; + + err =cleanup_pad_body_for_subprog (env, sub, start, end); +if (err) +return err; + +for (i = start; i < end; i++) +if (env->insn_aux_data[i].cleanup_pad) + cnt++; +if (!cnt) +return 0 ; + + recs =kvmalloc_array (cnt,sizeof (*recs),GFP_KERNEL_ACCOUNT | __GFP_NOWARN); +if (!recs) +return -ENOMEM ; + +for (i = start, cnt =0 ; i < end; i++) { +u32 pad = env->insn_aux_data[i].cleanup_pad; + +if (!pad) +continue ; + pad--; +if (verifier_bug_if (pad < start || pad >= end, env, +"insn %u is covered by a landing pad at %u outside its subprog [%u, %u)" , + i, pad, start, end)) { +kvfree (recs); +return -EFAULT ; + } + recs[cnt].begin_off = i - start; + recs[cnt].end_off = i - start +1 ; + recs[cnt].landing_pad_off = pad - start; + cnt++; + } +return bpf_cleanup_attach_info (sub->aux, recs, cnt); +}[ ... ]
Quoted 3 lines
static int jit_subprogs(struct bpf_verifier_env *env) { struct bpf_prog *prog = env->prog, **func, *tmp;[ ... ]
Quoted 10 lines
kernel/bpf/fixups.c static int jit_subprogs(struct bpf_verifier_env *env) · line 1371
Quoted 6 lines
kernel/bpf/fixups.c +2
-0@@ -1250,6 +1367,10 @@ static int jit_subprogs(struct bpf_verifier_env *env) func[i]->aux->token = prog->aux->token;if (!i) func[i]->aux->exception_boundary = env->seen_exception; + err =cleanup_info_for_subprog (env, func[i], subprog_start, + env->subprog_info[i +1 ].start);This isn't a bug, but would passing
subprog_end(already computed a few lines above from env->subprog_info[i + 1].start) be clearer here? It keeps the cleanup table's end bound explicitly tied to the same valuejit_subprogs()uses for func[i]->len.
Good point. subprog_end is a good choice.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerPATCH v2 11/20 bpf, x86: Dispatch exception cleanup pads at run time
In reply to Yonghong SongProvide the arch half for x86-64: force the full callee-saved spill for a program carrying a cleanup table so the walker can find a frame's r6-r9 in its callee's prologue, record where that spill area starts, build the native cleanup table from the JIT's addrs[], emit a bare return for a pad's bpf_unwind_resume(), and hand control to a pad from arch_bpf_run_cleanup_pad().
Support is gated on CONFIG_UNWINDER_ORC, the same requirement arch_bpf_stack_walk() and therefore bpf_throw() already have here.
--- arch/x86/net/Makefile | 2 +- arch/x86/net/bpf_cleanup_pad.S | 74 ++++++++++++++++++++++++++++++++ arch/x86/net/bpf_jit_comp.c | 78 ++++++++++++++++++++++++++++------ 3 files changed, 141 insertions(+), 13 deletions(-) create mode 100644 arch/x86/net/bpf_cleanup_pad.S
arch/x86/net/Makefile +1 -1
@@ -6,5 +6,5 @@ ifeq ($(CONFIG_X86_32),y) obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o else - obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o + obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o endif
arch/x86/net/bpf_cleanup_pad.S new file +74 -0
74 lines not on this page · raw message
arch/x86/net/bpf_jit_comp.c +66 -12
170 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 11/20] bpf, x86: Dispatch exception cleanup pads at run time
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] bpf, x86: garbage kfunc stack arguments in exception cleanup pads --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf, x86: garbage kfunc stack arguments in exception cleanup pads --
commit 1e32297a3c2c4f6bbc4d458bc25563bed8167f6c
Author: Yonghong Song <[email protected]>
bpf, x86: Dispatch exception cleanup pads at run time
This commit provides the architecture-specific support for x86-64 exception cleanup pads. It forces a full callee-saved spill for programs with a cleanup table and introduces arch_bpf_run_cleanup_pad to execute the pads.
Quoted 5 lines
arch/x86/net/bpf_cleanup_pad.S new file +0
-0
[ ... ]
Quoted 17 lines
arch/x86/net/bpf_cleanup_pad.S new file +17
-0+SYM_FUNC_START(arch_bpf_run_cleanup_pad) + ANNOTATE_NOENDBR + + pushq %rbp + pushq %rbx + pushq %r12 + pushq %r13 + pushq %r14 + pushq %r15 + /* Keep the pad's entry rsp congruent to a normal call's. */ + subq $8, %rsp + + movq 0(%rdx), %r15 + movq 8(%rdx), %r14 + movq 16(%rdx), %r13 + movq 24(%rdx), %rbx + movq 32(%rdx), %r12
arch/x86/net/bpf_cleanup_pad.S line 64
Quoted 10 lines
arch/x86/net/bpf_cleanup_pad.S new file +10
-0+ /* rbp is BPF r10, so this is the whole of the pad's frame setup. */ + movq %rsi, %rbp + + /* CALL_NOSPEC needs the target in a register; rcx is BPF r4, dead. */ + movq %rdi, %rcx + + /* BPF r0 on the way into a pad, not whatever the kernel left in rax. */ + movl $BPF_PAD_ENTRY_R0, %eax + + CALL_NOSPEC rcx
[Severity: High]
Does this execution environment cause kfuncs with more than 5 arguments to read uninitialized garbage stack arguments?
In arch_bpf_run_cleanup_pad, the pad is executed on the unwinder's stack (rsp), while rbp is set to the original BPF frame.
However, the JIT unconditionally writes outgoing kfunc stack arguments relative to BPF_REG_FP (rbp), as seen in emit_kfunc_arg_moves in arch/x86/net/bpf_jit_comp.c:
else if (dst_mem)
emit_stx(&prog, BPF_DW, BPF_REG_FP, reg,
stack_base + (dst - nreg) * 8);
Because rbp and rsp are desynchronized in the pad, the arguments are written to the old frame. When the kfunc is called, the C calling convention requires it to read stack arguments from [rsp], which points to the unwinder's stack instead of the original frame, leading to garbage reads.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Yonghong Song
reviewerRe: [PATCH bpf-next v2 11/20] bpf, x86: Dispatch exception cleanup pads at run time
Quoted 42 lines · On 9/17/26 10:03 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf, x86: garbage kfunc stack arguments in exception cleanup pads --commit
1e32297a3c2c4f6bbc4d458bc25563bed8167f6c
Author: Yonghong Song <[email protected]>bpf, x86: Dispatch exception cleanup pads at run time
This commit provides the architecture-specific support for x86-64 exception cleanup pads. It forces a full callee-saved spill for programs with a cleanup table and introduces
arch_bpf_run_cleanup_padto execute the pads.Quoted 5 lines
arch/x86/net/bpf_cleanup_pad.S new file +0
-0[ ... ]
Quoted 22 lines
arch/x86/net/bpf_cleanup_pad.S new file +22
-0+SYM_FUNC_START(arch_bpf_run_cleanup_pad) + ANNOTATE_NOENDBR + + pushq %rbp + pushq %rbx + pushq %r12 + pushq %r13 + pushq %r14 + pushq %r15 + /* Keep the pad's entry rsp congruent to a normal call's. */ + subq $8, %rsp + + movq 0(%rdx), %r15 + movq 8(%rdx), %r14 + movq 16(%rdx), %r13 + movq 24(%rdx), %rbx + movq 32(%rdx), %r12 + /* rbp is BPF r10, so this is the whole of the pad's frame setup. */ + movq %rsi, %rbp + + /* CALL_NOSPEC needs the target in a register; rcx is BPF r4, dead. */ + movq %rdi, %rcx
Quoted 8 lines
arch/x86/net/bpf_cleanup_pad.S line 64
Quoted 5 lines
arch/x86/net/bpf_cleanup_pad.S new file +5
-0+ + /* BPF r0 on the way into a pad, not whatever the kernel left in rax. */ + movl $BPF_PAD_ENTRY_R0, %eax + + CALL_NOSPEC rcx[Severity: High]
Does this execution environment cause kfuncs with more than 5 arguments to read uninitialized garbage stack arguments?
We should be okay since we already have:
+ if (bpf_pseudo_kfunc_call(insn)) {
+ struct bpf_call_summary cs;
+
+ if (bpf_get_call_summary(env, insn, &cs) &&
+ cs.arg_slot_cnt > MAX_BPF_FUNC_REG_ARGS) {
+ verbose(env,
+ "insn %u passes an on-stack call argument in an exception cleanup landing pad\n",
+ i);
+ return -EINVAL;
+ }
+ }
Quoted 18 lines
In
arch_bpf_run_cleanup_pad, the pad is executed on the unwinder's stack (rsp), while rbp is set to the original BPF frame.However, the JIT unconditionally writes outgoing kfunc stack arguments relative to
BPF_REG_FP(rbp), as seen inemit_kfunc_arg_movesinarch/x86/net/bpf_jit_comp.c:else if (
dst_mem)
emit_stx(&prog,BPF_DW,BPF_REG_FP, reg,
stack_base+ (dst - nreg) * 8);Because rbp and rsp are desynchronized in the pad, the arguments are written to the old frame. When the kfunc is called, the C calling convention requires it to read stack arguments from [rsp], which points to the unwinder's stack instead of the original frame, leading to garbage reads.
[email protected]
AI reviewRe: [PATCH bpf-next v2 11/20] bpf, x86: Dispatch exception cleanup pads at run time
In reply to Yonghong SongWill this build with CONFIG_STACK_VALIDATION enabled?
Quoted 16 lines
arch/x86/net/Makefile +1
-1@@ -6,5 +6,5 @@ ifeq ($(CONFIG_X86_32),y) obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o else - obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o + obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o endifarch/x86/net/bpf_cleanup_pad.S new file +0
-0
[ ... ]
Quoted 27 lines
arch/x86/net/bpf_cleanup_pad.S new file +27
-0+SYM_FUNC_START(arch_bpf_run_cleanup_pad) + ANNOTATE_NOENDBR + + pushq %rbp + pushq %rbx + pushq %r12 + pushq %r13 + pushq %r14 + pushq %r15 + /* Keep the pad's entry rsp congruent to a normal call's. */ + subq $8, %rsp + + movq 0(%rdx), %r15 + movq 8(%rdx), %r14 + movq 16(%rdx), %r13 + movq 24(%rdx), %rbx + movq 32(%rdx), %r12 + /* rbp is BPF r10, so this is the whole of the pad's frame setup. */ + movq %rsi, %rbp + + /* CALL_NOSPEC needs the target in a register; rcx is BPF r4, dead. */ + movq %rdi, %rcx + + /* BPF r0 on the way into a pad, not whatever the kernel left in rax. */ + movl $BPF_PAD_ENTRY_R0, %eax + + CALL_NOSPEC rcx
arch/x86/net/bpf_cleanup_pad.S line 74
Quoted 10 lines
arch/x86/net/bpf_cleanup_pad.S new file +10
-0+ + addq $8, %rsp + popq %r15 + popq %r14 + popq %r13 + popq %r12 + popq %rbx + popq %rbp + RET +SYM_FUNC_END(arch_bpf_run_cleanup_pad)
Will this build with CONFIG_STACK_VALIDATION enabled?
arch_bpf_run_cleanup_pad() is declared with SYM_FUNC_START (an STT_FUNC symbol) and saves rbp with 'pushq %rbp', but never establishes a frame pointer with 'mov %rsp, %rbp'. Instead, it overwrites rbp with the BPF frame pointer (movq %rsi, %rbp) and then makes a call (CALL_NOSPEC rcx).
objtool's frame-pointer validation (CONFIG_STACK_VALIDATION with UNWINDER_FRAME_POINTER) checks for two conditions in validate_call() and validate_return():
tools/objtool/check.c:has_valid_stack_frame() requires cfi->cfa.base == CFI_BP, which is only set when the function does 'mov %rsp, %rbp' to establish a frame. This function never does that, so the CALL_NOSPEC would trigger:
WARN_INSN(insn, "call without frame pointer save/setup");
Additionally, the 'pushq %rbp' while cfa->base != CFI_BP sets cfi->bp_scratch = true, which is only cleared by 'mov %rsp, %rbp'. This flag is still set at the RET, where validate_return() warns:
WARN_INSN(insn, "BP used as a scratch register");
The Makefile adds bpf_cleanup_pad.o unconditionally under obj-$(CONFIG_BPF_JIT), even though the feature is gated on CONFIG_UNWINDER_ORC per bpf_jit_supports_cleanup_pads(). This means in a CONFIG_STACK_VALIDATION=y build (frame-pointer unwinder), the object would fail objtool validation, and with CONFIG_OBJTOOL_WERROR=y the build would fail.
The neighbouring arch/x86/net/bpf_timed_may_goto.S does not have this issue because it sets up a proper frame:
arch/x86/net/bpf_timed_may_goto.S:
pushq %rbp
movq %rsp, %rbp
...
leave
rbp cannot be the frame pointer here because it must carry BPF r10 into the pad. This is the same situation arch/x86/kernel/ftrace_64.S handles with STACK_FRAME_NON_STANDARD_FP(ftrace_caller). Could either STACK_FRAME_NON_STANDARD_FP(arch_bpf_run_cleanup_pad) after SYM_FUNC_END or building the object only when CONFIG_UNWINDER_ORC is set (matching the feature gate) address this?
arch/x86/net/bpf_jit_comp.c struct jit_context { · line 363
Quoted 12 lines
arch/x86/net/bpf_jit_comp.c +4
-0@@ -357,6 +357,11 @@ struct jit_context {/* Number of bytes that will be skipped on tailcall */ #define X86_TAIL_CALL_OFFSET (12 + ENDBR_INSN_SIZE) +/* Throw-site spill: r15, r14, r13, rbx, r12 low to high, the layout the +* prologue's pushes leave, so arch_bpf_run_cleanup_pad() reads both alike. +*/ +#define X86_CLEANUP_SPILL_SZ (5 * 8)
This isn't a bug, but could the comment format be updated to match the BPF subsystem style? The BPF subsystem guide requires the modern kernel multi-line comment style, where the opening /* is on its own line. The patch already uses this format for its other new multi-line comments (the "Lowest address of each spill area" block and the "Spill r6-r9 and r12" block later in do_jit), so is there a reason for the inconsistency here?
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerRe: [PATCH bpf-next v2 11/20] bpf, x86: Dispatch exception cleanup pads at run time
Quoted 95 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 16 lines
arch/x86/net/Makefile +1
-1@@ -6,5 +6,5 @@ ifeq ($(CONFIG_X86_32),y) obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o else - obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o + obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o endifarch/x86/net/bpf_cleanup_pad.S new file +0
-0[ ... ]
Quoted 27 lines
arch/x86/net/bpf_cleanup_pad.S new file +27
-0+SYM_FUNC_START(arch_bpf_run_cleanup_pad) + ANNOTATE_NOENDBR + + pushq %rbp + pushq %rbx + pushq %r12 + pushq %r13 + pushq %r14 + pushq %r15 + /* Keep the pad's entry rsp congruent to a normal call's. */ + subq $8, %rsp + + movq 0(%rdx), %r15 + movq 8(%rdx), %r14 + movq 16(%rdx), %r13 + movq 24(%rdx), %rbx + movq 32(%rdx), %r12 + /* rbp is BPF r10, so this is the whole of the pad's frame setup. */ + movq %rsi, %rbp + + /* CALL_NOSPEC needs the target in a register; rcx is BPF r4, dead. */ + movq %rdi, %rcx + + /* BPF r0 on the way into a pad, not whatever the kernel left in rax. */ + movl $BPF_PAD_ENTRY_R0, %eax + + CALL_NOSPEC rcxarch/x86/net/bpf_cleanup_pad.S line 74
Quoted 10 lines
arch/x86/net/bpf_cleanup_pad.S new file +10
-0+ + addq $8, %rsp + popq %r15 + popq %r14 + popq %r13 + popq %r12 + popq %rbx + popq %rbp + RET +SYM_FUNC_END(arch_bpf_run_cleanup_pad)Will this build with
CONFIG_STACK_VALIDATIONenabled?
arch_bpf_run_cleanup_pad()is declared withSYM_FUNC_START(anSTT_FUNCsymbol) and saves rbp with 'pushq %rbp', but never establishes a frame pointer with 'mov %rsp, %rbp'. Instead, it overwrites rbp with the BPF frame pointer (movq %rsi, %rbp) and then makes a call (CALL_NOSPECrcx).objtool's frame-pointer validation (
CONFIG_STACK_VALIDATIONwithUNWINDER_FRAME_POINTER) checks for two conditions invalidate_call()andvalidate_return():tools/objtool/check.c:has_valid_stack_frame() requires
cfi->cfa.base==CFI_BP, which is only set when the function does 'mov %rsp, %rbp' to establish a frame. This function never does that, so theCALL_NOSPECwould trigger:WARN_INSN(insn, "call without frame pointer save/setup");Additionally, the 'pushq %rbp' while
cfa->base!=CFI_BPsetscfi->bp_scratch= true, which is only cleared by 'mov %rsp, %rbp'. This flag is still set at the RET, wherevalidate_return()warns:WARN_INSN(insn, "BP used as a scratch register");The
Makefileadds bpf_cleanup_pad.o unconditionally under obj-$(CONFIG_BPF_JIT), even though the feature is gated onCONFIG_UNWINDER_ORCperbpf_jit_supports_cleanup_pads(). This means in aCONFIG_STACK_VALIDATION=ybuild (frame-pointer unwinder), the object would fail objtool validation, and withCONFIG_OBJTOOL_WERROR=ythe build would fail.The neighbouring
arch/x86/net/bpf_timed_may_goto.Sdoes not have this issue because it sets up a proper frame:
arch/x86/net/bpf_timed_may_goto.S:
pushq %rbp
movq %rsp, %rbp
...
leave
Quoted 6 lines
rbp cannot be the frame pointer here because it must carry BPF r10 into the pad. This is the same situation
arch/x86/kernel/ftrace_64.Shandles withSTACK_FRAME_NON_STANDARD_FP(ftrace_caller). Could eitherSTACK_FRAME_NON_STANDARD_FP(arch_bpf_run_cleanup_pad)afterSYM_FUNC_ENDor building the object only whenCONFIG_UNWINDER_ORCis set (matching the feature gate) address this?
Yes, jit only supports CONFIG_UNWINDER_ORC for x86, and CONFIG_STACK_VALIDATION requires CONFIG_UNWINDER_FRAME_POINTER.
Quoted 9 lines
Quoted 9 lines
arch/x86/net/bpf_jit_comp.c +0
-0@@ -357,6 +357,11 @@ struct jit_context {/* Number of bytes that will be skipped on tailcall */ #define X86_TAIL_CALL_OFFSET (12 + ENDBR_INSN_SIZE)
Quoted 11 lines
arch/x86/net/bpf_jit_comp.c struct jit_context { · line 363
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +4
-0+/* Throw-site spill: r15, r14, r13, rbx, r12 low to high, the layout the +* prologue's pushes leave, so arch_bpf_run_cleanup_pad() reads both alike. +*/ +#define X86_CLEANUP_SPILL_SZ (5 * 8)This isn't a bug, but could the comment format be updated to match the BPF subsystem style? The BPF subsystem guide requires the modern kernel multi-line comment style, where the opening /* is on its own line. The patch already uses this format for its other new multi-line comments (the "Lowest address of each spill area" block and the "Spill r6-r9 and r12" block later in
do_jit), so is there a reason for the inconsistency here?
Will adjust comment.
Quoted 9 lines
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerPATCH v2 12/20 bpf, arm64: Dispatch exception cleanup pads at run time
In reply to Yonghong SongThe same arch half for arm64: force the full callee-saved spill, record where it starts, build the native table from the JIT's byte offsets, and hand control to a pad from arch_bpf_run_cleanup_pad().
Unlike x86-64 a pad cannot simply return: every call it makes clobbers x30, so no return address survives to its resume. x23 carries it instead. bpf2a64[] maps nothing to x23 or x24 -- they are pushed only to keep the frame shape the exception callback expects -- so nothing else in generated code touches them, and being callee-saved they survive every kfunc the pad calls. The JIT emits "br x23" for a pad's bpf_unwind_resume().
x24 is the other one, and it is what makes a pad able to touch its own frame at all. Generated code addresses the BPF frame through the stack pointer -- the frame sits directly on top of it, which turns every offset positive and each access into one instruction -- and in a pad the stack pointer is the walker's. So the JIT has a pad recompute the equivalent of its frame's stack pointer from BPF r10 on entry, into x24, and addresses the frame off x24 for every instruction the previous patches marked as running only while unwinding. One instruction per pad, and the accesses keep the shape and the immediate range they have everywhere else.
--- arch/arm64/net/Makefile | 2 +- arch/arm64/net/bpf_cleanup_pad.S | 95 +++++++++++++++++++++++++++++++ arch/arm64/net/bpf_jit_comp.c | 97 ++++++++++++++++++++++++++++++-- 3 files changed, 187 insertions(+), 7 deletions(-) create mode 100644 arch/arm64/net/bpf_cleanup_pad.S
arch/arm64/net/Makefile +1 -1
@@ -2,4 +2,4 @@ # # ARM64 networking code # -obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o +obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o
arch/arm64/net/bpf_cleanup_pad.S new file +95 -0
95 lines not on this page · raw message
arch/arm64/net/bpf_jit_comp.c +91 -6
196 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 12/20] bpf, arm64: Dispatch exception cleanup pads at run time
This isn't a bug, but would it read better to put throw_spill next to stack_size and stack_arg_size, which it is always used with, rather than ahead of prog at the head of the struct?
Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +0
-0
[ ... ]
Quoted 12 lines
arch/arm64/net/bpf_jit_comp.c +8
-0@@ -75,7 +76,21 @@ static const int bpf2a64[] = { [ARENA_VM_START] = A64_R(28), }; +/* Throw-site spill: the five pairs push_callee_regs() forces on, same size and +* slot order, so arch_bpf_run_cleanup_pad() reads both alike. +*/ +#define A64_CLEANUP_SPILL_SZ (5 * 16) + +/* +* Where a landing pad's frame is anchored, since the stack pointer generated +* code normally addresses it through is the walker's inside a pad. bpf2a64[]
arch/arm64/net/bpf_jit_comp.c static const int bpf2a64[] = { · line 96
Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +6
-0+* maps nothing to x24, so nothing else in generated code touches it. +*/ +#define A64_CLEANUP_FP A64_R(24) +struct jit_ctx { +/* Bytes reserved for the throw-site spill; see bpf_cleanup_force_spill(). */ +u32 throw_spill;const struct bpf_prog *prog;int idx;int epilogue_offset;
This isn't a bug, but would it read better to put throw_spill next to stack_size and stack_arg_size, which it is always used with, rather than ahead of prog at the head of the struct?
[ ... ]
Quoted 11 lines
arch/arm64/net/bpf_jit_comp.c +1
-0@@ -1367,6 +1411,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insnconst s16 off = insn->off;const s32 imm = insn->imm;const int i = insn - ctx->prog->insnsi; +const bool in_pad =bpf_cleanup_insn_in_pad (ctx->prog, i);const bool is64 =BPF_CLASS (code) ==BPF_ALU64 ||BPF_CLASS (code) ==BPF_JMP ;u8 jmp_cond;@@ -1378,9 +1423,14 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insnint ret;bool sign_extend;
arch/arm64/net/bpf_jit_comp.c static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn · line 1433
Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +6
-1-if (bpf_insn_is_indirect_target (env, ctx->prog, i)) +if (bpf_insn_is_indirect_target (env, ctx->prog, i) || +bpf_cleanup_insn_is_pad (ctx->prog, i))emit_bti (A64_BTI_J , ctx); +if (bpf_cleanup_insn_is_pad (ctx->prog, i)) +emit (A64_SUB_I (1 ,A64_CLEANUP_FP , fp, + ctx->stack_size + ctx->stack_arg_size), ctx); +
This isn't a bug, but could the two bpf_cleanup_insn_is_pad() calls share a local the way in_pad does, and could the throw-site check further down use the existing 'i' instead of recomputing insn - ctx->prog->insnsi?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerRe: [PATCH bpf-next v2 12/20] bpf, arm64: Dispatch exception cleanup pads at run time
Quoted 23 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +0
-0[ ... ]
Quoted 17 lines
arch/arm64/net/bpf_jit_comp.c +12
-0@@ -75,7 +76,21 @@ static const int bpf2a64[] = { [ARENA_VM_START] = A64_R(28), }; +/* Throw-site spill: the five pairs push_callee_regs() forces on, same size and +* slot order, so arch_bpf_run_cleanup_pad() reads both alike. +*/ +#define A64_CLEANUP_SPILL_SZ (5 * 16) + +/* +* Where a landing pad's frame is anchored, since the stack pointer generated +* code normally addresses it through is the walker's inside a pad. bpf2a64[] +* maps nothing to x24, so nothing else in generated code touches it. +*/ +#define A64_CLEANUP_FP A64_R(24) +struct jit_ctx {
Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c static const int bpf2a64[] = { · line 96
Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +2
-0+/* Bytes reserved for the throw-site spill; see bpf_cleanup_force_spill(). */ +u32 throw_spill;const struct bpf_prog *prog;int idx;int epilogue_offset;This isn't a bug, but would it read better to put
throw_spillnext tostack_sizeandstack_arg_size, which it is always used with, rather than ahead of prog at the head of the struct?
Okay, will do.
Quoted 20 lines
[ ... ]
Quoted 17 lines
arch/arm64/net/bpf_jit_comp.c +3
-1@@ -1367,6 +1411,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insnconst s16 off = insn->off;const s32 imm = insn->imm;const int i = insn - ctx->prog->insnsi; +const bool in_pad =bpf_cleanup_insn_in_pad (ctx->prog, i);const bool is64 =BPF_CLASS (code) ==BPF_ALU64 ||BPF_CLASS (code) ==BPF_JMP ;u8 jmp_cond;@@ -1378,9 +1423,14 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insnint ret;bool sign_extend; -if (bpf_insn_is_indirect_target (env, ctx->prog, i)) +if (bpf_insn_is_indirect_target (env, ctx->prog, i) || +bpf_cleanup_insn_is_pad (ctx->prog, i))emit_bti (A64_BTI_J , ctx);
Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn · line 1433
Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +4
-0+if (bpf_cleanup_insn_is_pad (ctx->prog, i)) +emit (A64_SUB_I (1 ,A64_CLEANUP_FP , fp, + ctx->stack_size + ctx->stack_arg_size), ctx); +This isn't a bug, but could the two
bpf_cleanup_insn_is_pad()calls share a local the wayin_paddoes, and could the throw-site check further down use the existing 'i' instead of recomputing insn -ctx->prog->insnsi?
Will do.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerPATCH v2 13/20 libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
In reply to Yonghong SongLLVM terminates a cleanup landing pad with a call to _Unwind_Resume: the base unwind ABI's entry point for carrying an unwind on once a frame's cleanups have run. The kernel provides that terminator as a kfunc, but not under that name: claiming _Unwind_Resume in the kernel's own symbol table, for a function whose body never runs, would be needlessly confusing, so it is called bpf_unwind_resume.
Both load paths take the detour. A direct load resolves the name against the kernel's BTF while libbpf runs. A light skeleton instead writes the name into the loader program's blob of bytes, for that program to resolve when it runs, so the name recorded there has to be translated as well.
--- tools/lib/bpf/libbpf.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
tools/lib/bpf/libbpf.c +16 -5
@@ -8351,6 +8351,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s } } +/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the +* base unwind ABI's entry point for carrying an unwind on once a frame's +* cleanups have run. The kernel knows it as bpf_unwind_resume. +*/ +static const char *kern_extern_name (const char *name) +{ +return strcmp (name,"_Unwind_Resume" ) ? name :"bpf_unwind_resume" ; +} +static int bpf_program_record_relos (struct bpf_program *prog) {struct bpf_object *obj = prog->obj;
@@ -8367,12 +8376,12 @@ static int bpf_program_record_relos(struct bpf_program *prog)continue ; kind =btf_is_var (btf__type_by_id (obj->btf, ext->btf_id)) ?BTF_KIND_VAR :BTF_KIND_FUNC ; -bpf_gen__record_extern (obj->gen_loader, ext->name, +bpf_gen__record_extern (obj->gen_loader,kern_extern_name (ext->name), ext->is_weak, !ext->ksym.type_id, true, kind, relo->insn_idx);break ;case RELO_EXTERN_CALL : -bpf_gen__record_extern (obj->gen_loader, ext->name, +bpf_gen__record_extern (obj->gen_loader,kern_extern_name (ext->name), ext->is_weak, false, false,BTF_KIND_FUNC , relo->insn_idx);break ;
@@ -8812,17 +8821,19 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,struct module_btf *mod_btf =NULL ;const struct btf_type *kern_func;struct btf *kern_btf =NULL ; +const char *kern_name;int ret; local_func_proto_id = ext->ksym.type_id; - kfunc_id =find_ksym_btf_id (obj, ext->essent_name ?: ext->name,BTF_KIND_FUNC , &kern_btf, - &mod_btf); + kern_name =kern_extern_name (ext->essent_name ?: ext->name); + + kfunc_id =find_ksym_btf_id (obj, kern_name,BTF_KIND_FUNC , &kern_btf, &mod_btf);if (kfunc_id <0 ) {if (kfunc_id == -ESRCH && ext->is_weak)return 0 ;pr_warn ("extern (func ksym) '%s': not found in kernel or module BTFs\n" , - ext->name); +strcmp (kern_name,"bpf_unwind_resume" ) ? ext->name : kern_name);return kfunc_id; }
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v2 14/20 libbpf: Add cleanup_info to bpf_prog_load_opts
In reply to Yonghong SongLet a caller hand the kernel an exception cleanup table. Extend struct bpf_prog_load_opts with cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, pass them through to BPF_PROG_LOAD, and grow the attr size bpf_prog_load() computes to cover the new fields.
--- tools/lib/bpf/bpf.c | 6 +++++- tools/lib/bpf/bpf.h | 7 ++++++- 2 files changed, 11 insertions(+), 2 deletions(-)
tools/lib/bpf/bpf.c +5 -1
@@ -295,7 +295,7 @@ int bpf_prog_load(enum bpf_prog_type prog_type, const struct bpf_insn *insns, size_t insn_cnt, struct bpf_prog_load_opts *opts) { - const size_t attr_sz = offsetofend(union bpf_attr, keyring_id); + const size_t attr_sz = offsetofend(union bpf_attr, cleanup_info_cnt);void *finfo =NULL , *linfo =NULL ;const char *func_info, *line_info;__u32 log_size, log_level, attach_prog_fd, attach_btf_obj_fd;
@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type, attr.fd_array =ptr_to_u64 (OPTS_GET (opts, fd_array,NULL )); attr.fd_array_cnt =OPTS_GET (opts, fd_array_cnt,0 ); + attr.cleanup_info =ptr_to_u64 (OPTS_GET (opts, cleanup_info,NULL )); + attr.cleanup_info_cnt =OPTS_GET (opts, cleanup_info_cnt,0 ); + attr.cleanup_info_rec_size =OPTS_GET (opts, cleanup_info_rec_size,0 ); +if (log_level) { attr.log_buf =ptr_to_u64 (log_buf); attr.log_size = log_size;
tools/lib/bpf/bpf.h +6 -1
@@ -128,9 +128,14 @@ struct bpf_prog_load_opts {/* if set, provides the length of fd_array */ __u32 fd_array_cnt; + +/* exception cleanup table, from the .bpf_cleanup section */ +const void *cleanup_info; +__u32 cleanup_info_cnt; +__u32 cleanup_info_rec_size; size_t :0; }; -#define bpf_prog_load_opts__last_field fd_array_cnt +#define bpf_prog_load_opts__last_field cleanup_info_rec_size LIBBPF_API int bpf_prog_load(enum bpf_prog_type prog_type, const char *prog_name, const char *license,
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 14/20] libbpf: Add cleanup_info to bpf_prog_load_opts
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] libbpf: missing E2BIG fallback for cleanup_info breaks forward compatibility --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] libbpf: missing E2BIG fallback for cleanup_info breaks forward compatibility --
commit d67b69538e39a329708fcc9fa943f0ad05afaa05
Author: Yonghong Song <[email protected]>
libbpf: Add cleanup_info to bpf_prog_load_opts
Let a caller hand the kernel an exception cleanup table. Extend struct bpf_prog_load_opts with cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, pass them through to BPF_PROG_LOAD, and grow the attr size bpf_prog_load() computes to cover the new fields.
Quoted 4 lines
tools/lib/bpf/bpf.c +0
-0
[ ... ]
tools/lib/bpf/bpf.c int bpf_prog_load(enum bpf_prog_type prog_type, · line 375
Quoted 7 lines
tools/lib/bpf/bpf.c +3
-0@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type, attr.fd_array =ptr_to_u64 (OPTS_GET (opts, fd_array,NULL )); attr.fd_array_cnt =OPTS_GET (opts, fd_array_cnt,0 ); + attr.cleanup_info =ptr_to_u64 (OPTS_GET (opts, cleanup_info,NULL )); + attr.cleanup_info_cnt =OPTS_GET (opts, cleanup_info_cnt,0 ); + attr.cleanup_info_rec_size =OPTS_GET (opts, cleanup_info_rec_size,0 );
[Severity: Medium]
Will this break forward compatibility for bpf_prog_load() if cleanup_info triggers an -E2BIG error?
Later in bpf_prog_load(), there is a retry loop to handle -E2BIG when the userspace-provided record size is larger than what the kernel supports:
while (errno == E2BIG && (!finfo || !linfo)) {
if (!finfo && attr.func_info_cnt &&
attr.func_info_rec_size < func_info_rec_size) {
/* try with corrected func info records */
...The loop condition and body don't seem to include checks for cleanup_info.
If the kernel returns -E2BIG because of a larger cleanup_info_rec_size, won't this hit the else { break; } block or skip the loop entirely, causing the load to fail instead of retrying with truncated records?
Quoted 4 lines
tools/lib/bpf/bpf.c +1
-0+if (log_level) { attr.log_buf =ptr_to_u64 (log_buf); attr.log_size = log_size;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
Yonghong Song
reviewerRe: [PATCH bpf-next v2 14/20] libbpf: Add cleanup_info to bpf_prog_load_opts
Quoted 39 lines · On 9/17/26 9:57 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] libbpf: missing E2BIG fallback forcleanup_infobreaks forward compatibility --commit
d67b69538e39a329708fcc9fa943f0ad05afaa05
Author: Yonghong Song <[email protected]>libbpf: Add
cleanup_infotobpf_prog_load_optsLet a caller hand the kernel an exception cleanup table. Extend struct
bpf_prog_load_optswithcleanup_info,cleanup_info_cntandcleanup_info_rec_size, pass them through toBPF_PROG_LOAD, and grow the attr sizebpf_prog_load()computes to cover the new fields.Quoted 4 lines
tools/lib/bpf/bpf.c +0
-0[ ... ]
tools/lib/bpf/bpf.c int bpf_prog_load(enum bpf_prog_type prog_type, · line 375
Quoted 7 lines
tools/lib/bpf/bpf.c +3
-0@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type, attr.fd_array =ptr_to_u64 (OPTS_GET (opts, fd_array,NULL )); attr.fd_array_cnt =OPTS_GET (opts, fd_array_cnt,0 ); + attr.cleanup_info =ptr_to_u64 (OPTS_GET (opts, cleanup_info,NULL )); + attr.cleanup_info_cnt =OPTS_GET (opts, cleanup_info_cnt,0 ); + attr.cleanup_info_rec_size =OPTS_GET (opts, cleanup_info_rec_size,0 );[Severity: Medium]
Will this break forward compatibility forbpf_prog_load()ifcleanup_infotriggers an-E2BIGerror?Later in
bpf_prog_load(), there is a retry loop to handle-E2BIGwhen the userspace-provided record size is larger than what the kernel supports:while (errno == E2BIG && (!finfo || !linfo)) { if (!finfo && attr.func_info_cnt && attr.func_info_rec_size < func_info_rec_size) { /* try with corrected func info records */ ...
Quoted 5 lines
The loop condition and body don't seem to include checks for
cleanup_info.If the kernel returns
-E2BIGbecause of a largercleanup_info_rec_size, won't this hit the else { break; } block or skip the loop entirely, causing the load to fail instead of retrying with truncated records?
This is the first version to support .bpf_cleanup. It is too early to talk about the future extension for .bpf_cleanup although we do have some room to grow struct size if needed:
#define MAX_FUNCINFO_REC_SIZE 252
#define MAX_CLEANUP_INFO_REC_SIZE MAX_FUNCINFO_REC_SIZE
Quoted 5 lines
Quoted 5 lines
tools/lib/bpf/bpf.c +1
-0+if (log_level) { attr.log_buf =ptr_to_u64 (log_buf); attr.log_size = log_size;
Yonghong Song
reviewerPATCH v2 15/20 libbpf: Collect .bpf_cleanup records and pass them to the kernel
In reply to Yonghong SongParse the compiler-emitted .bpf_cleanup section and hand the resulting table to BPF_PROG_LOAD.
Each record is three 4-byte fields, and each field is a byte offset into some code section named by a matching .rel.bpf_cleanup relocation. bpf_object__init_cleanup_info() resolves both halves once at open time and keeps (section, instruction index) pairs; it rejects a record whose field has no relocation, whose relocation is not one of the two 32-bit types that spell a data reference to a code section -- R_BPF_64_NODYLD32 from LLVM, R_BPF_64_ABS32 from GNU as -- or whose offset is not instruction aligned.
The records are sorted by begin_off once the offsets are final: the kernel wants the table sorted with disjoint ranges so that it can find the record covering a call site with a binary search, and records arrive in .bpf_cleanup order, which says nothing about where the subprograms they describe were appended. Overlapping ranges are reported here, where the program name and both regions are still at hand.
bpf_object_load_prog() then passes the per-program table through the bpf_prog_load() options added in the previous patch, with the record size carried on the program the way func_info and line_info carry theirs rather than taken from a sizeof() at the call site. bpf_program__clone() carries it too. That is a second load path -- the one veristat uses -- and without the table the kernel sees landing pads nothing reaches and refuses the program with "unreachable insn".
--- tools/lib/bpf/libbpf.c | 291 ++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf_internal.h | 3 + 2 files changed, 294 insertions(+)
tools/lib/bpf/libbpf.c +291 -0
@@ -514,6 +514,11 @@ struct bpf_program {void *line_info;__u32 line_info_rec_size;__u32 line_info_cnt; + +struct bpf_cleanup_info *cleanup_info; +__u32 cleanup_info_rec_size; +__u32 cleanup_info_cnt; +__u32 prog_flags;__u8 hash[SHA256_DIGEST_LENGTH ];
@@ -549,6 +554,7 @@ struct bpf_struct_ops {#define STRUCT_OPS_SEC ".struct_ops"#define STRUCT_OPS_LINK_SEC ".struct_ops.link"#define ARENA_SEC ".addr_space.1" +#define CLEANUP_SEC ".bpf_cleanup"enum libbpf_map_type {LIBBPF_MAP_UNSPEC ,
@@ -677,6 +683,25 @@ struct elf_sec_desc {Elf_Data *data; }; +#define CLEANUP_REC_FIELDS (sizeof(struct bpf_cleanup_info) / sizeof(__u32)) + +/* Index of each field of struct bpf_cleanup_info, read as an array of __u32. */ +enum { +CLEANUP_REC_BEGIN , +CLEANUP_REC_END , +CLEANUP_REC_PAD , +}; + +/* One (begin, end, landing_pad) triple from .bpf_cleanup, with each field +* resolved from its relocation to an ELF section plus a section-relative +* instruction index. The mapping to final program instruction indices can only +* happen after subprogram placement, which differs per main program. +*/ +struct cleanup_raw_rec { +int sec_idx[CLEANUP_REC_FIELDS ]; +size_t insn_idx[CLEANUP_REC_FIELDS ]; +}; +struct elf_state {int fd;const void *obj_buf;
@@ -696,6 +721,8 @@ struct elf_state {bool has_st_ops;int arena_data_shndx;int jumptables_data_shndx; +Elf_Data *cleanup_data; +int cleanup_shndx; };struct usdt_manager ;
@@ -771,6 +798,9 @@ struct bpf_object {void *jumptables_data;size_t jumptables_data_sz; +struct cleanup_raw_rec *cleanup_recs; +size_t cleanup_rec_cnt; +struct {struct bpf_program *prog;unsigned int sym_off;
@@ -817,7 +847,10 @@ static void bpf_program__exit(struct bpf_program *prog)zfree (&prog->sec_name);zfree (&prog->insns);zfree (&prog->reloc_desc); +zfree (&prog->cleanup_info); + prog->cleanup_info_rec_size =0 ; + prog->cleanup_info_cnt =0 ; prog->nr_reloc =0 ; prog->insns_cnt =0 ; prog->sec_idx =-1 ;
@@ -1559,6 +1592,7 @@ static struct bpf_object *bpf_object__new(const char *path, obj->efile.obj_buf = obj_buf; obj->efile.obj_buf_sz = obj_buf_sz; obj->efile.btf_maps_shndx =-1 ; + obj->efile.cleanup_shndx =-1 ; obj->kconfig_map_idx =-1 ; obj->arena_map_idx =-1 ;
@@ -4045,6 +4079,9 @@ static int bpf_object__elf_collect(struct bpf_object *obj) sec_desc->shdr = sh; sec_desc->data = data; obj->efile.has_st_ops = true; + }else if (strcmp (name,CLEANUP_SEC ) ==0 ) { + obj->efile.cleanup_data = data; + obj->efile.cleanup_shndx = idx; }else if (strcmp (name,ARENA_SEC ) ==0 ) { obj->efile.arena_data = data; obj->efile.arena_data_shndx = idx;
@@ -4072,6 +4109,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj) strcmp(name, ".rel" STRUCT_OPS_LINK_SEC) && strcmp(name, ".rel?" STRUCT_OPS_SEC) && strcmp(name, ".rel?" STRUCT_OPS_LINK_SEC) && + strcmp(name, ".rel" CLEANUP_SEC) && strcmp(name, ".rel" MAPS_ELF_SEC)) { pr_info("elf: skipping relo section(%d) %s for section(%d) %s\n", idx, name, targ_sec_idx,
@@ -4852,6 +4890,221 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,return NULL ; } +static int bpf_object__init_cleanup_info (struct bpf_object *obj) +{ +Elf_Data *data = obj->efile.cleanup_data; +Elf_Data *relo =NULL ; +size_t i, nrels, nslots, nrecs; +struct cleanup_raw_rec *recs; +int *slot_sec, ret =0 ; +size_t *slot_val; +const __u32 *vals; +bool native; + +if (!data || obj->efile.cleanup_shndx <0 ) +return 0 ; + + native =is_native_endianness (obj); + +for (i =0 ; i < obj->efile.sec_cnt; i++) { +struct elf_sec_desc *sd = &obj->efile.secs[i]; + +if (sd->sec_type ==SEC_RELO && sd->shdr && + sd->shdr->sh_info == (Elf64_Word )obj->efile.cleanup_shndx) { + relo = sd->data; +break ; + } + } +if (!relo) { +pr_warn ("%s present without relocations\n" ,CLEANUP_SEC ); +return -LIBBPF_ERRNO__FORMAT ; + } +if (data->d_size %sizeof (struct bpf_cleanup_info )) { +pr_warn ("%s size %zu is not a multiple of the record size %zu\n" , +CLEANUP_SEC , data->d_size,sizeof (struct bpf_cleanup_info )); +return -LIBBPF_ERRNO__FORMAT ; + } + + vals = data->d_buf; + nslots = data->d_size /sizeof (__u32); + nrecs = data->d_size /sizeof (struct bpf_cleanup_info ); + + slot_sec =calloc (nslots,sizeof (*slot_sec)); + slot_val =calloc (nslots,sizeof (*slot_val)); + recs =calloc (nrecs ?:1 ,sizeof (*recs)); +if (!slot_sec || !slot_val || !recs) { + ret = -ENOMEM ; + goto out; + } +for (i =0 ; i < nslots; i++) + slot_sec[i] =-1 ; + +/* One relocation per 4-byte field, naming the section it points into. */ + nrels = relo->d_size /sizeof (Elf64_Rel); +for (i =0 ; i < nrels; i++) { +Elf64_Rel *rel =elf_rel_by_idx (relo, i); +Elf64_Sym *sym =elf_sym_by_idx (obj,ELF64_R_SYM (rel->r_info)); +size_t type =ELF64_R_TYPE (rel->r_info); +size_t slot = rel->r_offset /sizeof (__u32); + +if (type !=R_BPF_64_NODYLD32 && type !=R_BPF_64_ABS32 ) { +pr_warn ("%s: relocation %zu has unexpected type %zu\n" , +CLEANUP_SEC , i, type); + ret = -LIBBPF_ERRNO__FORMAT ; + goto out; + } +if (!sym || slot >= nslots || rel->r_offset %sizeof (__u32)) { +pr_warn ("%s: bad relocation %zu\n" ,CLEANUP_SEC , i); + ret = -LIBBPF_ERRNO__FORMAT ; + goto out; + } + slot_sec[slot] = sym->st_shndx; +/* The addend lives in the section data, which libelf leaves in +* the object's byte order; a non-section symbol additionally +* contributes its own value. +*/ + slot_val[slot] = (native ? vals[slot] :bswap_32 (vals[slot])) + + sym->st_value; + } + +for (i =0 ; i < nslots; i++) { +struct cleanup_raw_rec *rec = &recs[i /CLEANUP_REC_FIELDS ]; +size_t field = i %CLEANUP_REC_FIELDS ; + +if (slot_sec[i] <0 ) { +pr_warn ("%s: field %zu has no relocation\n" ,CLEANUP_SEC , i); + ret = -LIBBPF_ERRNO__FORMAT ; + goto out; + } +if (slot_val[i] %BPF_INSN_SZ ) { +pr_warn ("%s: field %zu offset %zu is not instruction aligned\n" , +CLEANUP_SEC , i, slot_val[i]); + ret = -LIBBPF_ERRNO__FORMAT ; + goto out; + } + rec->sec_idx[field] = slot_sec[i]; + rec->insn_idx[field] = slot_val[i] /BPF_INSN_SZ ; + } + + obj->cleanup_recs = recs; + obj->cleanup_rec_cnt = nrecs; + recs =NULL ; +out: +free (recs); +free (slot_val); +free (slot_sec); +return ret; +} + +static int cmp_cleanup_info (const void *a,const void *b) +{ +const struct bpf_cleanup_info *x = a, *y = b; + +if (x->begin_off == y->begin_off) +return 0 ; +return x->begin_off < y->begin_off ?-1 :1 ; +} + +static int bpf_prog_collect_cleanup_info (struct bpf_object *obj, +struct bpf_program *prog) +{ +size_t i; +int j; + +for (i =0 ; i < obj->cleanup_rec_cnt; i++) { +struct cleanup_raw_rec *raw = &obj->cleanup_recs[i]; +struct bpf_program *owner =NULL ; +struct bpf_cleanup_info ci = {}; +__u32 fields[CLEANUP_REC_FIELDS ]; +void *tmp; + +for (j =0 ; j <CLEANUP_REC_FIELDS ; j++) { +size_t idx = raw->insn_idx[j], final; +struct bpf_program *p; + +/* The end of a range is exclusive, so it may name the +* instruction just past the last one of a function, +* which belongs to the next function or to nothing at +* all. Ask about the last instruction the range covers, +* the way the kernel does. +*/ +if (j ==CLEANUP_REC_END ) { +if (!idx) { +pr_warn ("%s: record %zu is an empty range\n" , +CLEANUP_SEC , i); +return -LIBBPF_ERRNO__FORMAT ; + } + idx--; + } + + p =find_prog_by_sec_insn (obj, raw->sec_idx[j], idx); +if (!p) { +pr_warn ("%s: record %zu field %d is not inside a function\n" , +CLEANUP_SEC , i, j); +return -LIBBPF_ERRNO__FORMAT ; + } +if (!owner) { + owner = p; + }else if (owner != p) { +pr_warn ("%s: record %zu spans functions '%s' and '%s'\n" , +CLEANUP_SEC , i, owner->name, p->name); +return -LIBBPF_ERRNO__FORMAT ; + } + +if (owner == prog) { + final = raw->insn_idx[j] - prog->sec_insn_off; + }else if (prog_is_subprog (obj, owner) && owner->sub_insn_off) { +/* sub_insn_off is where this subprogram was +* appended to the main program being relocated; +* zero means it is not part of it. +*/ + final = owner->sub_insn_off + + raw->insn_idx[j] - owner->sec_insn_off; + }else { + owner =NULL ; +break ; + } + fields[j] = final; + } +if (!owner) +continue ; + + ci.begin_off = fields[CLEANUP_REC_BEGIN ]; + ci.end_off = fields[CLEANUP_REC_END ]; + ci.landing_pad_off = fields[CLEANUP_REC_PAD ]; + + tmp =libbpf_reallocarray (prog->cleanup_info, prog->cleanup_info_cnt +1 , +sizeof (*prog->cleanup_info)); +if (!tmp) +return -ENOMEM ; + prog->cleanup_info = tmp; + prog->cleanup_info_rec_size =sizeof (struct bpf_cleanup_info ); + prog->cleanup_info[prog->cleanup_info_cnt++] = ci; + +pr_debug ("prog '%s': cleanup region [%u,%u) -> landing pad %u\n" , + prog->name, ci.begin_off, ci.end_off, ci.landing_pad_off); + } + +if (!prog->cleanup_info_cnt) +return 0 ; + +qsort (prog->cleanup_info, prog->cleanup_info_cnt, +sizeof (*prog->cleanup_info), cmp_cleanup_info); +for (i =1 ; i < prog->cleanup_info_cnt; i++) { +struct bpf_cleanup_info *prev = &prog->cleanup_info[i -1 ]; +struct bpf_cleanup_info *cur = &prog->cleanup_info[i]; + +if (cur->begin_off < prev->end_off) { +pr_warn ("prog '%s': overlapping cleanup regions [%u,%u) and [%u,%u)\n" , + prog->name, prev->begin_off, prev->end_off, + cur->begin_off, cur->end_off); +return -LIBBPF_ERRNO__FORMAT ; + } + } + +return 0 ; +} +static int bpf_object__collect_prog_relos (struct bpf_object *obj,Elf64_Shdr *shdr,Elf_Data *data) {
@@ -7561,6 +7814,13 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_patreturn err; } } + + err =bpf_prog_collect_cleanup_info (obj, prog); +if (err) { +pr_warn ("prog '%s': failed to collect cleanup info: %s\n" , + prog->name,errstr (err)); +return err; + } }for (i =0 ; i < obj->nr_programs; i++) { prog = &obj->programs[i];
@@ -7751,6 +8011,9 @@ static int bpf_object__collect_relos(struct bpf_object *obj)return -LIBBPF_ERRNO__INTERNAL ; } +if (idx == obj->efile.cleanup_shndx) +continue ; +if (obj->efile.secs[idx].sec_type ==SEC_ST_OPS ) err =bpf_object__collect_st_ops_relos (obj, shdr, data);else if (idx == obj->efile.btf_maps_shndx)
@@ -8023,6 +8286,11 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog load_attr.line_info_rec_size = prog->line_info_rec_size; load_attr.line_info_cnt = prog->line_info_cnt; } +if (prog->cleanup_info_cnt) { + load_attr.cleanup_info = prog->cleanup_info; + load_attr.cleanup_info_cnt = prog->cleanup_info_cnt; + load_attr.cleanup_info_rec_size = prog->cleanup_info_rec_size; + } load_attr.log_level = log_level; load_attr.prog_flags = prog->prog_flags; load_attr.fd_array = obj->fd_array;
@@ -8579,6 +8847,7 @@ static struct bpf_object *bpf_object_open(const char *path, const void *obj_buf, err = err ? :bpf_object__init_maps (obj, opts); err = err ? :bpf_object_init_progs (obj, opts); err = err ? :bpf_object__collect_relos (obj); + err = err ? :bpf_object__init_cleanup_info (obj);if (err) goto out;
@@ -9692,6 +9961,9 @@ void bpf_object__close(struct bpf_object *obj)zfree (&obj->jumptables_data); obj->jumptables_data_sz =0 ; +zfree (&obj->cleanup_recs); + obj->cleanup_rec_cnt =0 ; +for (i =0 ; i < obj->jumptable_map_cnt; i++)close (obj->jumptable_maps[i].fd);zfree (&obj->jumptable_maps);
@@ -10084,6 +10356,25 @@ int bpf_program__clone(struct bpf_program *prog, const struct bpf_prog_load_opts attr.line_info_rec_size = info ? info_rec_size : prog->line_info_rec_size; } +/* exception cleanup table */ + info =OPTS_GET (opts, cleanup_info,NULL ); + info_cnt =OPTS_GET (opts, cleanup_info_cnt,0 ); + info_rec_size =OPTS_GET (opts, cleanup_info_rec_size,0 ); +if (!!info != !!info_cnt || !!info != !!info_rec_size) { +pr_warn ("prog '%s': cleanup_info, cleanup_info_cnt, and cleanup_info_rec_size must all be specified or all omitted\n" , + prog->name); +return libbpf_err (-EINVAL ); + } +if (info) { + attr.cleanup_info = info; + attr.cleanup_info_cnt = info_cnt; + attr.cleanup_info_rec_size = info_rec_size; + }else if (prog->cleanup_info_cnt) { + attr.cleanup_info = prog->cleanup_info; + attr.cleanup_info_cnt = prog->cleanup_info_cnt; + attr.cleanup_info_rec_size = prog->cleanup_info_rec_size; + } +/* Logging is caller-controlled; no fallback to prog/obj log settings */ attr.log_buf =OPTS_GET (opts, log_buf,NULL ); attr.log_size =OPTS_GET (opts, log_size,0 );
tools/lib/bpf/libbpf_internal.h +3 -0
@@ -56,6 +56,9 @@#ifndef R_BPF_64_ABS32 #define R_BPF_64_ABS32 3#endif +#ifndef R_BPF_64_NODYLD32 +#define R_BPF_64_NODYLD32 4 +#endif #ifndef R_BPF_64_32 #define R_BPF_64_32 10#endif
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 15/20] libbpf: Collect .bpf_cleanup records and pass them to the kernel
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] bpf: empty cleanup range check in bpf_prog_collect_cleanup_info() only catches offset 0…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] bpf: empty cleanup range check in bpf_prog_collect_cleanup_info() only catches offset 0
- [Medium] bpf: bpf_object__init_cleanup_info() rejects valid objects with 0-sized .bpf_cleanup sections --
commit a645d3bd249aceee4909edcc78e5fe4247fd3432
Author: Yonghong Song <[email protected]>
libbpf: Collect .bpf_cleanup records and pass them to the kernel
Parse the compiler-emitted .bpf_cleanup section and hand the resulting table to BPF_PROG_LOAD.
Quoted 4 lines
tools/lib/bpf/libbpf.c +0
-0
[ ... ]
Quoted 23 lines
tools/lib/bpf/libbpf.c +19
-0@@ -4852,6 +4890,221 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,return NULL ; } +static int bpf_object__init_cleanup_info (struct bpf_object *obj) +{ +Elf_Data *data = obj->efile.cleanup_data; +Elf_Data *relo =NULL ; +size_t i, nrels, nslots, nrecs; +struct cleanup_raw_rec *recs; +int *slot_sec, ret =0 ; +size_t *slot_val; +const __u32 *vals; +bool native; + +if (!data || obj->efile.cleanup_shndx <0 ) +return 0 ; + + native =is_native_endianness (obj); + +for (i =0 ; i < obj->efile.sec_cnt; i++) { +struct elf_sec_desc *sd = &obj->efile.secs[i]; +
tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 4921
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+if (sd->sec_type ==SEC_RELO && sd->shdr && + sd->shdr->sh_info == (Elf64_Word )obj->efile.cleanup_shndx) { + relo = sd->data; +break ; + } + } +if (!relo) { +pr_warn ("%s present without relocations\n" ,CLEANUP_SEC ); +return -LIBBPF_ERRNO__FORMAT ; + }
[Severity: Medium]
Could this reject valid objects if the compiler emits an empty 0-byte .bpf_cleanup section?
If the section is empty (e.g., exceptions were optimized out), no relocations are emitted and there is no corresponding .rel.bpf_cleanup section. However, bpf_object__init_cleanup_info() requires the relocation section to exist if .bpf_cleanup is present.
Would it be better to return early if data->d_size is 0 before checking for relocations?
[ ... ]
Quoted 21 lines
tools/lib/bpf/libbpf.c +21
-0+static int bpf_prog_collect_cleanup_info (struct bpf_object *obj, +struct bpf_program *prog) +{ +size_t i; +int j; + +for (i =0 ; i < obj->cleanup_rec_cnt; i++) { +struct cleanup_raw_rec *raw = &obj->cleanup_recs[i]; +struct bpf_program *owner =NULL ; +struct bpf_cleanup_info ci = {}; +__u32 fields[CLEANUP_REC_FIELDS ]; +void *tmp; + +for (j =0 ; j <CLEANUP_REC_FIELDS ; j++) { +size_t idx = raw->insn_idx[j], final; +struct bpf_program *p; + +/* The end of a range is exclusive, so it may name the +* instruction just past the last one of a function, +* which belongs to the next function or to nothing at +* all. Ask about the last instruction the range covers,
tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 5038
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+* the way the kernel does. +*/ +if (j ==CLEANUP_REC_END ) { +if (!idx) { +pr_warn ("%s: record %zu is an empty range\n" , +CLEANUP_SEC , i); +return -LIBBPF_ERRNO__FORMAT ; + } + idx--; + }
[Severity: Medium]
Will this check catch empty ranges emitted by the compiler at non-zero offsets?
If the compiler emits a .bpf_cleanup record where begin and end are both 5, the check for !idx evaluates to false. The code will then decrement idx to 4 and look up the program for the previous instruction.
If this empty range is located at the start of a function, looking up the previous instruction will fall into the preceding function, potentially triggering a spurious "spans functions" error or silently accepting invalid records.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
Yonghong Song
reviewerRe: [PATCH bpf-next v2 15/20] libbpf: Collect .bpf_cleanup records and pass them to the kernel
Quoted 56 lines · On 9/17/26 10:00 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] bpf: empty cleanup range check inbpf_prog_collect_cleanup_info()only catches offset 0
- [Medium] bpf:bpf_object__init_cleanup_info()rejects valid objects with 0-sized .bpf_cleanup sections --commit
a645d3bd249aceee4909edcc78e5fe4247fd3432
Author: Yonghong Song <[email protected]>libbpf: Collect .bpf_cleanup records and pass them to the kernel
Parse the compiler-emitted .bpf_cleanup section and hand the resulting table to
BPF_PROG_LOAD.Quoted 4 lines
tools/lib/bpf/libbpf.c +0
-0[ ... ]
Quoted 23 lines
tools/lib/bpf/libbpf.c +19
-0@@ -4852,6 +4890,221 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,return NULL ; } +static int bpf_object__init_cleanup_info (struct bpf_object *obj) +{ +Elf_Data *data = obj->efile.cleanup_data; +Elf_Data *relo =NULL ; +size_t i, nrels, nslots, nrecs; +struct cleanup_raw_rec *recs; +int *slot_sec, ret =0 ; +size_t *slot_val; +const __u32 *vals; +bool native; + +if (!data || obj->efile.cleanup_shndx <0 ) +return 0 ; + + native =is_native_endianness (obj); + +for (i =0 ; i < obj->efile.sec_cnt; i++) { +struct elf_sec_desc *sd = &obj->efile.secs[i]; +tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 4921
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+if (sd->sec_type ==SEC_RELO && sd->shdr && + sd->shdr->sh_info == (Elf64_Word )obj->efile.cleanup_shndx) { + relo = sd->data; +break ; + } + } +if (!relo) { +pr_warn ("%s present without relocations\n" ,CLEANUP_SEC ); +return -LIBBPF_ERRNO__FORMAT ; + }[Severity: Medium]
Could this reject valid objects if the compiler emits an empty 0-byte .bpf_cleanup section?
Quoted 7 lines
If the section is empty (e.g., exceptions were optimized out), no relocations are emitted and there is no corresponding .rel.bpf_cleanup section. However,
bpf_object__init_cleanup_info()requires the relocation section to exist if .bpf_cleanup is present.Would it be better to return early if
data->d_sizeis 0 before checking for relocations?
I think it is okay, as you mentioned, there will be one error message later when relocation section checks.
Quoted 38 lines
[ ... ]
Quoted 21 lines
tools/lib/bpf/libbpf.c +21
-0+static int bpf_prog_collect_cleanup_info (struct bpf_object *obj, +struct bpf_program *prog) +{ +size_t i; +int j; + +for (i =0 ; i < obj->cleanup_rec_cnt; i++) { +struct cleanup_raw_rec *raw = &obj->cleanup_recs[i]; +struct bpf_program *owner =NULL ; +struct bpf_cleanup_info ci = {}; +__u32 fields[CLEANUP_REC_FIELDS ]; +void *tmp; + +for (j =0 ; j <CLEANUP_REC_FIELDS ; j++) { +size_t idx = raw->insn_idx[j], final; +struct bpf_program *p; + +/* The end of a range is exclusive, so it may name the +* instruction just past the last one of a function, +* which belongs to the next function or to nothing at +* all. Ask about the last instruction the range covers, tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 5038
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+* the way the kernel does. +*/ +if (j ==CLEANUP_REC_END ) { +if (!idx) { +pr_warn ("%s: record %zu is an empty range\n" , +CLEANUP_SEC , i); +return -LIBBPF_ERRNO__FORMAT ; + } + idx--; + }[Severity: Medium]
Will this check catch empty ranges emitted by the compiler at non-zero offsets?
Quoted 8 lines
If the compiler emits a .bpf_cleanup record where begin and end are both 5, the check for !idx evaluates to false. The code will then decrement idx to 4 and look up the program for the previous instruction.
If this empty range is located at the start of a function, looking up the previous instruction will fall into the preceding function, potentially triggering a spurious "spans functions" error or silently accepting invalid records.
Good point. Will fix this corner case.
Quoted 1 line
Yonghong Song
reviewerPATCH v2 16/20 libbpf: Carry the exception cleanup table through the light skeleton
In reply to Yonghong SongA light skeleton does not call bpf_prog_load(). bpf_gen__prog_load() builds its own union bpf_attr field by field, and the loader program it emits is what issues BPF_PROG_LOAD when the skeleton runs -- so a program loaded this way reached the kernel without the table the previous patch collected for it. The load then fails, because the landing pads are code nothing reaches and the verifier says so. It says "unreachable insn", which names neither the skeleton nor the table.
Carry it the way func_info and line_info are carried: the records go into the loader's blob of bytes, the count and record size into the attr, and a relocation stores the blob's address into attr.cleanup_info once that address is known. The attr grows to its new last field, cleanup_info_cnt. Records are 4-byte fields like the other info blobs, so a cross-endian build has to swap them too.
Growing the attr is what makes the relocation conditional. Every other attr in this file stops at the last field it sets, and this one now runs to the end of the union: a kernel that predates cleanup_info accepts an attr longer than its own only while the tail it does not know reads as zero. The count and the record size are already zero for a program with no table, but the relocation stores a blob address, which never is, so a program that has no landing pads only keeps loading on such a kernel if the relocation is left out.
--- tools/lib/bpf/gen_loader.c | 29 +++++++++++++++++++++++++---- tools/lib/bpf/libbpf_internal.h | 7 +++++++ 2 files changed, 32 insertions(+), 4 deletions(-)
tools/lib/bpf/gen_loader.c +25 -4
@@ -981,13 +981,15 @@ static void cleanup_relos(struct bpf_gen *gen, int insns)cleanup_core_relo (gen); } -/* Convert func, line, and core relo info blobs to target endianness */ +/* Convert func, line, core relo and cleanup info blobs to target endianness */ static void info_blob_bswap (struct bpf_gen *gen,int func_info,int line_info, -int core_relos,struct bpf_prog_load_opts *load_attr) +int core_relos,int cleanup_info, +struct bpf_prog_load_opts *load_attr) {struct bpf_func_info *fi = gen->data_start + func_info;struct bpf_line_info *li = gen->data_start + line_info;struct bpf_core_relo *cr = gen->data_start + core_relos; +struct bpf_cleanup_info *ci = gen->data_start + cleanup_info;int i;for (i =0 ; i < load_attr->func_info_cnt; i++)
@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,for (i =0 ; i < gen->core_relo_cnt; i++)bpf_core_relo_bswap (cr++); + +for (i =0 ; i < load_attr->cleanup_info_cnt; i++) +bpf_cleanup_info_bswap (ci++); } void bpf_gen__prog_load(struct bpf_gen *gen,
@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen, load_attr->line_info_rec_size;int core_relo_tot_sz = gen->core_relo_cnt *sizeof (struct bpf_core_relo ); +int cleanup_info_tot_sz = load_attr->cleanup_info_cnt * + load_attr->cleanup_info_rec_size;int prog_load_attr, license_off, insns_off, func_info, line_info, core_relos; - int attr_size = offsetofend(union bpf_attr, core_relo_rec_size); + int attr_size = offsetofend(union bpf_attr, cleanup_info_cnt); +int cleanup_info;union bpf_attr attr;memset (&attr,0 , attr_size);
@@ -1061,9 +1069,17 @@ void bpf_gen__prog_load(struct bpf_gen *gen, core_relos, gen->core_relo_cnt, sizeof(struct bpf_core_relo)); + attr.cleanup_info_rec_size =tgt_endian (load_attr->cleanup_info_rec_size); + attr.cleanup_info_cnt =tgt_endian (load_attr->cleanup_info_cnt); + cleanup_info =add_data (gen, load_attr->cleanup_info, cleanup_info_tot_sz); +pr_debug ("gen: prog_load: cleanup_info: off %d cnt %u rec size %u\n" , + cleanup_info, load_attr->cleanup_info_cnt, + load_attr->cleanup_info_rec_size); +/* convert all info blobs to target endianness */ if (gen->swapped_endian && !gen->error) -info_blob_bswap (gen, func_info, line_info, core_relos, load_attr); +info_blob_bswap (gen, func_info, line_info, core_relos, cleanup_info, + load_attr);libbpf_strlcpy (attr.prog_name, prog_name,sizeof (attr.prog_name)); prog_load_attr =add_data (gen, &attr, attr_size);
@@ -1085,6 +1101,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,/* populate union bpf_attr with a pointer to core_relos */ emit_rel_store (gen,attr_field (prog_load_attr, core_relos), core_relos); +/* populate union bpf_attr with a pointer to cleanup_info, if there is one */ +if (load_attr->cleanup_info_cnt) +emit_rel_store (gen,attr_field (prog_load_attr, cleanup_info), + cleanup_info); +/* populate union bpf_attr fd_array with a pointer to data where map_fds are saved */ emit_rel_store (gen,attr_field (prog_load_attr, fd_array), gen->fd_array);
tools/lib/bpf/libbpf_internal.h +7 -0
@@ -572,6 +572,13 @@ static inline void bpf_core_relo_bswap(struct bpf_core_relo *i) i->kind =bswap_32 (i->kind); } +static inline void bpf_cleanup_info_bswap (struct bpf_cleanup_info *i) +{ + i->begin_off =bswap_32 (i->begin_off); + i->end_off =bswap_32 (i->end_off); + i->landing_pad_off =bswap_32 (i->landing_pad_off); +} +enum btf_field_iter_kind {BTF_FIELD_ITER_IDS ,BTF_FIELD_ITER_STRS ,
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 16/20] libbpf: Carry the exception cleanup table through the light skeleton
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] libbpf: integer overflow in cleanup_info_tot_sz leads to heap OOB read/write --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] libbpf: integer overflow in cleanup_info_tot_sz leads to heap OOB read/write --
commit b9300a1f6022da2aa8b343e05882b46e9f329c64
Author: Yonghong Song <[email protected]>
libbpf: Carry the exception cleanup table through the light skeleton
This patch modifies the light skeleton generation to carry the exception cleanup table (cleanup_info), similar to func_info and line_info. It updates bpf_gen__prog_load() to allocate space for the records, set up union bpf_attr, and perform endianness swapping for cross-compiled cases.
Quoted 4 lines
tools/lib/bpf/gen_loader.c +0
-0
[ ... ]
tools/lib/bpf/gen_loader.c static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info, · line 1005
Quoted 7 lines
tools/lib/bpf/gen_loader.c +3
-0@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,for (i =0 ; i < gen->core_relo_cnt; i++)bpf_core_relo_bswap (cr++); + +for (i =0 ; i < load_attr->cleanup_info_cnt; i++) +bpf_cleanup_info_bswap (ci++);
[Severity: High]
Could this loop result in an out-of-bounds heap read and write?
If an attacker crafts a BPF ELF object with an extremely large .bpf_cleanup section, load_attr->cleanup_info_cnt can be very large. If the allocation size calculated in bpf_gen__prog_load() overflowed, this loop will process hundreds of millions of elements in an undersized buffer.
This could cause memory corruption when cross-compiling (gen->swapped_endian is true).
tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1020
Quoted 9 lines
tools/lib/bpf/gen_loader.c +2
-0} void bpf_gen__prog_load(struct bpf_gen *gen,@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen, load_attr->line_info_rec_size;int core_relo_tot_sz = gen->core_relo_cnt *sizeof (struct bpf_core_relo ); +int cleanup_info_tot_sz = load_attr->cleanup_info_cnt * + load_attr->cleanup_info_rec_size;
[Severity: High]
Does this unbounded 32-bit integer multiplication risk an overflow?
If load_attr->cleanup_info_cnt is excessively large, multiplying it by cleanup_info_rec_size could wrap around modulo 2^32 to a small positive integer. This would result in add_data() allocating an undersized heap buffer for the cleanup info blob.
Combined with the loop in info_blob_bswap(), this could lead to crashes or potentially arbitrary code execution when processing untrusted files using the light skeleton generator.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
Yonghong Song
reviewerRe: [PATCH bpf-next v2 16/20] libbpf: Carry the exception cleanup table through the light skeleton
Quoted 29 lines · On 9/17/26 10:02 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] libbpf: integer overflow incleanup_info_tot_szleads to heap OOB read/write --commit
b9300a1f6022da2aa8b343e05882b46e9f329c64
Author: Yonghong Song <[email protected]>libbpf: Carry the exception cleanup table through the light skeleton
This patch modifies the light skeleton generation to carry the exception cleanup table (
cleanup_info), similar tofunc_infoandline_info. It updatesbpf_gen__prog_load()to allocate space for the records, set up unionbpf_attr, and perform endianness swapping for cross-compiled cases.Quoted 4 lines
tools/lib/bpf/gen_loader.c +0
-0[ ... ]
tools/lib/bpf/gen_loader.c static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info, · line 1005
Quoted 7 lines
tools/lib/bpf/gen_loader.c +3
-0@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,for (i =0 ; i < gen->core_relo_cnt; i++)bpf_core_relo_bswap (cr++); + +for (i =0 ; i < load_attr->cleanup_info_cnt; i++) +bpf_cleanup_info_bswap (ci++);[Severity: High]
Could this loop result in an out-of-bounds heap read and write?
Quoted 7 lines
If an attacker crafts a BPF ELF object with an extremely large .bpf_cleanup section,
load_attr->cleanup_info_cntcan be very large. If the allocation size calculated inbpf_gen__prog_load()overflowed, this loop will process hundreds of millions of elements in an undersized buffer.This could cause memory corruption when cross-compiling (
gen->swapped_endianis true).
It is user's problem if they intentionally want to have of lots and lots of .bpf_cleanup section.
Quoted 13 lines
tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1020
Quoted 10 lines
tools/lib/bpf/gen_loader.c +2
-0} void bpf_gen__prog_load(struct bpf_gen *gen,@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen, load_attr->line_info_rec_size;int core_relo_tot_sz = gen->core_relo_cnt *sizeof (struct bpf_core_relo ); +int cleanup_info_tot_sz = load_attr->cleanup_info_cnt * + load_attr->cleanup_info_rec_size;[Severity: High]
Does this unbounded 32-bit integer multiplication risk an overflow?
Quoted 8 lines
If
load_attr->cleanup_info_cntis excessively large, multiplying it bycleanup_info_rec_sizecould wrap around modulo 2^32 to a small positive integer. This would result inadd_data()allocating an undersized heap buffer for the cleanup info blob.Combined with the loop in
info_blob_bswap(), this could lead to crashes or potentially arbitrary code execution when processing untrusted files using the light skeleton generator.
We should be okay. Similar to line_info, core_relo, etc., all of them (including cleanup_info) will have a guard in add_data which ensures the total size must be smaller than INT32_MAX.
Quoted 1 line
Yonghong Song
reviewerPATCH v2 17/20 libbpf: Let the static linker carry .bpf_cleanup relocations
In reply to Yonghong SongAn object that carries a compiler-emitted exception cleanup table cannot be linked today. The table's fields are byte offsets into a code section, materialised by a 32-bit relocation against that section's symbol with the offset itself as the implicit addend, and the linker rejects both halves of
that: the relocation type is not in the list it accepts, and a relocation against an STT_SECTION symbol from a non-executable section is an outright error.
Both spellings of that relocation have to be taken. LLVM emits R_BPF_64_NODYLD32 for a .long against a section symbol; GNU as emits R_BPF_64_ABS32, which is what bpf_reloc_type_lookup() maps BFD_RELOC_32 to. They describe the same value, and the selftests are built with both compilers.
--- tools/lib/bpf/linker.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)
tools/lib/bpf/linker.c +18 -1
@@ -1036,7 +1036,8 @@ static int linker_sanity_check_elf_relos(struct src_obj *obj, struct src_sec *sesize_t sym_type =ELF64_R_TYPE (relo->r_info);if (sym_type !=R_BPF_64_64 && sym_type !=R_BPF_64_32 && - sym_type !=R_BPF_64_ABS64 && sym_type !=R_BPF_64_ABS32 ) { + sym_type !=R_BPF_64_ABS64 && sym_type !=R_BPF_64_ABS32 && + sym_type !=R_BPF_64_NODYLD32 ) {pr_warn ("ELF relo #%d in section #%zu has unexpected type %zu in %s\n" , i, sec->sec_idx, sym_type, obj->filename);return -EINVAL ;
@@ -2274,6 +2275,22 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; + }else if (sym_type ==R_BPF_64_NODYLD32 || + sym_type ==R_BPF_64_ABS32 ) { +__u32 *val; + +/* Two spellings of the one thing: LLVM +* emits NODYLD32 for a .long against a +* section symbol, GNU as emits ABS32 +* (bpf_reloc_type_lookup() maps +* BFD_RELOC_32 to it), and the value +* they describe is the same. +*/ + val = dst_linked_sec->raw_data + dst_rel->r_offset; +if (linker->swapped_endian) + *val =bswap_32 (bswap_32 (*val) + sec->dst_off); +else + *val += sec->dst_off; }else {pr_warn ("relocation against STT_SECTION in non-exec section is not supported!\n" );return -EINVAL ;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] bpf/linker: NULL pointer deref in linker_append_elf_relos() - [Medium] bpf/linker: unaligned…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] bpf/linker: NULL pointer deref in linker_append_elf_relos()
- [Medium] bpf/linker: unaligned memory access in data relocations
- [Low] bpf/linker: multi-line comment style violation
--
commit 2eadf13e135621b2e34d8946fe67664cd72e0e3d
Author: Yonghong Song <[email protected]>
libbpf: Let the static linker carry .bpf_cleanup relocations
This commit updates the BPF static linker to support .bpf_cleanup relocations. It specifically adds handling for R_BPF_64_NODYLD32 and R_BPF_64_ABS32 to process compiler-emitted exception cleanup tables.
Quoted 4 lines
tools/lib/bpf/linker.c +0
-0
[ ... ]
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2282
Quoted 9 lines
tools/lib/bpf/linker.c +5
-0@@ -2274,6 +2275,22 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; + }else if (sym_type ==R_BPF_64_NODYLD32 || + sym_type ==R_BPF_64_ABS32 ) { +__u32 *val; + +/* Two spellings of the one thing: LLVM
[Severity: Low]
This isn't a bug, but does this violate the BPF subsystem multi-line comment style? The preferred kernel style is to have the opening /* on its own line rather than putting text on the same line as the marker.
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2289
Quoted 7 lines
tools/lib/bpf/linker.c +7
-0+* emits NODYLD32 for a .long against a +* section symbol, GNU as emits ABS32 +* (bpf_reloc_type_lookup() maps +* BFD_RELOC_32 to it), and the value +* they describe is the same. +*/ + val = dst_linked_sec->raw_data + dst_rel->r_offset;
[Severity: High]
Could this lead to a NULL pointer dereference? Looking at linker_sanity_check_elf_relos(), SHT_NOBITS sections are explicitly allowed as relocatable sections.
For SHT_NOBITS sections (like .bss), extend_sec() typically skips allocating raw_data, leaving it as NULL. If we process a relocation against such a section here, we might compute val = NULL + dst_rel->r_offset and then dereference it on the next line when handling sec->dst_off.
[Severity: Medium]
Is it also possible to hit an unaligned memory access here?
While executable sections enforce alignment checks in linker_sanity_check_elf_relos(), there doesn't seem to be an equivalent alignment check for data section offsets. If an ELF provides an unaligned r_offset, the direct dereference of the unaligned val pointer might cause a crash on architectures with strict memory alignment requirements.
Quoted 7 lines
tools/lib/bpf/linker.c +4
-0+if (linker->swapped_endian) + *val =bswap_32 (bswap_32 (*val) + sec->dst_off); +else + *val += sec->dst_off; }else {pr_warn ("relocation against STT_SECTION in non-exec section is not supported!\n" );return -EINVAL ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
Yonghong Song
reviewerRe: [PATCH bpf-next v2 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations
Quoted 52 lines · On 9/17/26 10:01 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] bpf/linker: NULL pointer deref inlinker_append_elf_relos()
- [Medium] bpf/linker: unaligned memory access in data relocations
- [Low] bpf/linker: multi-line comment style violation
--commit
2eadf13e135621b2e34d8946fe67664cd72e0e3d
Author: Yonghong Song <[email protected]>libbpf: Let the static linker carry .bpf_cleanup relocations
This commit updates the BPF static linker to support .bpf_cleanup relocations. It specifically adds handling for
R_BPF_64_NODYLD32andR_BPF_64_ABS32to process compiler-emitted exception cleanup tables.Quoted 4 lines
tools/lib/bpf/linker.c +0
-0[ ... ]
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2282
Quoted 9 lines
tools/lib/bpf/linker.c +5
-0@@ -2274,6 +2275,22 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; + }else if (sym_type ==R_BPF_64_NODYLD32 || + sym_type ==R_BPF_64_ABS32 ) { +__u32 *val; + +/* Two spellings of the one thing: LLVM [Severity: Low]
This isn't a bug, but does this violate the BPF subsystem multi-line comment style? The preferred kernel style is to have the opening /* on its own line rather than putting text on the same line as the marker.tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2289
Quoted 7 lines
tools/lib/bpf/linker.c +7
-0+* emits NODYLD32 for a .long against a +* section symbol, GNU as emits ABS32 +* (bpf_reloc_type_lookup() maps +* BFD_RELOC_32 to it), and the value +* they describe is the same. +*/ + val = dst_linked_sec->raw_data + dst_rel->r_offset;[Severity: High]
Could this lead to a NULL pointer dereference? Looking atlinker_sanity_check_elf_relos(),SHT_NOBITSsections are explicitly allowed as relocatable sections.For
SHT_NOBITSsections (like .bss),extend_sec()typically skips allocatingraw_data, leaving it as NULL. If we process a relocation against such a section here, we might compute val = NULL +dst_rel->r_offsetand then dereference it on the next line when handlingsec->dst_off.
Quoted 8 lines
[Severity: Medium]
Is it also possible to hit an unaligned memory access here?While executable sections enforce alignment checks in
linker_sanity_check_elf_relos(), there doesn't seem to be an equivalent alignment check for data section offsets. If an ELF provides an unalignedr_offset, the direct dereference of the unaligned val pointer might cause a crash on architectures with strict memory alignment requirements.
Okay, I will add a guard to check NULL pointer and unaligned memory access. If this is true, will return an error.
Quoted 8 lines
Quoted 8 lines
tools/lib/bpf/linker.c +4
-0+if (linker->swapped_endian) + *val =bswap_32 (bswap_32 (*val) + sec->dst_off); +else + *val += sec->dst_off; }else {pr_warn ("relocation against STT_SECTION in non-exec section is not supported!\n" );return -EINVAL ;
[email protected]
AI reviewRe: [PATCH bpf-next v2 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations
In reply to Yonghong SongThis looks like a bug fix for objects that carry compiler-emitted exception cleanup tables. Should this include:
Quoted 8 lines
An object that carries a compiler-emitted exception cleanup table cannot be linked today. The table's fields are byte offsets into a code section, materialised by a 32-bit relocation against that section's symbol with the offset itself as the implicit addend, and the linker rejects both halves of
that: the relocation type is not in the list it accepts, and a relocation against anSTT_SECTIONsymbol from a non-executable section is an outright error.
Quoted 7 lines
Both spellings of that relocation have to be taken. LLVM emits
R_BPF_64_NODYLD32for a .long against a section symbol; GNU as emitsR_BPF_64_ABS32, which is whatbpf_reloc_type_lookup()mapsBFD_RELOC_32to. They describe the same value, and the selftests are built with both compilers.Signed-off-by Yonghong Song <[email protected]>
This looks like a bug fix for objects that carry compiler-emitted exception cleanup tables. Should this include:
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerRe: [PATCH bpf-next v2 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations
Quoted 14 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 14 lines
An object that carries a compiler-emitted exception cleanup table cannot be linked today. The table's fields are byte offsets into a code section, materialised by a 32-bit relocation against that section's symbol with the offset itself as the implicit addend, and the linker rejects both halves of
that: the relocation type is not in the list it accepts, and a relocation against anSTT_SECTIONsymbol from a non-executable section is an outright error.Both spellings of that relocation have to be taken. LLVM emits
R_BPF_64_NODYLD32for a .long against a section symbol; GNU as emitsR_BPF_64_ABS32, which is whatbpf_reloc_type_lookup()mapsBFD_RELOC_32to. They describe the same value, and the selftests are built with both compilers.
Quoted 5 lines
Quoted 1 line
Signed-off-by Yonghong Song <[email protected]>This looks like a bug fix for objects that carry compiler-emitted exception cleanup tables. Should this include:
Fixes 9f0c317f6aa1 ("libbpf: Add support for new llvm bpf relocations")
No need for a Fixes tag.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerPATCH v2 18/20 selftests/bpf: Add an end-to-end .bpf_cleanup exception test
In reply to Yonghong SongC has no unwinding, so nothing here comes out of the frontend: the frames that own a resource are written as __naked inline assembly, which spells out by hand exactly what a frontend emits -- a call site bracketed by two labels, a landing pad unreachable in the compiler's CFG, and a .bpf_cleanup record tying them together. The assembler turns ".long <text label>" into the same R_BPF_64_NODYLD32 relocation the BPF AsmPrinter emits, so libbpf and the kernel see an object indistinguishable from a compiler-generated one.
Call chain: entry -> foo1 -> foo1v -> foo2 -> foo3. foo3 holds a non-preemptible section and throws inside it; foo2 holds an RCU read lock and has two call sites sharing one pad, one of them its own throw; foo1v is a void frame whose pad ends in a jump to a resume block placed after an unrelated block that ends in a plain exit; foo1 owns nothing and gets no record; entry is the boundary.
There are also some shapes the kernel refuses -- the ones with no correct answer, and the ones a pad running on the bpf_throw() walker's stack cannot express -- plus the return-value rejection that delivering an exception at the boundary of a program type which constrains its return value produces. The test skips rather than fails where the JIT cannot dispatch a landing pad at all.
--- .../selftests/bpf/exceptions_cleanup.h | 29 + .../bpf/prog_tests/exceptions_cleanup.c | 79 +++ .../selftests/bpf/progs/exceptions_cleanup.c | 152 +++++ .../bpf/progs/exceptions_cleanup_fail.c | 600 ++++++++++++++++++ 4 files changed, 860 insertions(+) create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
tools/testing/selftests/bpf/exceptions_cleanup.h new file +29 -0
@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#ifndef __EXCEPTIONS_CLEANUP_H__ +#define __EXCEPTIONS_CLEANUP_H__ + +#define THROW_COOKIE 0x100 + +/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */ +#define RAN_FOO3_PREEMPT 0x1 +#define RAN_FOO2_RCU 0x2 +#define RAN_FOO1V_PREEMPT 0x4 +#define RAN_FOO2_DROP 0x8 +#define RAN_BUMP 0x10 + +#define CLEANUP_REC (begin, end, landing_pad) \ + ".pushsection .bpf_cleanup,\"a\",@progbits;" \ + ".long " begin ";" \ + ".long " end ";" \ + ".long " landing_pad ";" \ + ".popsection;" + +/* Set a bit in @pads_ran. */ +#define PAD_RAN (bit) \ + "r1 = %[pads_ran] ll;" \ + "r2 = *(u64 *)(r1 + 0);" \ + "r2 |= " bit ";" \ + "*(u64 *)(r1 + 0) = r2;" + +#endif /* __EXCEPTIONS_CLEANUP_H__ */
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +79 -0
79 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_cleanup.c new file +152 -0
152 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +600 -0
600 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 18/20] selftests/bpf: Add an end-to-end .bpf_cleanup exception test
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] bpf: multi-line comments do not follow modern BPF style --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: multi-line comments do not follow modern BPF style
--
commit 8ad5e7ea23b9dcb55029206d5cb36453b231f9c8
Author: Yonghong Song <[email protected]>
selftests/bpf: Add an end-to-end .bpf_cleanup exception test
This commit adds an end-to-end test for .bpf_cleanup exception handling. It uses inline assembly to construct precise call-site brackets, landing pads, and cleanup records that mirror compiler output. It also tests shapes that the kernel should refuse, ensuring the JIT correctly handles or rejects different landing pad configurations.
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +0
-0
[ ... ]
Quoted 1 line
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +0
-0@@ -0,0 +1,79 @@
[ ... ]
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c line 72
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +7
-0+/* foo3 throws; every pad runs and the cookie is delivered at entry. */ +if (test__start_subtest ("throw_from_foo3" )) +run (skel,101 ,THROW_COOKIE ,PADS_FOO3_THREW ); + +/* foo3 returns 2 ^ 1 == 3, so foo2 throws from its own second region; +* foo3's frame is long gone, so its pad must not run. +*/
[Severity: Low]
Does this comment block follow the proper multi-line comment formatting for the BPF subsystem?
The BPF subsystem strict guidelines require multi-line comments to have the opening /* on its own line:
/*
* foo3 returns 2 ^ 1 == 3, so foo2 throws from its own second region;
* foo3's frame is long gone, so its pad must not run.
*/
Quoted 13 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +7
-0+if (test__start_subtest ("throw_from_foo2" )) +run (skel,2 ,THROW_COOKIE ,PADS_FOO2_THREW ); + +exceptions_cleanup__destroy (skel); + +RUN_TESTS (exceptions_cleanup_fail); +}tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +0
-0
[ ... ]
Quoted 1 line
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +0
-0@@ -0,0 +1,600 @@
[ ... ]
Quoted 16 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +16
-0+SEC("socket") +__failure __msg("reaches an indirect jump") +__naked void gotox_in_pad (void ) +{ + asmvolatile ( +".pushsection .jumptables,\"\",@progbits;" +"jt0_%=:" +".quad l0_%= - socket;" +".quad l1_%= - socket;" +".size jt0_%=, 16;" +".global jt0_%=;" +".popsection;" + +"1:" "call gotox_thrower;" /* cleanup region */ +"2:" + "r0 = 0;"
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c line 586
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +10
-0+"exit;" +"3:" /* landing pad */ +"r1 = jt0_%= ll;" +"r1 += 8;" +"r2 = *(u64 *)(r1 + 0);" +/* gotox r2. Spelled as a raw insn on purpose: the "gotox" mnemonic +* only reached the LLVM assembler in llvm 22, and BPF_RAW_INSN() +* needs <linux/bpf.h>, which this file cannot have -- vmlinux.h +* already defines the uapi enums. +*/
[Severity: Low]
Does this multi-line comment format conform to the BPF style guide?
As with the previous comment in exceptions_cleanup.c, the BPF subsystem mandates that the opening /* resides on its own line for multi-line comments.
Quoted 14 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +14
-0+".8byte 0x20d;" +"l0_%=:" +"call bpf_unwind_resume;" +"exit;" +"l1_%=:" +"call bpf_unwind_resume;" +"exit;" +CLEANUP_REC ("1b" ,"2b" ,"3b" ) + ::: __clobber_all); +} + +#endif /* x86 || arm64 */ + +char _license[] SEC("license") = "GPL";
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
[email protected]
AI reviewRe: [PATCH bpf-next v2 18/20] selftests/bpf: Add an end-to-end .bpf_cleanup exception test
In reply to Yonghong SongIs bpf_unwind_resume() declared anywhere? It is called from C here but bpf_experimental.h declares only the other kfuncs this anchor exists for:
Quoted 14 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +8
-0@@ -0,0 +1,600 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_experimental.h" +#include "bpf_misc.h" +#include "../test_kmods/bpf_testmod_kfunc.h" +#include "exceptions_cleanup.h"
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c line 18
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +10
-0+ +__u64 input =0 ; + +static __used __noinline void __kfunc_btf_anchor(void) +{ +bpf_throw (0 ); +bpf_preempt_disable (); +bpf_preempt_enable (); +bpf_unwind_resume (); +}
Is bpf_unwind_resume() declared anywhere? It is called from C here but bpf_experimental.h declares only the other kfuncs this anchor exists for:
tools/testing/selftests/bpf/bpf_experimental.h declares bpf_throw():
extern void bpf_throw(u64 cookie) __ksym;
and the preempt pair:
void bpf_preempt_disable(void) __weak __ksym;
void bpf_preempt_enable(void) __weak __ksym;but bpf_unwind_resume() does not appear in any header. vmlinux.h contains type definitions only, not kfunc prototypes.
BPF objects are built with BPF_CFLAGS = -g -Wall -Werror -std=gnu11 (tools/testing/selftests/bpf/Makefile). -Wimplicit-function-declaration is part of -Wall, and in C99+ mode modern Clang promotes it to an error, which with -Werror becomes fatal:
progs/exceptions_cleanup_fail.c:17:2: error: call to undeclared
function 'bpf_unwind_resume'; ISO C99 and later do not support implicit function declarations
This would break the build of the whole BPF selftests tree. The same C-level call appears in the three other new prog files added by this series (progs/exceptions_cleanup.c, and in follow-ups progs/exceptions_cleanup_shapes.c and progs/exceptions_cleanup_light.c), so all of them would fail to compile.
The anchor cannot simply drop the call either: the C call is what makes Clang emit the BTF extern FUNC for bpf_unwind_resume, which libbpf needs to resolve the inline-asm "call bpf_unwind_resume;" relocations. The fix would be to add the declaration:
extern void bpf_unwind_resume(void) __ksym;
in bpf_experimental.h alongside bpf_throw(). For comparison, progs/exceptions_cleanup_shapes.c (commit 198e7e8d6d72) does declare its sibling explicitly:
extern void _Unwind_Resume(void) __ksym;
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerRe: [PATCH bpf-next v2 18/20] selftests/bpf: Add an end-to-end .bpf_cleanup exception test
Quoted 58 lines · On 9/17/26 10:58 PM, [email protected] wrote:
Quoted 14 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +8
-0@@ -0,0 +1,600 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_experimental.h" +#include "bpf_misc.h" +#include "../test_kmods/bpf_testmod_kfunc.h" +#include "exceptions_cleanup.h" tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c line 18
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +10
-0+ +__u64 input =0 ; + +static __used __noinline void __kfunc_btf_anchor(void) +{ +bpf_throw (0 ); +bpf_preempt_disable (); +bpf_preempt_enable (); +bpf_unwind_resume (); +}Is
bpf_unwind_resume()declared anywhere? It is called from C here butbpf_experimental.hdeclares only the other kfuncs this anchor exists for:
tools/testing/selftests/bpf/bpf_experimental.hdeclaresbpf_throw():extern void bpf_throw(u64 cookie) __ksym;and the preempt pair:
void bpf_preempt_disable(void) __weak __ksym; void bpf_preempt_enable(void) __weak __ksym;but
bpf_unwind_resume()does not appear in any header.vmlinux.hcontains type definitions only, not kfunc prototypes.BPF objects are built with
BPF_CFLAGS= -g -Wall -Werror -std=gnu11 (tools/testing/selftests/bpf/Makefile). -Wimplicit-function-declaration is part of -Wall, and in C99+ mode modern Clang promotes it to an error, which with -Werror becomes fatal:
progs/exceptions_cleanup_fail.c:17:2: error: call to undeclared
function 'bpf_unwind_resume'; ISO C99 and later do not support implicit function declarationsThis would break the build of the whole BPF selftests tree. The same C-level call appears in the three other new prog files added by this series (
progs/exceptions_cleanup.c, and in follow-upsprogs/exceptions_cleanup_shapes.candprogs/exceptions_cleanup_light.c), so all of them would fail to compile.The anchor cannot simply drop the call either: the C call is what makes Clang emit the BTF extern FUNC for
bpf_unwind_resume, which libbpf needs to resolve the inline-asm "callbpf_unwind_resume;" relocations. The fix would be to add the declaration:
Quoted 7 lines
extern void bpf_unwind_resume(void) __ksym;in
bpf_experimental.halongsidebpf_throw(). For comparison,progs/exceptions_cleanup_shapes.c(commit198e7e8d6d72) does declare its sibling explicitly:extern void _Unwind_Resume(void) __ksym;
I didn't find anything wrong with this patch set and with my normal environment, the test works fine.
Quoted 6 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerPATCH v2 19/20 selftests/bpf: Cover the exception cleanup shapes the chain does not reach
In reply to Yonghong SongThe end-to-end test walks one call chain with a pad in most of its frames. This adds the shapes that chain does not reach: a callee called from both a covered and an uncovered site, a pad that reads its frame's callee-saved registers, a tail-call-reachable callee, a tail call that is taken, an extension standing in for a covered call, a pad in the main program's own frame, a record covering bpf_throw() itself, a record covering only a nounwind call, a throwing subprogram named by a BPF_PSEUDO_FUNC, a region ending on a 16-byte instruction, a pad that reloads from and writes to its own frame, a pad two frames up, a pad that calls a subprogram which tail calls, an extension carrying a table of its own, and a pad terminated by _Unwind_Resume rather than bpf_unwind_resume.
--- .../selftests/bpf/exceptions_cleanup.h | 20 + .../bpf/prog_tests/exceptions_cleanup.c | 314 +++++++ .../bpf/progs/exceptions_cleanup_ext_table.c | 48 + .../bpf/progs/exceptions_cleanup_freplace.c | 17 + .../progs/exceptions_cleanup_pad_freplace.c | 17 + .../bpf/progs/exceptions_cleanup_shapes.c | 863 ++++++++++++++++++ 6 files changed, 1279 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
tools/testing/selftests/bpf/exceptions_cleanup.h +20 -0
@@ -4,6 +4,7 @@#define __EXCEPTIONS_CLEANUP_H__#define THROW_COOKIE 0x100 +#define INNER_COOKIE 0x200/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */ #define RAN_FOO3_PREEMPT 0x1
@@ -12,6 +13,25 @@#define RAN_FOO2_DROP 0x8#define RAN_BUMP 0x10 +/* progs/exceptions_cleanup_shapes.c: one bit per shape, numbered its own way. */ +#define RAN_SWEEP 0x1 +#define RAN_SHARED 0x2 +#define RAN_REGS 0x4 +#define RAN_TAIL_CALL 0x8 +#define RAN_MAIN_PAD 0x10 +#define RAN_TC_TAKEN 0x20 +#define RAN_FREPLACE 0x40 +#define RAN_ADDR_TAKEN 0x80 +#define RAN_NO_SUBPROG 0x100 +#define RAN_PAD_CALLS 0x200 +#define RAN_PAD_FIRST 0x400 +#define RAN_WIDE_REC 0x800 +#define RAN_PAD_STACK 0x1000 +#define RAN_DEEP_PAD 0x2000 +#define RAN_NOUNWIND_REC 0x4000 +#define RAN_RESUME_ALIAS 0x8000 +#define RAN_PAD_TAIL_CALL 0x10000 +#define CLEANUP_REC (begin, end, landing_pad) \ ".pushsection .bpf_cleanup,\"a\",@progbits;" \ ".long " begin ";" \
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +314 -0
331 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c new file +48 -0
48 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "exceptions_cleanup.h" + +/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */ +int btf_int_anchor; + +SEC("freplace/fr_callee") +__u64 new_fr_callee (__u64 x) +{ +bpf_throw (THROW_COOKIE ); +return 0 ; +} + +char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "exceptions_cleanup.h" + +/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */ +int btf_int_anchor; + +SEC("freplace/pad_callee") +__u64 new_pad_callee (__u64 x) +{ +bpf_throw (INNER_COOKIE ); +return 0 ; +} + +char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +863 -0
863 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v2 19/20] selftests/bpf: Cover the exception cleanup shapes the chain does not reach
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] bpf: multi-line comment style violation --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: multi-line comment style violation
--
commit 59a07b8ba8f5c3976a5da9bea3b5d6359874813a
Author: Yonghong Song <[email protected]>
selftests/bpf: Cover the exception cleanup shapes the chain does not reach
This commit adds end-to-end exception cleanup tests for various edge cases and shapes that the primary call chain does not reach. It tests things like tail calls, nounwind calls, subprogram pads, and landing pads that interact with the stack.
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +0
-0
[ ... ]
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +2
-0@@ -35,6 +39,314 @@ static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, +static void test_shapes (void ) +{
[ ... ]
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 215
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +3
-0+/* The same call site either way: the subprogram's throw unwinds into +* this frame and runs its pad, an extension's stops at its own boundary. +*/
[Severity: Low]
This isn't a bug, but does this comment follow the BPF subsystem coding style? Multi-line comments should have their opening /* on its own line before the text begins.
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 224
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +9
-0+if (test__start_subtest ("freplace_subprog_throws" )) +run_shape (skel, skel->progs.entry_freplace,7 ,THROW_COOKIE , +RAN_FREPLACE ); +if (test__start_subtest ("freplace_extension_throws" )) +test_freplace (skel); + +/* A tail call that is taken: the walk ends at the target, so the cookie +* comes back from there and this frame's pad does not run. +*/
[Severity: Low]
This isn't a bug, but is this the preferred multi-line comment format? The BPF subsystem coding style mandates that the opening /* should be on its own line.
Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +2
-0+if (test__start_subtest ("tail_call_taken" )) { +int key =0 , prog_fd =bpf_program__fd (skel->progs.tc_target);
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +0
-0
[ ... ]
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 5
Quoted 6 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +5
-0@@ -0,0 +1,863 @@ +static __used __noinline __u64 tc_thrower(void *ctx) +{ +/* Never taken; its presence is what makes this frame, whose spill the +* walker reads, tail-call-reachable. +*/
[Severity: Low]
This isn't a bug, but could this comment formatting be adjusted? Multi-line comments in the BPF subsystem should place their opening /* on a separate line.
Quoted 1 line
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +1
-0+bpf_tail_call_static (ctx, &jmp_table,0 );
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19
[email protected]
AI reviewRe: [PATCH bpf-next v2 19/20] selftests/bpf: Cover the exception cleanup shapes the chain does not reach
In reply to Yonghong SongDoes this comment accurately describe what the addr_taken shape tests?
Quoted 12 lines
tools/testing/selftests/bpf/exceptions_cleanup.h +1
-0@@ -4,6 +4,7 @@#define __EXCEPTIONS_CLEANUP_H__#define THROW_COOKIE 0x100 +#define INNER_COOKIE 0x200/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */ #define RAN_FOO3_PREEMPT 0x1
[ ... ]
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +0
-0
[ ... ]
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +7
-0+static void test_shapes (void ) +{ +struct exceptions_cleanup_shapes *skel; + + skel =exceptions_cleanup_shapes__open_and_load (); +if (!ASSERT_OK_PTR (skel,"shapes open_and_load" )) +return ;
[ ... ]
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 242
Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +8
-0+/* A throwing subprog named by a BPF_PSEUDO_FUNC no helper is handed: the +* callback check has to look at the bpf_loop(), not at the ld_imm64. +*/ +if (test__start_subtest ("addr_taken_no_throw" )) +run_shape (skel, skel->progs.entry_addr_taken,1 ,2 ,0 ); +if (test__start_subtest ("addr_taken_throw" )) +run_shape (skel, skel->progs.entry_addr_taken,101 ,THROW_COOKIE , +RAN_ADDR_TAKEN );
Does this comment accurately describe what the addr_taken shape tests?
Looking at addr_taken_callee() in progs/exceptions_cleanup_shapes.c, the function does hand cb_thrower to bpf_loop(). The ld_imm64 and the bpf_loop() call are on the same path, after bpf_throw(). Since bpf_throw() is not declared noreturn, LLVM keeps that tail, and what actually makes the program load is that the verifier never reaches the bpf_loop() at all (it is dead after the throw).
So push_callback_call() -> bpf_cleanup_check_callback() never fires.
The shape exercised is 'a BPF_PSEUDO_FUNC whose helper call the verifier never reaches', not 'a BPF_PSEUDO_FUNC no helper is handed'. Could the comment be rephrased to reflect what the verifier actually sees?
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +0
-0
[ ... ]
Quoted 29 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +29
-0+/* +* 2. A callee called from both a covered and an uncovered site: the pad is +* recorded on the call site, not on the callee. The lock sits between the two +* calls because the frame really would leak it if the uncovered call unwound. +*/ +static __used __noinline __u64 shared_callee(__u64 x) +{ +if (x >100 ) +bpf_throw (THROW_COOKIE ); +return x +1 ; +} + +static __used __naked __noinline __u64 shared_frame(void) +{ + asmvolatile ( +"r1 = %[input] ll;" +"r6 = *(u64 *)(r1 + 0);" +"r1 = 0;" +"call shared_callee;" +"call bpf_rcu_read_lock;" +"r1 = r6;" +"1:" "call shared_callee;" /* cleanup region */ +"2:" +"r6 = r0;" +"call bpf_rcu_read_unlock;" +"r0 = r6;" +"exit;" +"3:" /* landing pad */ + "r7 = r0;"
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 118
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +10
-0+"call bpf_rcu_read_unlock;" + PAD_RAN("%[ran]") +"r1 = r7;" +"call bpf_unwind_resume;" +"exit;" +CLEANUP_REC ("1b" ,"2b" ,"3b" ) + : + : [ran]"i"(RAN_SHARED), __imm_addr(input), __imm_addr(pads_ran) + : __clobber_all); +}
Does the uncovered call site verify that the pad wouldn't be wrongly dispatched if an exception occurred there?
The uncovered site is reached with the constant 0 in r1, and shared_callee only throws when x > 100, so the verifier proves that call cannot throw and no unwind ever leaves the uncovered site - at verification time or at run time.
The structural half of the property (only the second call's return address falls inside the record's native range) is exercised, but the behavioral half is not: a kernel that wrongly matched the uncovered site's return address against the record and ran the pad would go unnoticed, because that site never unwinds.
[ ... ]
Quoted 44 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +44
-0+/* +* 6. A tail call that is really taken: the target is a program in its own +* right, so the walk ends there and this frame's pad does not run. The callee +* can also throw on a path never taken, which keeps the pad out of the sweep. +*/ +struct { + __uint(type, BPF_MAP_TYPE_PROG_ARRAY); + __uint(max_entries, 1); + __uint(key_size, sizeof(__u32)); + __uint(value_size, sizeof(__u32)); +} taken_table SEC(".maps"); + +SEC("syscall") +int tc_target (void *ctx) +{ +bpf_throw (THROW_COOKIE ); +return 0 ; +} + +static __used __noinline __u64 tc_taken_callee(void *ctx, __u64 x) +{ +/* Never true at run time; the verifier cannot know that, and its +* unwind out of here is what keeps the caller's pad alive. +*/ +if (x ==7 ) +bpf_throw (THROW_COOKIE ); +bpf_tail_call_static (ctx, &taken_table,0 ); +return 0 ; +} + +SEC("syscall") +__naked int entry_tail_taken (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = r1;" /* the context, straight from entry */ +"r1 = %[input] ll;" +"r2 = *(u64 *)(r1 + 0);" +"if r2 < 101 goto 8f;" +"r1 = *(u64 *)(r10 - 8);" +"1:" "call tc_taken_callee;" /* cleanup region */ +"2:" +"exit;" /* the cookie, delivered at tc_target */ +"8:" + "r0 = 0;"
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 344
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +10
-0+"exit;" +"3:" /* landing pad: must not run */ + PAD_RAN("%[ran]") +"call bpf_unwind_resume;" +"exit;" +CLEANUP_REC ("1b" ,"2b" ,"3b" ) + : + : [ran]"i"(RAN_TC_TAKEN), __imm_addr(input), __imm_addr(pads_ran) + : __clobber_all); +}
Can the tail_call_taken subtest fail if the kernel wrongly continues the walk past the tail-call boundary?
entry_tail_taken narrows the argument before the covered call:
"r2 = *(u64 *)(r1 + 0);" /* r2 = input */
"if r2 < 101 goto 8f;" /* fallthrough => r2 in [101, U64_MAX] */
tc_taken_callee is a static subprog, so check_func_call() copies the caller's r1-r5 verbatim into the callee frame. x therefore arrives with umin_value == 101, and is_branch_taken() resolves 'if (x == 7)' to never-taken, so the verifier never explores the bpf_throw(). Nothing then unwinds into entry_tail_taken during verification, the landing pad at label 3 is never marked seen, and opt_remove_dead_code() removes it.
fixups.c then clears the annotation for any call site whose cleanup_pad falls inside the removed window:
if (env->cleanup_info_cnt) {
for (i = 0; i < env->insn_aux_data_len; i++) {
u32 pad = aux_data[i].cleanup_pad;
if (pad > off + cnt) aux_data[i].cleanup_pad = pad - cnt;
else if (pad > off) aux_data[i].cleanup_pad = 0;
}
}so no native range is emitted for the region at all. The program still loads (mark_subprog_might_throw() is a static analysis, so bpf_check_cfg() still sees the pad edge and check_cleanup_info() runs before the sweep), and pads_ran stays 0 unconditionally. A kernel that wrongly continued bpf_stack_walker() past the tail-call boundary into entry_tail_taken's frame would find no record to match and would still leave pads_ran == 0, so the subtest passes on a correct and on a broken kernel alike.
The in-code comment is also wrong on both clauses: the verifier CAN know x != 7 here, and consequently the unwind out of tc_taken_callee is NOT what keeps the caller's pad alive - nothing does.
Contrast with shape 7 (entry_freplace, lines 862-878), the analogous 'walk ends in the callee's frame' shape, which deliberately passes the unnarrowed input value to fr_callee() with no 'if r1 < 101' guard, keeping fr_callee's identical 'if (x == 7) bpf_throw()' live. Dropping the 'if r2 < 101 goto 8f' guard from entry_tail_taken (or otherwise passing a value the verifier cannot exclude 7 from) would make the pad survive and give the subtest something to fail on.
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerRe: [PATCH bpf-next v2 19/20] selftests/bpf: Cover the exception cleanup shapes the chain does not reach
Quoted 46 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 12 lines
tools/testing/selftests/bpf/exceptions_cleanup.h +1
-0@@ -4,6 +4,7 @@#define __EXCEPTIONS_CLEANUP_H__#define THROW_COOKIE 0x100 +#define INNER_COOKIE 0x200/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */ #define RAN_FOO3_PREEMPT 0x1[ ... ]
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +0
-0[ ... ]
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +7
-0+static void test_shapes (void ) +{ +struct exceptions_cleanup_shapes *skel; + + skel =exceptions_cleanup_shapes__open_and_load (); +if (!ASSERT_OK_PTR (skel,"shapes open_and_load" )) +return ;[ ... ]
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 242
Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +8
-0+/* A throwing subprog named by a BPF_PSEUDO_FUNC no helper is handed: the +* callback check has to look at the bpf_loop(), not at the ld_imm64. +*/ +if (test__start_subtest ("addr_taken_no_throw" )) +run_shape (skel, skel->progs.entry_addr_taken,1 ,2 ,0 ); +if (test__start_subtest ("addr_taken_throw" )) +run_shape (skel, skel->progs.entry_addr_taken,101 ,THROW_COOKIE , +RAN_ADDR_TAKEN );Does this comment accurately describe what the
addr_takenshape tests?Looking at
addr_taken_callee()inprogs/exceptions_cleanup_shapes.c, the function does handcb_throwertobpf_loop(). Theld_imm64and thebpf_loop()call are on the same path, afterbpf_throw(). Sincebpf_throw()is not declared noreturn, LLVM keeps that tail, and what actually makes the program load is that the verifier never reaches thebpf_loop()at all (it is dead after the throw).
Quoted 5 lines
So
push_callback_call()->bpf_cleanup_check_callback()never fires.The shape exercised is 'a
BPF_PSEUDO_FUNCwhose helper call the verifier never reaches', not 'aBPF_PSEUDO_FUNCno helper is handed'. Could the comment be rephrased to reflect what the verifier actually sees?
Okay, will change comments.
Quoted 57 lines
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +0
-0[ ... ]
Quoted 29 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +29
-0+/* +* 2. A callee called from both a covered and an uncovered site: the pad is +* recorded on the call site, not on the callee. The lock sits between the two +* calls because the frame really would leak it if the uncovered call unwound. +*/ +static __used __noinline __u64 shared_callee(__u64 x) +{ +if (x >100 ) +bpf_throw (THROW_COOKIE ); +return x +1 ; +} + +static __used __naked __noinline __u64 shared_frame(void) +{ + asmvolatile ( +"r1 = %[input] ll;" +"r6 = *(u64 *)(r1 + 0);" +"r1 = 0;" +"call shared_callee;" +"call bpf_rcu_read_lock;" +"r1 = r6;" +"1:" "call shared_callee;" /* cleanup region */ +"2:" +"r6 = r0;" +"call bpf_rcu_read_unlock;" +"r0 = r6;" +"exit;" +"3:" /* landing pad */ + "r7 = r0;"tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 118
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +10
-0+"call bpf_rcu_read_unlock;" + PAD_RAN("%[ran]") +"r1 = r7;" +"call bpf_unwind_resume;" +"exit;" +CLEANUP_REC ("1b" ,"2b" ,"3b" ) + : + : [ran]"i"(RAN_SHARED), __imm_addr(input), __imm_addr(pads_ran) + : __clobber_all); +}Does the uncovered call site verify that the pad wouldn't be wrongly dispatched if an exception occurred there?
The uncovered site is reached with the constant 0 in r1, and
shared_calleeonly throws when x > 100, so the verifier proves that call cannot throw and no unwind ever leaves the uncovered site - at verification time or at run time.
Quoted 5 lines
The structural half of the property (only the second call's return address falls inside the record's native range) is exercised, but the behavioral half is not: a kernel that wrongly matched the uncovered site's return address against the record and ran the pad would go unnoticed, because that site never unwinds.
The above analysis exactly described the code so the above 'wrongly dispatched if an exception occurred there' does not happen. I guess this tries to capture incorrect kernel implementation.
Quoted 51 lines
[ ... ]
Quoted 48 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +48
-0+/* +* 6. A tail call that is really taken: the target is a program in its own +* right, so the walk ends there and this frame's pad does not run. The callee +* can also throw on a path never taken, which keeps the pad out of the sweep. +*/ +struct { + __uint(type, BPF_MAP_TYPE_PROG_ARRAY); + __uint(max_entries, 1); + __uint(key_size, sizeof(__u32)); + __uint(value_size, sizeof(__u32)); +} taken_table SEC(".maps"); + +SEC("syscall") +int tc_target (void *ctx) +{ +bpf_throw (THROW_COOKIE ); +return 0 ; +} + +static __used __noinline __u64 tc_taken_callee(void *ctx, __u64 x) +{ +/* Never true at run time; the verifier cannot know that, and its +* unwind out of here is what keeps the caller's pad alive. +*/ +if (x ==7 ) +bpf_throw (THROW_COOKIE ); +bpf_tail_call_static (ctx, &taken_table,0 ); +return 0 ; +} + +SEC("syscall") +__naked int entry_tail_taken (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = r1;" /* the context, straight from entry */ +"r1 = %[input] ll;" +"r2 = *(u64 *)(r1 + 0);" +"if r2 < 101 goto 8f;" +"r1 = *(u64 *)(r10 - 8);" +"1:" "call tc_taken_callee;" /* cleanup region */ +"2:" +"exit;" /* the cookie, delivered at tc_target */ +"8:" +"r0 = 0;" +"exit;" +"3:" /* landing pad: must not run */ + PAD_RAN("%[ran]") +"call bpf_unwind_resume;"
Quoted 8 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 344
Quoted 6 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +6
-0+"exit;" +CLEANUP_REC ("1b" ,"2b" ,"3b" ) + : + : [ran]"i"(RAN_TC_TAKEN), __imm_addr(input), __imm_addr(pads_ran) + : __clobber_all); +}Can the
tail_call_takensubtest fail if the kernel wrongly continues the walk past the tail-call boundary?
This is a test which fixed some early regression tests. Yes, if kernel implementation is wrong, tail_call_taken test may fail.
Quoted 32 lines
entry_tail_takennarrows the argument before the covered call:"r2 = *(u64 *)(r1 + 0);" /* r2 = input */ "if r2 < 101 goto 8f;" /* fallthrough => r2 in [101, U64_MAX] */
tc_taken_calleeis a static subprog, socheck_func_call()copies the caller's r1-r5 verbatim into the callee frame. x therefore arrives withumin_value== 101, andis_branch_taken()resolves 'if (x == 7)' to never-taken, so the verifier never explores thebpf_throw(). Nothing then unwinds intoentry_tail_takenduring verification, the landing pad at label 3 is never marked seen, andopt_remove_dead_code()removes it.
fixups.cthen clears the annotation for any call site whosecleanup_padfalls inside the removed window:if (env->cleanup_info_cnt) { for (i = 0; i < env->insn_aux_data_len; i++) { u32 pad = aux_data[i].cleanup_pad; if (pad > off + cnt) aux_data[i].cleanup_pad = pad - cnt; else if (pad > off) aux_data[i].cleanup_pad = 0; } }so no native range is emitted for the region at all. The program still loads (
mark_subprog_might_throw()is a static analysis, sobpf_check_cfg()still sees the pad edge andcheck_cleanup_info()runs before the sweep), andpads_ranstays 0 unconditionally. A kernel that wrongly continuedbpf_stack_walker()past the tail-call boundary into entry_tail_taken's frame would find no record to match and would still leavepads_ran== 0, so the subtest passes on a correct and on a broken kernel alike.
Quoted 3 lines
The in-code comment is also wrong on both clauses: the verifier CAN know x != 7 here, and consequently the unwind out of
tc_taken_calleeis NOT what keeps the caller's pad alive - nothing does.
The comment is:
/* Never true at run time; the verifier cannot know that, and its
* unwind out of here is what keeps the caller's pad alive.
*/
if (x == 7)
bpf_throw(THROW_COOKIE);Yes, the 'x' will be a known value at run time. Comments need update.
Quoted 17 lines
Contrast with shape 7 (
entry_freplace, lines 862-878), the analogous 'walk ends in the callee's frame' shape, which deliberately passes the unnarrowed input value tofr_callee()with no 'if r1 < 101' guard, keeping fr_callee's identical 'if (x == 7)bpf_throw()' live. Dropping the 'if r2 < 101 goto 8f' guard fromentry_tail_taken(or otherwise passing a value the verifier cannot exclude 7 from) would make the pad survive and give the subtest something to fail on.[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711
Yonghong Song
reviewerPATCH v2 20/20 selftests/bpf: Load an exception cleanup program from a light skeleton
In reply to Yonghong SongFor a light skeleton, libbpf hands the records to bpf_gen__prog_load(), which writes them into a blob and emits a loader program that issues BPF_PROG_LOAD from inside the kernel. Nothing about that is shared with the ordinary path: the attr is built field by field, and the kfunc names are resolved by the loader program when it runs.
exceptions_cleanup_light.c is the smallest program that can tell whether a table survives that trip: one record covering the bpf_throw() call itself, one pad that sets a bit and resumes. If the table arrives, the pad runs and the cookie comes back; if it does not, the program does not load at all.
--- tools/testing/selftests/bpf/Makefile | 2 +- .../selftests/bpf/exceptions_cleanup.h | 3 ++ .../bpf/prog_tests/exceptions_cleanup.c | 28 +++++++++++++ .../bpf/progs/exceptions_cleanup_light.c | 39 +++++++++++++++++++ 4 files changed, 71 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c
tools/testing/selftests/bpf/Makefile +1 -1
@@ -523,7 +523,7 @@ LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ LSKELS := fexit_sleep.c trace_printk.c trace_vprintk.c map_ptr_kern.c \ core_kern.c core_kern_overflow.c test_ringbuf.c \ test_ringbuf_n.c test_ringbuf_map_key.c test_ringbuf_write.c \ - test_ringbuf_overwrite.c + test_ringbuf_overwrite.c exceptions_cleanup_light.c LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
tools/testing/selftests/bpf/exceptions_cleanup.h +3 -0
@@ -32,6 +32,9 @@#define RAN_RESUME_ALIAS 0x8000#define RAN_PAD_TAIL_CALL 0x10000 +/* progs/exceptions_cleanup_light.c: the one pad it has. */ +#define RAN_LIGHT 0x1 +#define CLEANUP_REC (begin, end, landing_pad) \ ".pushsection .bpf_cleanup,\"a\",@progbits;" \ ".long " begin ";" \
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +28 -0
@@ -8,6 +8,7 @@#include "exceptions_cleanup_freplace.skel.h" #include "exceptions_cleanup_pad_freplace.skel.h" #include "exceptions_cleanup_ext_table.skel.h" +#include "exceptions_cleanup_light.lskel.h" /* foo3 threw: every frame that has a pad ran it. */ #define PADS_FOO3_THREW \
@@ -170,6 +171,30 @@ static void test_ext_table(struct exceptions_cleanup_shapes *skel)exceptions_cleanup_ext_table__destroy (fr); } +static void test_light_skeleton (void ) +{ +struct exceptions_cleanup_light_lskel *skel; +__u64 ctx =0 ; +int err; + +LIBBPF_OPTS (bpf_test_run_opts, topts, + .ctx_in = &ctx, + .ctx_size_in = sizeof(ctx), + ); + + skel =exceptions_cleanup_light_lskel__open_and_load (); +if (!ASSERT_OK_PTR (skel,"light open_and_load" )) +return ; + + err =bpf_prog_test_run_opts (skel->progs.entry_light.prog_fd, &topts); +if (!ASSERT_OK (err,"run" )) + goto out; +ASSERT_EQ (topts.retval,THROW_COOKIE ,"retval" ); +ASSERT_EQ (skel->bss->pads_ran,RAN_LIGHT ,"pads_ran" ); +out: +exceptions_cleanup_light_lskel__destroy (skel); +} +static void test_shapes (void ) {struct exceptions_cleanup_shapes *skel;
@@ -387,6 +412,9 @@ void test_exceptions_cleanup(void)exceptions_cleanup__destroy (skel); +if (test__start_subtest ("light_skeleton" )) +test_light_skeleton (); +test_shapes ();RUN_TESTS (exceptions_cleanup_fail);
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +39 -0
@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" +#include "exceptions_cleanup.h" + +static __used __noinline void __kfunc_btf_anchor(void) +{ +bpf_throw (0 ); +bpf_preempt_disable (); +bpf_preempt_enable (); +bpf_unwind_resume (); +} + +__u64 pads_ran =0 ; + +SEC("syscall") +__naked int entry_light (void ) +{ + asmvolatile ( +"call bpf_preempt_disable;" +"r1 = %[cookie];" +"1:" "call bpf_throw;" /* cleanup region */ +"2:" +"exit;" +"3:" /* landing pad */ +"call bpf_preempt_enable;" + PAD_RAN("%[ran]") +"call bpf_unwind_resume;" +"exit;" +CLEANUP_REC ("1b" ,"2b" ,"3b" ) + : + : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_LIGHT), +__imm_addr (pads_ran) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.53.0-Meta