bpf: Run exception cleanup landing pads when bpf_throw() unwinds
Yonghong Song
reviewercover bpf: Run exception cleanup landing pads when bpf_throw() unwinds
bpf_throw() walks the BPF call stack to the exception boundary and discards every frame in between. A frame that owns something -- an RCU read lock, a preemption-disabled section, a referenced kptr -- never gets to give it back, so the verifier refuses to let such a frame throw at all. That is the whole reason a Rust program cannot use bpf_throw() as its panic path today: Rust's Drop glue *is* that give-back, and there is nowhere to run it.
LLVM 23 added the compiler half ([1]). A Rust function that owns a value across a call that can unwind
fn foo() {
let _guard = RcuReadGuard::new(); /* bpf_rcu_read_lock() */
may_throw(); /* extern "C-unwind" */
} /* Drop: rcu_read_unlock */lowers to an invoke with a cleanup landing pad holding the Drop call, and the BPF backend writes one record per invoke region into a .bpf_cleanup
section: a flat table of 12-byte (begin, end, landing_pad) triples, each field a byte offset into the code section. The rule is "if a frame unwinds with its return address in [begin, end), run landing_pad before discarding it". A pad ends with a call to _Unwind_Resume(), which the kernel provides as the bpf_unwind_resume() kfunc.
This series is the kernel half: take that table at BPF_PROG_LOAD, teach the verifier that a covered call can also go to its landing pad, and have bpf_throw() run the pads as it walks.
C has no unwinding, so the selftests spell out by hand what a frontend emits -- a call site bracketed by two labels, a landing pad, and a record tying them together. The frame above, written that way:
"call bpf_rcu_read_lock;"
"1:" "call foo3;" /* cleanup region */
"2:"
... normal path, ends in bpf_rcu_read_unlock ...
"6:" /* landing pad */
"call bpf_rcu_read_unlock;"
"call bpf_unwind_resume;" CLEANUP_REC("1b", "2b", "6b")
Today that program does not load: the throw inside foo3 is reported as "bpf_throw cannot be used inside bpf_rcu_read_lock-ed region", because as far as the verifier is concerned nothing will ever unlock. With the series the verifier walks the unwind the same way the run time will -- into the pad, which unlocks, then on to the next frame -- and the program both loads and releases the lock when it throws.
Design
======
A pad is run, not lowered. bpf_throw() already walks the frames with arch_bpf_stack_walk(); it now looks each frame's return address up in that (sub)program's table and calls the pad as a subroutine of the walker, with the unwinding frame's frame pointer and its callee-saved registers restored from the spill its callee's prologue left. The pad therefore sees its own frame but runs on the walker's stack, far below it, so nothing it calls can disturb the frame it is cleaning up after. The JIT turns its bpf_unwind_resume() into the way back to the walker.
The verifier walks the same thing, step for step, so the resource rules are unchanged: whatever a pad releases is released in the verifier state too, and check_resource_leak() simply moves from "a throw was seen" to the end of the walk.
1-2 uapi: cleanup_info in BPF_PROG_LOAD, struct bpf_cleanup_info,
and the bpf_unwind_resume() kfunc
3-9 verifier: mark the covered call sites, give them an edge to the
pad, walk the unwind, and refuse the shapes that cannot be dispatched
10 bpf_throw(): dispatch pads while walking 11-12 x86-64 and arm64 JITs 13-17 libbpf: collect .bpf_cleanup, pass it to the kernel, resolve
_Unwind_Resume, carry it through the light skeleton and the static linker
18-20 selftests
Limitations
===========
- Cleanup pads only. A catch pad -- one that ends in a plain exit rather than a resume, which is what Rust's catch_unwind would need -- is
refused: the walker calls a pad as a subroutine and cannot hand a frame back its own execution. LLVM refuses type-specific catches and filters on its side as well.
- A JIT that can dispatch pads is required: x86-64 (with CONFIG_UNWINDER_ORC, which bpf_throw() already needs there) and arm64. Anywhere else the load fails with -EOPNOTSUPP rather than silently doing nothing.
- No offloaded programs, no private stack, and no combining a table with an exception callback.
- In a pad body: no tail call, no indirect jump, and no on-stack call arguments -- all of them read or write a stack the pad does not own.
- The Rust toolchain does not properly support BPF exception handling yet. The tables the selftests use are hand-written inline asm, which the assembler turns into the same relocations the BPF AsmPrinter emits, so libbpf and the kernel see an object indistinguishable from a compiler-generated one.
[1] https://github.com/llvm/llvm-project/pull/192164
llvm commit 9d51c891b719 ("[BPF] Add exception handling support with .bpf_cleanup section")
Yonghong Song (20):
bpf: Accept the compiler's exception cleanup table at program load
bpf: Add the bpf_unwind_resume() kfunc
bpf: Add lookups for exception cleanup resumes and landing pads
bpf: Mark the call sites an exception cleanup table covers
bpf: Make exception landing pads reachable in the CFG
bpf: Explore the landing pads no call site reaches
bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch
bpf: Walk the exception unwind in the verifier
bpf: Refuse a private stack for a program with an exception cleanup
table
bpf: Dispatch exception cleanup pads from bpf_throw()
bpf, x86: Dispatch exception cleanup pads at run time
bpf, arm64: Dispatch exception cleanup pads at run time
libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
libbpf: Add cleanup_info to bpf_prog_load_opts
libbpf: Collect .bpf_cleanup records and pass them to the kernel
libbpf: Carry the exception cleanup table through the light skeleton
libbpf: Let the static linker carry .bpf_cleanup relocations selftests/bpf: Add an end-to-end .bpf_cleanup exception test selftests/bpf: Cover the exception cleanup shapes the chain does not
reach
selftests/bpf: Load an exception cleanup program from a light skeleton
arch/arm64/net/Makefile | 2 +- arch/arm64/net/bpf_cleanup_pad.S | 95 ++ arch/arm64/net/bpf_jit_comp.c | 97 +- arch/x86/net/Makefile | 2 +- arch/x86/net/bpf_cleanup_pad.S | 74 ++ arch/x86/net/bpf_jit_comp.c | 78 +- include/linux/bpf.h | 75 ++ include/linux/bpf_cleanup_abi.h | 16 + include/linux/bpf_verifier.h | 11 + include/linux/filter.h | 2 + include/uapi/linux/bpf.h | 9 + kernel/bpf/Makefile | 2 +- kernel/bpf/cfg.c | 64 +- kernel/bpf/check_btf.c | 147 +++ kernel/bpf/core.c | 35 +- kernel/bpf/exception.c | 654 +++++++++++++ kernel/bpf/exception.h | 22 + kernel/bpf/fixups.c | 143 +++ kernel/bpf/helpers.c | 46 + kernel/bpf/liveness.c | 20 + kernel/bpf/states.c | 3 + kernel/bpf/syscall.c | 2 +- kernel/bpf/verifier.c | 125 ++- tools/include/uapi/linux/bpf.h | 9 + tools/lib/bpf/bpf.c | 6 +- tools/lib/bpf/bpf.h | 7 +- tools/lib/bpf/gen_loader.c | 27 +- tools/lib/bpf/libbpf.c | 305 ++++++- tools/lib/bpf/libbpf_internal.h | 10 + tools/lib/bpf/linker.c | 19 +- tools/testing/selftests/bpf/Makefile | 2 +- .../selftests/bpf/exceptions_cleanup.h | 52 ++ .../bpf/prog_tests/exceptions_cleanup.c | 421 +++++++++ .../selftests/bpf/progs/exceptions_cleanup.c | 152 +++ .../bpf/progs/exceptions_cleanup_ext_table.c | 48 + .../bpf/progs/exceptions_cleanup_fail.c | 600 ++++++++++++ .../bpf/progs/exceptions_cleanup_freplace.c | 17 + .../bpf/progs/exceptions_cleanup_light.c | 39 + .../progs/exceptions_cleanup_pad_freplace.c | 17 + .../bpf/progs/exceptions_cleanup_shapes.c | 863 ++++++++++++++++++ 40 files changed, 4254 insertions(+), 64 deletions(-) create mode 100644 arch/arm64/net/bpf_cleanup_pad.S create mode 100644 arch/x86/net/bpf_cleanup_pad.S create mode 100644 include/linux/bpf_cleanup_abi.h create mode 100644 kernel/bpf/exception.c create mode 100644 kernel/bpf/exception.h create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 1/20 bpf: Accept the compiler's exception cleanup table at program load
LLVM 23 added exception handling support for BPF with the .bpf_cleanup section ([1]). Rust code compiled with panic=unwind runs cleanup code (Drop glue) when bpf_throw() fires, and the LLVM BPF backend emits that section from the landing pads the frontend produced. Plain C cannot generate .bpf_cleanup unless inline asm is used. The Rust compiler does not *properly* support BPF exception handling yet, but the kernel can support the table today, and inline assembly is enough to test it.
Add the UAPI to carry the .bpf_cleanup table into the kernel. BPF_PROG_LOAD grows cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, and struct bpf_cleanup_info describes one record as a triple of instruction indices:
the half-open call-site range [begin_off, end_off) and the landing_pad_off the frame resumes at. check_cleanup_info() validates the table a program is loaded with, so the rest of the kernel can rely on it.
[1] https://github.com/llvm/llvm-project/pull/192164
--- include/linux/bpf_verifier.h | 2 + include/uapi/linux/bpf.h | 9 ++ kernel/bpf/check_btf.c | 147 +++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 2 +- kernel/bpf/verifier.c | 1 + tools/include/uapi/linux/bpf.h | 9 ++ 6 files changed, 169 insertions(+), 1 deletion(-)
include/linux/bpf_verifier.h +2 -0
@@ -987,6 +987,8 @@ struct bpf_verifier_env {struct arg_track **callsite_at_stack;u32 pass_cnt;/* number of times do_check() was called */ u32 subprog_cnt; +struct bpf_cleanup_info *cleanup_info; +u32 cleanup_info_cnt;/* number of instructions analyzed by the verifier */ u32 prev_insn_processed, insn_processed;/* number of jmps, calls, exits analyzed so far */
include/uapi/linux/bpf.h +9 -0
@@ -1669,6 +1669,9 @@ union bpf_attr {* verification. */ __s32 keyring_id; +__aligned_u64 cleanup_info;/* exception cleanup table */ +__u32 cleanup_info_rec_size;/* userspace bpf_cleanup_info size */ +__u32 cleanup_info_cnt;/* number of bpf_cleanup_info records */ };struct {/* anonymous struct used by BPF_OBJ_* commands */
@@ -7588,6 +7591,12 @@ struct bpf_line_info {__u32 line_col; }; +struct bpf_cleanup_info { +__u32 begin_off; +__u32 end_off; +__u32 landing_pad_off; +}; +struct bpf_spin_lock {__u32 val; };
kernel/bpf/check_btf.c +147 -0
159 lines not on this page · raw message
kernel/bpf/syscall.c +1 -1
@@ -2912,7 +2912,7 @@ int __init __used bpf_multi_func(void) { return 0; }BTF_ID_LIST_GLOBAL_SINGLE (bpf_multi_func_btf_id, func, bpf_multi_func) /* last field in 'union bpf_attr' used by this command */ -#define BPF_PROG_LOAD_LAST_FIELD keyring_id +#define BPF_PROG_LOAD_LAST_FIELD cleanup_info_cntstatic int bpf_prog_load (union bpf_attr *attr,bpfptr_t uattr,struct bpf_log_attr *attr_log) {
kernel/bpf/verifier.c +1 -0
@@ -21845,6 +21845,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,kvfree (env->succ);kvfree (env->gotox_tmp_buf);bpf_diag_free (env); +kvfree (env->cleanup_info);kvfree (env);return ret; }
tools/include/uapi/linux/bpf.h +9 -0
@@ -1669,6 +1669,9 @@ union bpf_attr {* verification. */ __s32 keyring_id; +__aligned_u64 cleanup_info;/* exception cleanup table */ +__u32 cleanup_info_rec_size;/* userspace bpf_cleanup_info size */ +__u32 cleanup_info_cnt;/* number of bpf_cleanup_info records */ };struct {/* anonymous struct used by BPF_OBJ_* commands */
@@ -7588,6 +7591,12 @@ struct bpf_line_info {__u32 line_col; }; +struct bpf_cleanup_info { +__u32 begin_off; +__u32 end_off; +__u32 landing_pad_off; +}; +struct bpf_spin_lock {__u32 val; };
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 2/20 bpf: Add the bpf_unwind_resume() kfunc
In reply to Yonghong SongA compiler-emitted cleanup landing pad ends with a call to _Unwind_Resume(), which carries the unwind on once that frame's cleanups have run. The kernel provides the same terminator as a kfunc, named bpf_unwind_resume() to keep the 'bpf_' prefix kfunc convention; a later libbpf patch resolves the compiler's name to it.
The body never runs. A JIT emits the way back out of a landing pad in its place.
--- kernel/bpf/helpers.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/bpf/helpers.c +12 -0
@@ -3407,6 +3407,17 @@ __bpf_kfunc void bpf_throw(u64 cookie)WARN (1 ,"A call to BPF exception callback should never return\n" ); } +/* +* Terminator of a compiler-emitted cleanup landing pad. The compiler names +* this _Unwind_Resume, the base unwind ABI's entry point for carrying an +* unwind on once a frame's cleanups have run. To match kernel kfunc +* convention, the kernel calls it bpf_unwind_resume and libbpf maps the +* compiler's name onto it. +*/ +__bpf_kfunc void bpf_unwind_resume(void) +{ +} + __bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, unsigned int flags) {struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
@@ -4853,6 +4864,7 @@ BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_throw) +BTF_ID_FLAGS(func, bpf_unwind_resume)#ifdef CONFIG_BPF_EVENTS BTF_ID_FLAGS(func, bpf_send_signal_task)#endif
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 3/20 bpf: Add lookups for exception cleanup resumes and landing pads
In reply to Yonghong SongAdd two new files, exception.h and exception.c, to host the exception handling code. Only a few helpers so far: recognising a call to bpf_unwind_resume(), and asking which landing pad, if any, a call site unwinds to.
The pad of a call site is kept in insn_aux_data, so the two places that move instructions around -- bpf_patch_insn_data() and verifier_remove_insns() -- learn to keep it in step. Nothing sets the mark yet; the next patch does.
--- include/linux/bpf_verifier.h | 6 ++++++ kernel/bpf/Makefile | 2 +- kernel/bpf/exception.c | 36 ++++++++++++++++++++++++++++++++++++ kernel/bpf/exception.h | 12 ++++++++++++ kernel/bpf/fixups.c | 17 +++++++++++++++++ 5 files changed, 72 insertions(+), 1 deletion(-) create mode 100644 kernel/bpf/exception.c create mode 100644 kernel/bpf/exception.h
include/linux/bpf_verifier.h +6 -0
@@ -681,6 +681,11 @@ struct bpf_insn_aux_data {bool needs_zext;/* alu op needs to clear upper bits */ bool non_sleepable;/* helper/kfunc may be called from non-sleepable context */ bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ +/* +* 1 + the instruction index of the exception cleanup landing pad this +* call site unwinds to, or 0 for none. +*/ +u32 cleanup_pad;bool call_with_percpu_alloc_ptr;/* {this,per}_cpu_ptr() with prog percpu alloc */ u8 alu_state;/* used in combination with alu_limit */ /* true if STX or LDX instruction is a part of a spill/fill
@@ -1518,6 +1523,7 @@ u32 btf_func_arg_align(const struct btf *btf, const struct btf_type *t);int bpf_find_subprog (struct bpf_verifier_env *env,int off);bool bpf_is_throw_kfunc (struct bpf_insn *insn); +bool bpf_is_unwind_resume_kfunc (const struct bpf_insn *insn);int bpf_compute_const_regs (struct bpf_verifier_env *env);int bpf_prune_dead_branches (struct bpf_verifier_env *env);int bpf_check_cfg (struct bpf_verifier_env *env);
kernel/bpf/Makefile +1 -1
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_ obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o -obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o +obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o exception.o obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o obj-$(CONFIG_BPF_JIT) += trampoline.o
kernel/bpf/exception.c new file +36 -0
@@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <linux/bpf.h> +#include <linux/bpf_verifier.h> +#include <linux/btf.h> +#include <linux/btf_ids.h> +#include <linux/filter.h> +#include <linux/slab.h> +#include "exception.h" + +#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args) + +enum exc_kfunc { + EXC_KF_bpf_unwind_resume, +}; + +BTF_ID_LIST (exc_kfunc_list) +BTF_ID (func, bpf_unwind_resume) + +static bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf) +{ + return bpf_pseudo_kfunc_call(insn) && insn->off == 0 && + insn->imm == exc_kfunc_list[kf]; +} + +bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn) +{ +return insn_is_exc_kfunc (insn, EXC_KF_bpf_unwind_resume); +} + +int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx) +{ +u32 pad = env->insn_aux_data[idx].cleanup_pad; + +return pad ? (int )pad -1 :-1 ; +}
kernel/bpf/exception.h new file +12 -0
@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#ifndef _LINUX_BPF_EXCEPTION_H +#define _LINUX_BPF_EXCEPTION_H + +#include <linux/types.h> + +struct bpf_verifier_env ; + +int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx); + +#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +17 -0
@@ -261,6 +261,11 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env, } } +if (env->cleanup_info_cnt) +for (i =0 ; i < prog_len; i++) +if (data[i].cleanup_pad > off +1 ) + data[i].cleanup_pad += cnt -1 ; +/* * Last slot instruction could be a newly generated * BPF_ST/BPF_LDX/BPF_STX, systematically mark it for non-stack access
@@ -549,6 +554,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)struct bpf_insn_aux_data *aux_data = env->insn_aux_data;unsigned int orig_prog_len = env->prog->len;int err; +u32 i;if (bpf_prog_is_offloaded (env->prog->aux))bpf_prog_offload_remove_insns (env, off, cnt);
@@ -573,6 +579,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) sizeof(*aux_data) * (orig_prog_len - off - cnt)); env->insn_aux_data_len -= cnt; +if (env->cleanup_info_cnt) { +for (i =0 ; i < env->insn_aux_data_len; i++) { +u32 pad = aux_data[i].cleanup_pad; + +if (pad > off + cnt) + aux_data[i].cleanup_pad = pad - cnt; +else if (pad > off) + aux_data[i].cleanup_pad =0 ; + } + } +return 0 ; }
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 4/20 bpf: Mark the call sites an exception cleanup table covers
In reply to Yonghong SongSome plumbing work is done before bpf_check_cfg(). More specifically, insn_aux_data records cleanup_throw_site for every bpf_throw(), and cleanup_pad, the landing pad a frame resumes at, for every call within the [begin_off, end_off) range of a cleanup record. Subsequent commits consume both.
bpf_prepare_cleanup_exceptions() runs before bpf_check_cfg(), because what it produces is what the CFG walk consumes. It refuses a table on an offloaded program, on a program whose JIT cannot dispatch landing pads or which the JIT was not asked to compile, and on a program that also installs an exception callback -- two different answers to what runs on the way out. bpf_jit_supports_cleanup_pads() is weak here and says no; the arch patches provide the real ones.
--- include/linux/bpf_verifier.h | 1 + include/linux/filter.h | 1 + kernel/bpf/core.c | 5 ++++ kernel/bpf/exception.c | 55 ++++++++++++++++++++++++++++++++++++ kernel/bpf/exception.h | 1 + kernel/bpf/verifier.c | 6 ++++ 6 files changed, 69 insertions(+)
include/linux/bpf_verifier.h +1 -0
@@ -681,6 +681,7 @@ struct bpf_insn_aux_data {bool needs_zext;/* alu op needs to clear upper bits */ bool non_sleepable;/* helper/kfunc may be called from non-sleepable context */ bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ +bool cleanup_throw_site;/* call to bpf_throw() */ /* * 1 + the instruction index of the exception cleanup landing pad this * call site unwinds to, or 0 for none.
include/linux/filter.h +1 -0
@@ -1242,6 +1242,7 @@ bool bpf_jit_supports_stack_args(void);bool bpf_jit_supports_arena_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void ); +bool bpf_jit_supports_cleanup_pads (void );bool bpf_jit_supports_ptr_xchg (void );bool bpf_jit_supports_arena (void );bool bpf_jit_supports_insn (struct bpf_insn *insn,bool in_arena);
kernel/bpf/core.c +5 -0
@@ -3470,6 +3470,11 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, { } +bool __weak bpf_jit_supports_cleanup_pads (void ) +{ +return false; +} +bool __weak bpf_jit_supports_timed_may_goto (void ) {return false;
kernel/bpf/exception.c +55 -0
61 lines not on this page · raw message
kernel/bpf/exception.h +1 -0
@@ -7,6 +7,7 @@struct bpf_verifier_env ; +int bpf_prepare_cleanup_exceptions (struct bpf_verifier_env *env);int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx);#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/verifier.c +6 -0
@@ -37,6 +37,7 @@#include "diagnostics.h" #include "disasm.h" +#include "exception.h" static const struct bpf_verifier_ops *const bpf_verifier_ops[] = { #define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
@@ -21638,6 +21639,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,if (ret <0 ) goto skip_full_check; +/* The CFG needs an edge from a call in a cleanup range to its pad. */ + ret =bpf_prepare_cleanup_exceptions (env); +if (ret <0 ) + goto skip_full_check; +/* Validate instructions and resolve the program's referenced resources. */ ret =check_and_resolve_insns (env);if (ret <0 )
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 5/20 bpf: Make exception landing pads reachable in the CFG
In reply to Yonghong SongBoth the CFG walk and liveness are involved. A bpf_throw() or a bpf2bpf call inside the [begin_off, end_off) range of a cleanup record can reach that record's landing pad, so both grow that edge; and a bpf_unwind_resume() reaches nothing after it, so it has no successors.
Liveness needs one more thing. bpf_stack_slot_alive() decides whether an outer frame's stack slot is still read after the call the frame is suspended at by looking at the instruction after the call. A slot whose only remaining reader is the landing pad -- which is every slot a compiler-generated pad reloads, since only four registers survive a call -- is dead by that measure, so clean_verifier_state() poisons it while the callee runs and the pad is then rejected for reading it. Ask about the pad as well when the call site names one.
--- kernel/bpf/cfg.c | 47 ++++++++++++++++++++++++++++++++++++++++--- kernel/bpf/liveness.c | 20 ++++++++++++++++++ 2 files changed, 64 insertions(+), 3 deletions(-)
kernel/bpf/cfg.c +44 -3
@@ -6,6 +6,7 @@#include <linux/sort.h> #include "diagnostics.h" +#include "exception.h" #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@@ -158,17 +159,57 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)return DONE_EXPLORING ; } +static int visit_cleanup_pad_edge (int t,struct bpf_verifier_env *env) +{ +int *insn_stack = env->cfg.insn_stack; +int *insn_state = env->cfg.insn_state; +int w; + +if (!env->cleanup_info_cnt) +return DONE_EXPLORING ; + w =bpf_cleanup_pad_of_call (env, t); +if (w <0 ) +return DONE_EXPLORING ; + +mark_prune_point (env, t); +mark_jmp_point (env, w); +mark_jump_target (env, w); + +if (insn_state[w]) +return DONE_EXPLORING ; +if (env->cfg.cur_stack >= env->prog->len) +return -E2BIG ; + insn_stack[env->cfg.cur_stack++] = w; + insn_state[w] |=DISCOVERED ; +return KEEP_EXPLORING ; +} + +static int merge_visit_ret (int a,int b) +{ +if (a <0 ) +return a; +if (b <0 ) +return b; +if (a ==KEEP_EXPLORING || b ==KEEP_EXPLORING ) +return KEEP_EXPLORING ; +return DONE_EXPLORING ; +} +static int visit_func_call_insn (int t,struct bpf_insn *insns,struct bpf_verifier_env *env,bool visit_callee) { -int ret, insn_sz; +int ret, insn_sz, pad_ret;int w; + pad_ret =visit_cleanup_pad_edge (t, env); +if (pad_ret <0 ) +return pad_ret; + insn_sz =bpf_is_ldimm64 (&insns[t]) ?2 :1 ; ret =push_insn (t, t + insn_sz,FALLTHROUGH , env);if (ret) -return ret; +return merge_visit_ret (pad_ret, ret);mark_prune_point (env, t + insn_sz);/* when we exit from subprog, we need to record non-linear history */
@@ -180,7 +221,7 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,merge_callee_effects (env, t, w); ret =push_insn (t, w,BRANCH , env); } -return ret; +return merge_visit_ret (pad_ret, ret); }struct bpf_iarray *bpf_iarray_realloc (struct bpf_iarray *old,size_t n_elem)
kernel/bpf/liveness.c +20 -0
@@ -8,6 +8,8 @@#include <linux/slab.h> #include <linux/sort.h> +#include "exception.h" + #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)struct per_frame_masks {
@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) succ = env->succ; succ->cnt =0 ; +if (unlikely (bpf_is_unwind_resume_kfunc (insn))) +return succ; + opcode_info = &opcode_info_tbl[BPF_CLASS (insn->code) |BPF_OP (insn->code)]; insn_sz =bpf_is_ldimm64 (insn) ?2 :1 ;if (opcode_info->can_fallthrough)
@@ -264,6 +269,13 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)if (opcode_info->can_jump) succ->items[succ->cnt++] = idx +bpf_jmp_offset (insn) +1 ; +if (unlikely (env->cleanup_info_cnt)) { +int pad =bpf_cleanup_pad_of_call (env, idx); + +if (pad >=0 ) + succ->items[succ->cnt++] = pad; + } +return succ; }
@@ -397,6 +409,14 @@ bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp alive =bpf_calls_callback (env, callsite) ?is_live_before (instance, callsite, rel, half_spi) :is_live_before (instance, callsite +1 , rel, half_spi); + +/* Control may also go to the landing pad. */ +if (!alive &&unlikely (env->cleanup_info_cnt)) { +int pad =bpf_cleanup_pad_of_call (env, callsite); + +if (pad >=0 ) + alive =is_live_before (instance, pad, rel, half_spi); + }if (alive)return true; }
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 05/20] bpf: Make exception landing pads reachable in the CFG
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] bpf: bpf_unwind_resume() retains fallthrough edge in CFG walk --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: bpf_unwind_resume() retains fallthrough edge in CFG walk
--
commit 663c539069f6d8aab713a59c4d2aa7c90391da34
Author: Yonghong Song <[email protected]>
bpf: Make exception landing pads reachable in the CFG
This commit updates the verifier's CFG walk and liveness analysis to account for exception landing pads. It adds edges from calls within a cleanup record's range to that record's landing pad. The commit message also states that bpf_unwind_resume() reaches nothing after it and should have no successors.
Quoted 4 lines
kernel/bpf/cfg.c +0
-0
[ ... ]
Quoted 1 line
kernel/bpf/cfg.c +0
-0@@ -158,17 +159,57 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
[ ... ]
Quoted 4 lines
static int visit_func_call_insn(int t, struct
bpf_insn*insns,kernel/bpf/cfg.c +0
-0struct bpf_verifier_env *env, bool visit_callee) {
kernel/bpf/cfg.c static int push_insn(int t, int w, int e, struct bpf_verifier_env *env) · line 210
Quoted 10 lines
kernel/bpf/cfg.c +5
-1-int ret, insn_sz; +int ret, insn_sz, pad_ret;int w; + pad_ret =visit_cleanup_pad_edge (t, env); +if (pad_ret <0 ) +return pad_ret; + insn_sz =bpf_is_ldimm64 (&insns[t]) ?2 :1 ; ret =push_insn (t, t + insn_sz,FALLTHROUGH , env);
[Severity: High]
Does this unconditionally push a fallthrough edge for all call instructions, failing to account for non-returning kfuncs like bpf_unwind_resume()?
If the compiler places a bpf_unwind_resume() call at the very end of the program (the end of the .text section) without any trailing instructions, will the CFG walk attempt to push a fallthrough edge out of bounds and incorrectly reject valid BPF programs with a 'jump out of range' error?
Quoted 5 lines
kernel/bpf/cfg.c +1
-1if (ret) -return ret; +return merge_visit_ret (pad_ret, ret);mark_prune_point (env, t + insn_sz);
[ ... ]
Quoted 4 lines
kernel/bpf/liveness.c +0
-0
[ ... ]
kernel/bpf/liveness.c bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) · line 263
Quoted 7 lines
kernel/bpf/liveness.c +3
-0@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) succ = env->succ; succ->cnt =0 ; +if (unlikely (bpf_is_unwind_resume_kfunc (insn))) +return succ; +
[Severity: High]
This correctly applies the logic mentioned in the commit message (that bpf_unwind_resume() reaches nothing after it and has no successors) to the liveness algorithm. Should visit_func_call_insn() in kernel/bpf/cfg.c be updated with similar logic to avoid pushing a fallthrough edge?
Quoted 3 lines
kernel/bpf/liveness.c +0
-0opcode_info = &opcode_info_tbl[BPF_CLASS (insn->code) |BPF_OP (insn->code)]; insn_sz =bpf_is_ldimm64 (insn) ?2 :1 ;if (opcode_info->can_fallthrough)
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Yonghong Song
reviewerRe: [PATCH bpf-next 05/20] bpf: Make exception landing pads reachable in the CFG
Quoted 39 lines · On 9/16/26 11:22 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf:bpf_unwind_resume()retains fallthrough edge in CFG walk
--commit
663c539069f6d8aab713a59c4d2aa7c90391da34
Author: Yonghong Song <[email protected]>bpf: Make exception landing pads reachable in the CFG
This commit updates the verifier's CFG walk and liveness analysis to account for exception landing pads. It adds edges from calls within a cleanup record's range to that record's landing pad. The commit message also states that
bpf_unwind_resume()reaches nothing after it and should have no successors.Quoted 4 lines
kernel/bpf/cfg.c +0
-0[ ... ]
Quoted 1 line
kernel/bpf/cfg.c +0
-0@@ -158,17 +159,57 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)[ ... ]
Quoted 4 lines
static int visit_func_call_insn(int t, struct
bpf_insn*insns,
structbpf_verifier_env*env, boolvisit_callee)
{kernel/bpf/cfg.c static int push_insn(int t, int w, int e, struct bpf_verifier_env *env) · line 210
Quoted 10 lines
kernel/bpf/cfg.c +5
-1-int ret, insn_sz; +int ret, insn_sz, pad_ret;int w; + pad_ret =visit_cleanup_pad_edge (t, env); +if (pad_ret <0 ) +return pad_ret; + insn_sz =bpf_is_ldimm64 (&insns[t]) ?2 :1 ; ret =push_insn (t, t + insn_sz,FALLTHROUGH , env);[Severity: High]
Does this unconditionally push a fallthrough edge for all call instructions, failing to account for non-returning kfuncs likebpf_unwind_resume()?
Quoted 4 lines
If the compiler places a
bpf_unwind_resume()call at the very end of the program (the end of the .text section) without any trailing instructions, will the CFG walk attempt to push a fallthrough edge out of bounds and incorrectly reject valid BPF programs with a 'jump out of range' error?
This won't happen. See verifier.c like below:
if (code != (BPF_JMP | BPF_EXIT) &&
code != (BPF_JMP32 | BPF_JA) &&
code != (BPF_JMP | BPF_JA)) {
verbose(env, "last insn is not an exit or jmp\n");
...
}Quoted 16 lines
Quoted 6 lines
if (ret)
kernel/bpf/cfg.c +1
-1-return ret; +return merge_visit_ret (pad_ret, ret);mark_prune_point (env, t + insn_sz);[ ... ]
Quoted 4 lines
kernel/bpf/liveness.c +0
-0[ ... ]
Quoted 4 lines
kernel/bpf/liveness.c +0
-0@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) succ = env->succ; succ->cnt =0 ;
Quoted 8 lines
kernel/bpf/liveness.c bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) · line 263
Quoted 3 lines
kernel/bpf/liveness.c +3
-0+if (unlikely (bpf_is_unwind_resume_kfunc (insn))) +return succ; +[Severity: High]
This correctly applies the logic mentioned in the commit message (thatbpf_unwind_resume()reaches nothing after it and has no successors) to the liveness algorithm. Shouldvisit_func_call_insn()inkernel/bpf/cfg.cbe updated with similar logic to avoid pushing a fallthrough edge?
The same as above, bpf_unwind_resume() cannot be the last insn in the prog.
Quoted 4 lines
Quoted 4 lines
kernel/bpf/liveness.c +0
-0opcode_info = &opcode_info_tbl[BPF_CLASS (insn->code) |BPF_OP (insn->code)]; insn_sz =bpf_is_ldimm64 (insn) ?2 :1 ;if (opcode_info->can_fallthrough)
Yonghong Song
reviewerPATCH 6/20 bpf: Explore the landing pads no call site reaches
In reply to Yonghong SongA cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving bpf_check_cfg() to refuse the program over code its own frontend had no way not to emit:
0: call bpf_preempt_disable
1: call bpf_preempt_enable record = { begin = 1, end = 2, pad = 4 }
2: r0 = 0
3: exit
4: r1 = pads_ran ll landing pad
6: r2 = *(u64 *)(r1 + 0)
7: r2 |= RAN_NOUNWIND_REC
8: *(u64 *)(r1 + 0) = r2
9: call bpf_unwind_resume
10: exit
The range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it. cleanup_mark_call_sites() marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.
Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other: do_check() never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.
--- kernel/bpf/cfg.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
kernel/bpf/cfg.c +17 -0
@@ -633,6 +633,7 @@ int bpf_check_cfg(struct bpf_verifier_env *env)int insn_cnt = env->prog->len;int *insn_stack, *insn_state;int ex_insn_beg, i, ret =0 ; +u32 pad_idx =0 ; insn_state = env->cfg.insn_state =kvzalloc_objs (int, insn_cnt,GFP_KERNEL_ACCOUNT );
@@ -688,6 +689,22 @@ int bpf_check_cfg(struct bpf_verifier_env *env) goto walk_cfg; } +/* +* A landing pad no call site was marked with -- a record whose range +* holds no call an exception can unwind out of -- is reached by +* nothing. Walk it from here, and let the dead code sweep remove it. +*/ +while (pad_idx < env->cleanup_info_cnt) { +u32 pad = env->cleanup_info[pad_idx++].landing_pad_off; + +if (insn_state[pad] !=EXPLORED ) { + insn_state[pad] =DISCOVERED ; + insn_stack[0 ] = pad; + env->cfg.cur_stack =1 ; + goto walk_cfg; + } + } +for (i =0 ; i < insn_cnt; i++) {struct bpf_insn *insn = &env->prog->insnsi[i];
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 7/20 bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch
In reply to Yonghong SongThe table is on the instructions and the landing pads are in the control flow graph. What is left before bpf_throw() can be taught to dispatch them is to work out what that walk will need, and refuse the shapes it could not handle.
bpf_check_cleanup_exceptions() runs after bpf_check_cfg(). Everything it needs is control flow, so it reads what that walk has already worked out rather than working it out again:
subprog_info.might_throw which subprograms an exception may leave,
closed over the call graph by merge_callee_effects() as the walk pops each callee
cleanup_reachability() what each instruction can reach -- a resume, a
plain exit, a throw, an indirect jump
cleanup_mark_pad_bodies() which instructions only ever run with an
exception already in flight
What it refuses:
- a pad that reaches both a resume and a plain exit, or neither: nothing says whether it is a cleanup pad or a catch pad
- a catch pad, which ends in a plain exit: the walker calls a pad as a subroutine and cannot hand a frame back its own execution
- a throw in a pad, or a call from a pad to a subprogram that can throw:
a second unwind over frames the first is still discarding
- a bpf_unwind_resume() outside a pad body
- a subprogram that may unwind used as a helper callback: the helper's own kernel frame would end the walk before it found a boundary
- a tail call, an indirect jump, or an outgoing on-stack call argument in a pad body, all of which touch a stack the pad does not own
--- include/linux/bpf_verifier.h | 1 + kernel/bpf/exception.c | 397 +++++++++++++++++++++++++++++++++++ kernel/bpf/exception.h | 2 + kernel/bpf/fixups.c | 1 + kernel/bpf/verifier.c | 8 + 5 files changed, 409 insertions(+)
include/linux/bpf_verifier.h +1 -0
@@ -682,6 +682,7 @@ struct bpf_insn_aux_data {bool non_sleepable;/* helper/kfunc may be called from non-sleepable context */ bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ bool cleanup_throw_site;/* call to bpf_throw() */ +bool in_cleanup_pad;/* only runs with an exception in flight */ /* * 1 + the instruction index of the exception cleanup landing pad this * call site unwinds to, or 0 for none.
kernel/bpf/exception.c +397 -0
421 lines not on this page · raw message
kernel/bpf/exception.h +2 -0
@@ -8,6 +8,8 @@struct bpf_verifier_env ;int bpf_prepare_cleanup_exceptions (struct bpf_verifier_env *env); +int bpf_check_cleanup_exceptions (struct bpf_verifier_env *env); +int bpf_cleanup_check_callback (struct bpf_verifier_env *env,int subprog);int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx);#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +1 -0
@@ -252,6 +252,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,/* Expand insni[off]'s seen count to the patched range. */ data[i].seen = old_seen; data[i].zext_dst =bpf_insn_def32 (new_prog, insn + i) >=0 ; + data[i].in_cleanup_pad = data[off + cnt -1 ].in_cleanup_pad;if (!memcmp (insn + i, original_insn,sizeof (struct bpf_insn ))) { data[i].non_stack_access = data[off + cnt -1 ].non_stack_access;
kernel/bpf/verifier.c +8 -0
@@ -10485,6 +10485,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins* callbacks */ env->subprog_info[subprog].is_cb = true; + err =bpf_cleanup_check_callback (env, subprog); +if (err) +return err; +if (bpf_pseudo_kfunc_call (insn) && !is_callback_calling_kfunc (insn->imm)) { verifier_bug(env, "kfunc %s#%d not marked as callback-calling",
@@ -21664,6 +21668,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,if (ret <0 ) goto skip_full_check; + ret =bpf_check_cleanup_exceptions (env); +if (ret <0 ) + goto skip_full_check; + ret =bpf_compute_postorder (env);if (ret <0 ) goto skip_full_check;
Signature · 1 line
2.53.0-Meta
Re: [PATCH bpf-next 07/20] bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch
Quoted 4 lines · Yonghong Song
kernel/bpf/exception.c +0
-0
[...]
Quoted 5 lines
kernel/bpf/exception.c +5
-0+static enum cleanup_insn_kind cleanup_classify (struct bpf_verifier_env *env,u32 i, +int *next,int *target) +{ +struct bpf_insn *insn = &env->prog->insnsi[i]; +u8 class =BPF_CLASS (insn->code);
kernel/bpf/exception.c static bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf) · line 122
Quoted 10 lines
kernel/bpf/exception.c +10
-0+ + *next = i +1 ; + *target =-1 ; + +if (insn->code == (BPF_LD |BPF_IMM |BPF_DW )) { + *next = i +2 ; +return CLEANUP_INSN_PLAIN ; + } +if (class !=BPF_JMP && class !=BPF_JMP32 ) +return CLEANUP_INSN_PLAIN ;
robot voice:
LD_ABS/LD_IND is not plain. bpf_check_cfg() treats it as an abnormal return and gen_ld_abs() later patches in a real "r0 = 0; exit" for the failed load. With this classification a pad in a tc prog that keeps skb in r6 and does an ld_abs passes t
he "reaches no exit" check,cleanup_check_pad_insn() doesn't look at it, and check_ld_abs() only wants no locks/refs held at that point. At run time a failed load runs the subprog epilogue on the walker's stack: callee regs get popped from arch_bpf_run_cleanup_pad()'s frame, leave/ret with rbp == the unwinding frame goes straight back into the BPF caller with garbage in rbx/r13-r15, and the rest of bpf_throw() is abandoned. Treat it like BPF_EXIT here or refuse it in a pad body like tail call.
Yonghong Song
reviewerRe: [PATCH bpf-next 07/20] bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch
Quoted 24 lines · Alexei Starovoitov
Quoted 4 lines · Yonghong Song
kernel/bpf/exception.c +0
-0[...]
Quoted 5 lines
kernel/bpf/exception.c +5
-0+static enum cleanup_insn_kind cleanup_classify (struct bpf_verifier_env *env,u32 i, +int *next,int *target) +{ +struct bpf_insn *insn = &env->prog->insnsi[i]; +u8 class =BPF_CLASS (insn->code);kernel/bpf/exception.c static bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf) · line 122
Quoted 10 lines
kernel/bpf/exception.c +10
-0+ + *next = i +1 ; + *target =-1 ; + +if (insn->code == (BPF_LD |BPF_IMM |BPF_DW )) { + *next = i +2 ; +return CLEANUP_INSN_PLAIN ; + } +if (class !=BPF_JMP && class !=BPF_JMP32 ) +return CLEANUP_INSN_PLAIN ;robot voice:
Quoted 12 lines
LD_ABS/LD_IND is not plain.
bpf_check_cfg()treats it as an abnormal return andgen_ld_abs()later patches in a real "r0 = 0; exit" for the failed load. With this classification a pad in a tc prog that keeps skb in r6 and does anld_abspasses t
he "reaches no exit" check,cleanup_check_pad_insn()doesn't look at it, andcheck_ld_abs()only wants no locks/refs held at that point. At run time a failed load runs the subprog epilogue on the walker's stack: callee regs get popped from arch_bpf_run_cleanup_pad()'s frame, leave/ret with rbp == the unwinding frame goes straight back into the BPF caller with garbage in rbx/r13-r15, and the rest ofbpf_throw()is abandoned. Treat it likeBPF_EXIThere or refuse it in a pad body like tail call.
Yes, this is an issue. For insns in landing pad, we should not allow LD_ABS/LD_IND since they will have a hidden return. So the above suggestion to refuse it in a pad body is correct.
Yonghong Song
reviewerPATCH 8/20 bpf: Walk the exception unwind in the verifier
In reply to Yonghong Songbpf_throw() is about to stop discarding the frames it unwinds and start running their landing pads instead. Have the verifier walk the same thing, step for step:
at a throw, with the frame chain in env->cur_state->frame[]
if a record covers the call control is at
run that record's landing pad in this frame on reaching its resume, pop the frame and carry on
else
pop the frame and carry on
at the boundary, deliver
Doing it this way is what keeps the resource rules unchanged. Whatever a pad releases is released in the verifier state too, so by the time the walk reaches the boundary the state says exactly what the program will really hold there -- and check_resource_leak(), which used to fire the moment a throw was seen, simply moves to the end of the walk. A frame that no record covers contributes nothing, so anything it held is still held when the walk ends, and that is what gets reported.
--- include/linux/bpf_cleanup_abi.h | 16 +++++ include/linux/bpf_verifier.h | 1 + kernel/bpf/states.c | 3 + kernel/bpf/verifier.c | 102 +++++++++++++++++++++++++------- 4 files changed, 99 insertions(+), 23 deletions(-) create mode 100644 include/linux/bpf_cleanup_abi.h
include/linux/bpf_cleanup_abi.h new file +16 -0
@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#ifndef _LINUX_BPF_CLEANUP_ABI_H +#define _LINUX_BPF_CLEANUP_ABI_H + +/* +* Value arch_bpf_run_cleanup_pad() leaves in r0 on the way into a landing pad. +* It has to be a constant the verifier knows: LLVM names r0 as both the +* exception pointer and the exception selector register, so every pad reads it +* before anything else and is free to store what it read. Kept on its own +* because the verifier and the arch dispatchers, which are assembly, have to +* agree on it. +*/ +#define BPF_PAD_ENTRY_R0 1 + +#endif /* _LINUX_BPF_CLEANUP_ABI_H */
include/linux/bpf_verifier.h +1 -0
@@ -509,6 +509,7 @@ struct bpf_verifier_state {bool speculative;bool in_sleepable; +bool unwinding;/* first and last insn idx of this verifier state */ u32 first_insn_idx;
kernel/bpf/states.c +3 -0
@@ -996,6 +996,9 @@ static bool states_equal(struct bpf_verifier_env *env,if (old->in_sleepable != cur->in_sleepable)return false; +if (old->unwinding != cur->unwinding) +return false; +if (!refsafe (old, cur, &env->idmap_scratch))return false;
kernel/bpf/verifier.c +79 -23
160 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
Re: [PATCH bpf-next 08/20] bpf: Walk the exception unwind in the verifier
Quoted 4 lines · Yonghong Song
kernel/bpf/verifier.c +0
-0
[...]
Quoted 8 lines
kernel/bpf/verifier.c +8
-0+static int unwind_step (struct bpf_verifier_env *env,u32 callsite,int *insn_idx) +{ +struct bpf_verifier_state *state = env->cur_state; + + state->unwinding = true; +for (;;) { +int pad =bpf_cleanup_pad_of_call (env, callsite); +
kernel/bpf/verifier.c enum { · line 18491
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+if (pad >=0 ) { +unwind_enter_pad (env); + *insn_idx = pad; +return INSN_IDX_UPDATED ; + } +if (!state->curframe) +return unwind_finish (env); + callsite =unwind_pop_frame (env); + } +}
robot voice:
mark_chain_precision() doesn't know about this edge. The jmp history gets (bpf_throw in frame N+k) -> (pad in frame N), or (resume in N+1) -> (pad in N), with no BPF_EXIT in between, and backtrack_insn() only switches frames on E
XIT and pseudo calls. So bt->frame stays at N
while it walks the callee's insns backwards. If the callee wrote its own r6 the request for the caller's r6 is cleared there, the caller's def of r6 is never marked precise, and a later state with a different r6 is pruned at the call site checkpoint even though the pad does r10 + r6 with it. If the callee didn't touch r6 the walk reaches the static call insn with r6 still set and hits verifier_bug("static subprog unexpected regs"). For a throwing global subprog it's verifier_bug_if(idx + 1 != subseq_idx) right away. The unwind transition needs its own jmp history flag and backtrack_insn() has to bt_subprog_enter() once per popped frame, like it does for BPF_EXIT.
Pls add a test that does a variable offset stack access in a pad with the offset coming from r6-r9 set before the throwing call.
kernel/bpf/verifier.c enum {
Quoted 3 lines
kernel/bpf/verifier.c +3
-0+static int process_cleanup_resume (struct bpf_verifier_env *env,int *insn_idx) +{ +struct bpf_verifier_state *state = env->cur_
state;
kernel/bpf/verifier.c enum { · line 18507
Quoted 12 lines
kernel/bpf/verifier.c +12
-0+ +/* A pad entered by ordinary control flow. */ +if (!state->unwinding) { +verbose (env, +"bpf_unwind_resume() at insn %d reached without an exception in flight\n" , + *insn_idx); +return -EINVAL ; + } +if (!state->curframe) +return unwind_finish (env); +return unwind_step (env,unwind_pop_frame (env), insn_idx); +}
unwinding is one bit for the whole state, so once a throw happened a bpf_unwind_resume() is accepted in any frame, not only in the frame whose pad the walker dispatched. The in_pad rule from patch 7 doesn't close it: take subprog S that never throws, has its own record, and whose pad is just "call bpf_unwind_resume" reachable by a plain branch from S's entry. F's pad does "call S" and S branches into its pad. Here the verifier pops S, finds no pad for the "call S" site, pops F and finishes, so whatever follows "call S" in F's pad is never walked in this state. On x86 S's resume is a bare ret, so at run time it
returns into F's pad right after "call S" and keeps executing it with S's leftover registers. On arm64 br x23 ends F's pad early instead. Remember the frame unwind_step() entered the pad in and reject bpf_unwind_resume() when curframe doesn't match.
pw-bot: cr
Yonghong Song
reviewerRe: [PATCH bpf-next 08/20] bpf: Walk the exception unwind in the verifier
Quoted 73 lines · Alexei Starovoitov
Quoted 4 lines · Yonghong Song
kernel/bpf/verifier.c +0
-0[...]
Quoted 8 lines
kernel/bpf/verifier.c +8
-0+static int unwind_step (struct bpf_verifier_env *env,u32 callsite,int *insn_idx) +{ +struct bpf_verifier_state *state = env->cur_state; + + state->unwinding = true; +for (;;) { +int pad =bpf_cleanup_pad_of_call (env, callsite); +kernel/bpf/verifier.c enum { · line 18491
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+if (pad >=0 ) { +unwind_enter_pad (env); + *insn_idx = pad; +return INSN_IDX_UPDATED ; + } +if (!state->curframe) +return unwind_finish (env); + callsite =unwind_pop_frame (env); + } +}robot voice:
mark_chain_precision()doesn't know about this edge. The jmp history gets (bpf_throwin frame N+k) -> (pad in frame N), or (resume in N+1) -> (pad in N), with noBPF_EXITin between, andbacktrack_insn()only switches frames on E
XIT and pseudo calls. Sobt->framestays at N
while it walks the callee's insns backwards. If the callee wrote its own r6 the request for the caller's r6 is cleared there, the caller's def of r6 is never marked precise, and a later state with a different r6 is pruned at the call site checkpoint even though the pad does r10 + r6 with it. If the callee didn't touch r6 the walk reaches the static call insn with r6 still set and hitsverifier_bug("static subprog unexpected regs"). For a throwing global subprog it'sverifier_bug_if(idx + 1 != subseq_idx)right away. The unwind transition needs its own jmp history flag andbacktrack_insn()has tobt_subprog_enter()once per popped frame, like it does forBPF_EXIT.
Pls add a test that does a variable offset stack access in a pad with the offset coming from r6-r9 set before the throwing call.kernel/bpf/verifier.c enum {
Quoted 3 lines
kernel/bpf/verifier.c +3
-0+static int process_cleanup_resume (struct bpf_verifier_env *env,int *insn_idx) +{ +struct bpf_verifier_state *state = env->cur_state;
kernel/bpf/verifier.c enum { · line 18507
Quoted 12 lines
kernel/bpf/verifier.c +12
-0+ +/* A pad entered by ordinary control flow. */ +if (!state->unwinding) { +verbose (env, +"bpf_unwind_resume() at insn %d reached without an exception in flight\n" , + *insn_idx); +return -EINVAL ; + } +if (!state->curframe) +return unwind_finish (env); +return unwind_step (env,unwind_pop_frame (env), insn_idx); +}unwinding is one bit for the whole state, so once a throw happened a
bpf_unwind_resume()is accepted in any frame, not only in the frame whose pad the walker dispatched. Thein_padrule from patch 7 doesn't close it: take subprog S that never throws, has its own record, and whose pad is just "callbpf_unwind_resume" reachable by a plain branch from S's entry. F's pad does "call S" and S branches into its pad. Here the verifier pops S, finds no pad for the "call S" site, pops F and finishes, so whatever follows "call S" in F's pad is never walked in this state. On x86 S's resume is a bare ret, so at run time it
Quoted 4 lines
returns into F's pad right after "call S" and keeps executing it with S's leftover registers. On arm64 br x23 ends F's pad early instead. Remember the frame
unwind_step()entered the pad in and rejectbpf_unwind_resume()when curframe doesn't match.
Yes, this is a real issue for backtracking. The problem is stated in the above curframe doesn't match for frames after unwind_step(), mostly due to backtracking from landing pad insn to bpf_throw. Will fix.
Quoted 2 lines
pw-bot: cr
Yonghong Song
reviewerPATCH 9/20 bpf: Refuse a private stack for a program with an exception cleanup table
In reply to Yonghong SongA landing pad is about to start running outside its own frame, on the bpf_throw() walker's stack, with that frame's registers put back from a spill area whose contents arch_bpf_run_cleanup_pad() knows how to read. The x86-64 JIT addresses a private-stack program's frame through a scratch register that it recomputes after each call rather than through rbp, and no spill area holds that register, so a pad there would address its frame through whatever the kernel left behind.
Refuse the combination in check_max_stack_depth(), where the choice is made. Everywhere rather than arch-conditionally: arm64 keeps its private stack pointer in x27, which is in the prologue spill and so survives, but a rule that holds on one arch and not the other is not worth the second code path when nothing is lost but an optimization.
--- kernel/bpf/verifier.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/bpf/verifier.c +8 -0
@@ -5591,6 +5591,14 @@ static int check_max_stack_depth(struct bpf_verifier_env *env) } } +/* +* A pad rebuilds its frame from a spill area, and on x86-64 a private +* stack's frame pointer is in no spill area. Refused on every arch +* rather than just that one. +*/ +if (env->cleanup_info_cnt) + priv_stack_mode =NO_PRIV_STACK ; +if (priv_stack_mode ==PRIV_STACK_UNKNOWN ) priv_stack_mode =bpf_enable_priv_stack (env->prog);
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 10/20 bpf: Dispatch exception cleanup pads from bpf_throw()
In reply to Yonghong SongStop discarding the frames an exception unwinds through. bpf_throw() already walks the BPF call stack with arch_bpf_stack_walk() to find the exception boundary; have it look each frame's return address up in that (sub)program's cleanup table on the way and run the landing pad a matching record names.
A pad is run as a subroutine of the walker, not jumped to. It executes with the unwinding frame's frame pointer and BPF callee-saved registers, so everything it reads is that frame's, but on the current stack far below it, so nothing it calls can disturb the frame it is cleaning up after. It ends in what the JIT emits for its bpf_unwind_resume(), which hands control back to the walker rather than to the frame's caller. That is exactly the procedure the LLVM commit emitting the section describes.
Restoring the frame's r6-r9 is what makes this work, and it is only possible because the callee about to be discarded spilled them in its own prologue. bpf_cleanup_force_spill() tells a JIT to make that spill unconditional and of a known shape for every subprogram of a program carrying a cleanup table, and aux->exc->spill_off records where it starts, so the walker needs no per-frame metadata. The frame that called bpf_throw() has no callee to have spilled anything and never runs its own epilogue, so the JIT spills that frame's registers at the throw site instead, in the area aux->exc->throw_spill_off names.
The main program needs the table handed to it rather than built for it. jit_subprogs() compiles it as func[0], but the ksym covering that image is the one bpf_prog_load() registers for the outer bpf_prog, and that is what the walker finds -- so without the handover a landing pad in the main program's own frame is never dispatched, silently, since the exception still reaches the boundary and the cookie still comes back.
All of it hangs off bpf_prog_aux by a single pointer. bpf_prog_aux is on every BPF program and this is a niche feature, so struct bpf_exception_info exists only for a program that carries a table, and aux->exc being set is what says the program carries one at all.
--- include/linux/bpf.h | 75 +++++++++++++++++++ include/linux/filter.h | 1 + kernel/bpf/core.c | 30 +++++++- kernel/bpf/exception.c | 166 +++++++++++++++++++++++++++++++++++++++++ kernel/bpf/exception.h | 7 ++ kernel/bpf/fixups.c | 125 +++++++++++++++++++++++++++++++ kernel/bpf/helpers.c | 34 +++++++++ 7 files changed, 435 insertions(+), 3 deletions(-)
include/linux/bpf.h +75 -0
@@ -1770,6 +1770,80 @@ enum bpf_sig_keyring { BPF_SIG_KEYRING_BPF, }; +/* +* One cleanup region of a JITed (sub)program: @pad is the landing pad to run +* for a return address in (begin, end], the native code of its call sites. +*/ +struct bpf_cleanup_range { +u64 begin; +u64 end; +u64 pad; +}; + +struct bpf_exception_info { +struct bpf_cleanup_info *info; +struct bpf_cleanup_range *ranges; +/* Landing pad instruction indices, sorted and deduplicated. */ +u32 *pad_at; +/* bpf_throw() call instruction indices, sorted. */ +u32 *throw_at; +/* One bit per instruction that only runs while unwinding. */ +unsigned long *pad_body; +u32 nr_info; +u32 nr_ranges; +u32 nr_pad_at; +u32 nr_throw_at; +u32 nr_pad_body; +/* Offset from a frame's FP to the caller's spilled r6-r9. */ +s32 spill_off; +/* Likewise, to the registers a frame spills before calling bpf_throw(). */ +s32 throw_spill_off; +}; + +#ifdef CONFIG_BPF_SYSCALL +bool bpf_cleanup_force_spill (const struct bpf_prog *prog); +bool bpf_cleanup_insn_is_pad (const struct bpf_prog *prog,u32 idx); +bool bpf_cleanup_insn_in_pad (const struct bpf_prog *prog,u32 idx); +bool bpf_cleanup_insn_is_throw (const struct bpf_prog *prog,u32 idx); +int bpf_cleanup_attach_main_prog (struct bpf_verifier_env *env,struct bpf_prog *prog); +void bpf_cleanup_fill_native_ranges (struct bpf_prog *prog,u32 *addrs,void *image); +void bpf_cleanup_free_info (struct bpf_prog_aux *aux); +#else +static inline bool bpf_cleanup_force_spill (const struct bpf_prog *prog) +{ +return false; +} + +static inline bool bpf_cleanup_insn_is_pad (const struct bpf_prog *prog,u32 idx) +{ +return false; +} + +static inline bool bpf_cleanup_insn_in_pad (const struct bpf_prog *prog,u32 idx) +{ +return false; +} + +static inline bool bpf_cleanup_insn_is_throw (const struct bpf_prog *prog,u32 idx) +{ +return false; +} + +static inline int bpf_cleanup_attach_main_prog (struct bpf_verifier_env *env, +struct bpf_prog *prog) +{ +return 0 ; +} + +static inline void bpf_cleanup_fill_native_ranges (struct bpf_prog *prog,u32 *addrs,void *image) +{ +} + +static inline void bpf_cleanup_free_info (struct bpf_prog_aux *aux) +{ +} +#endif +struct bpf_prog_aux {atomic64_t refcnt;u32 used_map_cnt;
@@ -1850,6 +1924,7 @@ struct bpf_prog_aux {char name[BPF_OBJ_NAME_LEN ];u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 ,u64 );u16 stack_arg_sp_adjust; +struct bpf_exception_info *exc;#ifdef CONFIG_SECURITY void *security;#endif
include/linux/filter.h +1 -0
@@ -1243,6 +1243,7 @@ bool bpf_jit_supports_arena_args(void);bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );bool bpf_jit_supports_cleanup_pads (void ); +void arch_bpf_run_cleanup_pad (u64 pad,u64 frame_fp,u64 spill_base);bool bpf_jit_supports_ptr_xchg (void );bool bpf_jit_supports_arena (void );bool bpf_jit_supports_insn (struct bpf_insn *insn,bool in_arena);
kernel/bpf/core.c +27 -3
@@ -292,6 +292,7 @@ void __bpf_prog_free(struct bpf_prog *fp)mutex_destroy (&fp->aux->dst_mutex);mutex_destroy (&fp->aux->st_ops_assoc_mutex);kfree (fp->aux->poke_tab); +bpf_cleanup_free_info (fp->aux);kfree (fp->aux); }free_percpu (fp->stats);
@@ -2625,13 +2626,21 @@ static bool bpf_prog_select_interpreter(struct bpf_prog *fp)return select_interpreter; } -static struct bpf_prog *bpf_prog_jit_compile (struct bpf_verifier_env *env,struct bpf_prog *prog) +static struct bpf_prog *bpf_prog_jit_compile (struct bpf_verifier_env *env,struct bpf_prog *prog, +int *err) {#ifdef CONFIG_BPF_JIT struct bpf_prog *orig_prog; +int ret; -if (!bpf_prog_need_blind (prog)) +if (!bpf_prog_need_blind (prog)) { + ret =bpf_cleanup_attach_main_prog (env, prog); +if (ret) { + *err = ret; +return prog; + }return bpf_int_jit_compile (env, prog); + } orig_prog = prog; prog =bpf_jit_blind_constants (env, prog);
@@ -2642,6 +2651,13 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, strucif (IS_ERR (prog)) goto out_restore; + ret =bpf_cleanup_attach_main_prog (env, prog); +if (ret) { + *err = ret; +bpf_jit_prog_release_other (orig_prog, prog); + goto out_restore; + } + prog =bpf_int_jit_compile (env, prog);if (prog->jited) {bpf_jit_prog_release_other (prog, orig_prog);
@@ -2681,8 +2697,10 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, structif (*err)return fp; - fp =bpf_prog_jit_compile (env, fp); + fp =bpf_prog_jit_compile (env, fp, err);bpf_prog_jit_attempt_done (fp); +if (*err) +return fp;if (!fp->jited && jit_needed) { *err = -ENOTSUPP ;return fp;
@@ -3475,6 +3493,12 @@ bool __weak bpf_jit_supports_cleanup_pads(void)return false; } +/* Call @pad with the frame pointer @frame_fp and r6-r9 spilled at @spill_base. */ +void __weak arch_bpf_run_cleanup_pad (u64 pad,u64 frame_fp,u64 spill_base) +{ +WARN_ON_ONCE (1 ); +} +bool __weak bpf_jit_supports_timed_may_goto (void ) {return false;
kernel/bpf/exception.c +166 -0
174 lines not on this page · raw message
kernel/bpf/exception.h +7 -0
@@ -5,11 +5,18 @@#include <linux/types.h> +struct bpf_cleanup_info ; +struct bpf_cleanup_range ; +struct bpf_prog ; +struct bpf_prog_aux ;struct bpf_verifier_env ;int bpf_prepare_cleanup_exceptions (struct bpf_verifier_env *env);int bpf_check_cleanup_exceptions (struct bpf_verifier_env *env);int bpf_cleanup_check_callback (struct bpf_verifier_env *env,int subprog);int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx); +int bpf_cleanup_alloc_info (struct bpf_prog_aux *aux); +int bpf_cleanup_attach_info (struct bpf_prog_aux *aux,struct bpf_cleanup_info *recs,u32 cnt); +const struct bpf_cleanup_range *bpf_cleanup_pad_for_ip (const struct bpf_prog *prog,u64 ip);#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +125 -0
166 lines not on this page · raw message
kernel/bpf/helpers.c +34 -0
@@ -31,6 +31,7 @@#include <linux/buildid.h> #include "../../lib/kstrtox.h" +#include "exception.h" /* If kernel subsystem is allowing eBPF programs to call this function, * inside its own verifier_ops->get_func_proto() callback it should return
@@ -3360,8 +3361,36 @@ struct bpf_throw_ctx {u64 sp;u64 bp;int cnt; +const struct bpf_prog *callee; +u64 callee_fp; }; +static void bpf_run_cleanup_pad (struct bpf_throw_ctx *ctx,const struct bpf_prog *prog, +u64 ip,u64 fp) +{ +const struct bpf_exception_info *exc = prog->aux->exc; +const struct bpf_cleanup_range *rec; +u64 spill_base; + +if (!exc || !exc->nr_ranges) +return ; + rec =bpf_cleanup_pad_for_ip (prog, ip); +if (!rec) +return ; + +/* +* The callee is always another subprogram of this program -- the walk +* ends at any frame that is not one -- so its prologue spilled these +* registers and its exc is there to say where. +*/ +if (ctx->callee) + spill_base = ctx->callee_fp + ctx->callee->aux->exc->spill_off; +else + spill_base = fp + exc->throw_spill_off; + +arch_bpf_run_cleanup_pad (rec->pad, fp, spill_base); +} +static bool bpf_stack_walker (void *cookie,u64 ip,u64 sp,u64 bp) {struct bpf_throw_ctx *ctx = cookie;
@@ -3378,6 +3407,11 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)if (!prog)return !ctx->cnt; ctx->cnt++; + +bpf_run_cleanup_pad (ctx, prog, ip, bp); + ctx->callee = prog; + ctx->callee_fp = bp; +if (bpf_is_subprog (prog))return true; ctx->aux = prog->aux;
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 11/20 bpf, x86: Dispatch exception cleanup pads at run time
In reply to Yonghong SongProvide the arch half for x86-64: force the full callee-saved spill for a program carrying a cleanup table so the walker can find a frame's r6-r9 in its callee's prologue, record where that spill area starts, build the native cleanup table from the JIT's addrs[], emit a bare return for a pad's bpf_unwind_resume(), and hand control to a pad from arch_bpf_run_cleanup_pad().
Support is gated on CONFIG_UNWINDER_ORC, the same requirement arch_bpf_stack_walk() and therefore bpf_throw() already have here.
--- arch/x86/net/Makefile | 2 +- arch/x86/net/bpf_cleanup_pad.S | 74 ++++++++++++++++++++++++++++++++ arch/x86/net/bpf_jit_comp.c | 78 ++++++++++++++++++++++++++++------ 3 files changed, 141 insertions(+), 13 deletions(-) create mode 100644 arch/x86/net/bpf_cleanup_pad.S
arch/x86/net/Makefile +1 -1
@@ -6,5 +6,5 @@ ifeq ($(CONFIG_X86_32),y) obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o else - obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o + obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o endif
arch/x86/net/bpf_cleanup_pad.S new file +74 -0
74 lines not on this page · raw message
arch/x86/net/bpf_jit_comp.c +66 -12
170 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
Re: [PATCH bpf-next 11/20] bpf, x86: Dispatch exception cleanup pads at run time
Quoted 4 lines · Yonghong Song
arch/x86/net/bpf_jit_comp.c +0
-0
[...]
arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · line 2947
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +5
-0+if (bpf_is_unwind_resume_kfunc (insn)) { +emit_return (&prog, image + addrs[i -1 ] + (prog - temp)); +break ; + } + func = (u8 *) __bpf_call_base + imm32;
bpf_is_unwind_resume_kfunc() matches imm == btf_id && off == 0, but by now bpf_fixup_kfunc_call() has rewritten every other kfunc's imm to an offset from __bpf_call_base. A vmlinux kfunc that happens to sit btf_id bytes after __bpf_call_base gets JITed as a bare ret in the middle of a normal prog. Looks incorrect.
Yonghong Song
reviewerRe: [PATCH bpf-next 11/20] bpf, x86: Dispatch exception cleanup pads at run time
Quoted 7 lines · Alexei Starovoitov
Quoted 4 lines · Yonghong Song
arch/x86/net/bpf_jit_comp.c +0
-0[...]
Quoted 11 lines
arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · line 2947
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +5
-0+if (bpf_is_unwind_resume_kfunc (insn)) { +emit_return (&prog, image + addrs[i -1 ] + (prog - temp)); +break ; + } + func = (u8 *) __bpf_call_base + imm32;
bpf_is_unwind_resume_kfunc()matches imm ==btf_id&& off == 0, but by nowbpf_fixup_kfunc_call()has rewritten every other kfunc's imm to an offset from__bpf_call_base. A vmlinux kfunc that happens to sitbtf_idbytes after__bpf_call_basegets JITed as a bare ret in the middle of a normal prog. Looks incorrect.
Right, bpf_is_unwind_resume_kfunc() is not correct as it compares 'imm == bpf_id' while imm has been rewritten.
Will fix.
Yonghong Song
reviewerPATCH 12/20 bpf, arm64: Dispatch exception cleanup pads at run time
In reply to Yonghong SongThe same arch half for arm64: force the full callee-saved spill, record where it starts, build the native table from the JIT's byte offsets, and hand control to a pad from arch_bpf_run_cleanup_pad().
Unlike x86-64 a pad cannot simply return: every call it makes clobbers x30, so no return address survives to its resume. x23 carries it instead. bpf2a64[] maps nothing to x23 or x24 -- they are pushed only to keep the frame shape the exception callback expects -- so nothing else in generated code touches them, and being callee-saved they survive every kfunc the pad calls. The JIT emits "br x23" for a pad's bpf_unwind_resume().
x24 is the other one, and it is what makes a pad able to touch its own frame at all. Generated code addresses the BPF frame through the stack pointer -- the frame sits directly on top of it, which turns every offset positive and each access into one instruction -- and in a pad the stack pointer is the walker's. So the JIT has a pad recompute the equivalent of its frame's stack pointer from BPF r10 on entry, into x24, and addresses the frame off x24 for every instruction the previous patches marked as running only while unwinding. One instruction per pad, and the accesses keep the shape and the immediate range they have everywhere else.
--- arch/arm64/net/Makefile | 2 +- arch/arm64/net/bpf_cleanup_pad.S | 95 +++++++++++++++++++++++++++++++ arch/arm64/net/bpf_jit_comp.c | 97 ++++++++++++++++++++++++++++++-- 3 files changed, 187 insertions(+), 7 deletions(-) create mode 100644 arch/arm64/net/bpf_cleanup_pad.S
arch/arm64/net/Makefile +1 -1
@@ -2,4 +2,4 @@ # # ARM64 networking code # -obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o +obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o
arch/arm64/net/bpf_cleanup_pad.S new file +95 -0
95 lines not on this page · raw message
arch/arm64/net/bpf_jit_comp.c +91 -6
196 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 13/20 libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
In reply to Yonghong SongLLVM terminates a cleanup landing pad with a call to _Unwind_Resume: the base unwind ABI's entry point for carrying an unwind on once a frame's cleanups have run. The kernel provides that terminator as a kfunc, but not under that name: claiming _Unwind_Resume in the kernel's own symbol table, for a function whose body never runs, would be needlessly confusing, so it is called bpf_unwind_resume.
--- tools/lib/bpf/libbpf.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.c +14 -3
@@ -8346,6 +8346,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s } } +/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the +* base unwind ABI's entry point for carrying an unwind on once a frame's +* cleanups have run. The kernel knows it as bpf_unwind_resume. +*/ +static const char *kern_extern_name (const char *name) +{ +return strcmp (name,"_Unwind_Resume" ) ? name :"bpf_unwind_resume" ; +} +static int bpf_program_record_relos (struct bpf_program *prog) {struct bpf_object *obj = prog->obj;
@@ -8807,17 +8816,19 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,struct module_btf *mod_btf =NULL ;const struct btf_type *kern_func;struct btf *kern_btf =NULL ; +const char *kern_name;int ret; local_func_proto_id = ext->ksym.type_id; - kfunc_id =find_ksym_btf_id (obj, ext->essent_name ?: ext->name,BTF_KIND_FUNC , &kern_btf, - &mod_btf); + kern_name =kern_extern_name (ext->essent_name ?: ext->name); + + kfunc_id =find_ksym_btf_id (obj, kern_name,BTF_KIND_FUNC , &kern_btf, &mod_btf);if (kfunc_id <0 ) {if (kfunc_id == -ESRCH && ext->is_weak)return 0 ;pr_warn ("extern (func ksym) '%s': not found in kernel or module BTFs\n" , - ext->name); +strcmp (kern_name,"bpf_unwind_resume" ) ? ext->name : kern_name);return kfunc_id; }
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 13/20] libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] bpf: multi-line comment violates BPF subsystem style guide --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: multi-line comment violates BPF subsystem style guide
--
commit 082b791c15a0d9341a2e4f4d61c9b916d48b95fc
Author: Yonghong Song <[email protected]>
libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
This commit maps LLVM's _Unwind_Resume function calls in cleanup landing pads to the kernel's corresponding bpf_unwind_resume kfunc. This ensures proper symbol resolution without polluting the kernel's symbol table.
tools/lib/bpf/libbpf.c static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s · line 8352
Quoted 12 lines
tools/lib/bpf/libbpf.c +4
-0@@ -8346,6 +8346,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s } } +/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the +* base unwind ABI's entry point for carrying an unwind on once a frame's +* cleanups have run. The kernel knows it as bpf_unwind_resume. +*/
[Severity: Low]
This isn't a bug, but does this multi-line comment format violate the BPF subsystem style guide? The guide specifies that multi-line comments must have the opening /* on its own line, rather than starting the text on the same line as the opening characters.
Quoted 4 lines
tools/lib/bpf/libbpf.c +4
-0+static const char *kern_extern_name (const char *name) +{ +return strcmp (name,"_Unwind_Resume" ) ? name :"bpf_unwind_resume" ; +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
[email protected]
AI reviewRe: [PATCH bpf-next 13/20] libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
In reply to Yonghong SongIs the kern_extern_name() mapping applied on the light skeleton load path?
Quoted 10 lines
lib/bpf/libbpf.c +2
-0@@ -8346,6 +8346,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s } } +/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the +* base unwind ABI's entry point for carrying an unwind on once a frame's
lib/bpf/libbpf.c static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s · line 8360
Quoted 10 lines
lib/bpf/libbpf.c +7
-0+* cleanups have run. The kernel knows it as bpf_unwind_resume. +*/ +static const char *kern_extern_name (const char *name) +{ +return strcmp (name,"_Unwind_Resume" ) ? name :"bpf_unwind_resume" ; +} +static int bpf_program_record_relos (struct bpf_program *prog) {struct bpf_object *obj = prog->obj;
Is the kern_extern_name() mapping applied on the light skeleton load path?
Looking at bpf_program_record_relos() below kern_extern_name(), the RELO_EXTERN_CALL case records ext->name directly without calling kern_extern_name():
static int bpf_program_record_relos(struct bpf_program *prog)
{
...
case RELO_EXTERN_CALL:
bpf_gen__record_extern(obj->gen_loader, ext->name,
ext->is_weak, false, false, BTF_KIND_FUNC,
relo->insn_idx);
break;
...
}This means the generated loader program would ask the kernel for BTF_KIND_FUNC '_Unwind_Resume', which doesn't exist in vmlinux BTF. The kernel only knows bpf_unwind_resume per BTF_ID_FLAGS(func, bpf_unwind_resume) in kernel/bpf/helpers.c.
For comparison, the direct BPF_PROG_LOAD path does use the mapping:
bpf_object__resolve_ksym_func_btf_id() calls kern_extern_name() at libbpf.c:8824, but that resolution is deliberately skipped when obj->gen_loader is set:
bpf_object__resolve_ksyms_btf_id() {
...
if (obj->gen_loader) {
ext->is_set = true;
ext->ksym.kernel_btf_obj_fd = 0;
ext->ksym.kernel_btf_id = 0;
continue;
}
...
}On the light skeleton path, bpf_btf_find_by_name_kind() would return -ESRCH for '_Unwind_Resume', causing emit_check_err() to jump to cleanup and fail the load. If the extern were weak, insn->imm/off would be stored as 0, which the verifier would reject as an invalid kfunc call.
The test added later in the series (selftests/bpf/progs/exceptions_cleanup_light.c) writes 'call bpf_unwind_resume' directly in inline asm, so it uses the kernel name and never exercises the mapping.
Quoted 12 lines
lib/bpf/libbpf.c +2
-2@@ -8807,17 +8816,19 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,struct module_btf *mod_btf =NULL ;const struct btf_type *kern_func;struct btf *kern_btf =NULL ; +const char *kern_name;int ret; local_func_proto_id = ext->ksym.type_id; - kfunc_id =find_ksym_btf_id (obj, ext->essent_name ?: ext->name,BTF_KIND_FUNC , &kern_btf, - &mod_btf); + kern_name =kern_extern_name (ext->essent_name ?: ext->name);
lib/bpf/libbpf.c static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj, · line 8833
Quoted 10 lines
lib/bpf/libbpf.c +3
-1+ + kfunc_id =find_ksym_btf_id (obj, kern_name,BTF_KIND_FUNC , &kern_btf, &mod_btf);if (kfunc_id <0 ) {if (kfunc_id == -ESRCH && ext->is_weak)return 0 ;pr_warn ("extern (func ksym) '%s': not found in kernel or module BTFs\n" , - ext->name); +strcmp (kern_name,"bpf_unwind_resume" ) ? ext->name : kern_name);return kfunc_id; }
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35188619605
Yonghong Song
reviewerRe: [PATCH bpf-next 13/20] libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
Quoted 65 lines · On 9/17/26 12:00 AM, [email protected] wrote:
Quoted 10 lines
lib/bpf/libbpf.c +2
-0@@ -8346,6 +8346,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s } } +/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the +* base unwind ABI's entry point for carrying an unwind on once a frame's lib/bpf/libbpf.c static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s · line 8360
Quoted 10 lines
lib/bpf/libbpf.c +7
-0+* cleanups have run. The kernel knows it as bpf_unwind_resume. +*/ +static const char *kern_extern_name (const char *name) +{ +return strcmp (name,"_Unwind_Resume" ) ? name :"bpf_unwind_resume" ; +} +static int bpf_program_record_relos (struct bpf_program *prog) {struct bpf_object *obj = prog->obj;Is the
kern_extern_name()mapping applied on the light skeleton load path?Looking at
bpf_program_record_relos()belowkern_extern_name(), theRELO_EXTERN_CALLcase recordsext->namedirectly without callingkern_extern_name():static int
bpf_program_record_relos(struct bpf_program *prog){ ... case RELO_EXTERN_CALL: bpf_gen__record_extern(obj->gen_loader, ext->name, ext->is_weak, false, false, BTF_KIND_FUNC, relo->insn_idx); break; ... }This means the generated loader program would ask the kernel for
BTF_KIND_FUNC'_Unwind_Resume', which doesn't exist in vmlinux BTF. The kernel only knowsbpf_unwind_resumeperBTF_ID_FLAGS(func, bpf_unwind_resume)inkernel/bpf/helpers.c.For comparison, the direct
BPF_PROG_LOADpath does use the mapping:
bpf_object__resolve_ksym_func_btf_id()callskern_extern_name()at libbpf.c:8824, but that resolution is deliberately skipped whenobj->gen_loaderis set:bpf_object__resolve_ksyms_btf_id() { ... if (obj->gen_loader) { ext->is_set = true; ext->ksym.kernel_btf_obj_fd = 0; ext->ksym.kernel_btf_id = 0; continue; } ... }On the light skeleton path,
bpf_btf_find_by_name_kind()would return-ESRCHfor '_Unwind_Resume', causingemit_check_err()to jump to cleanup and fail the load. If the extern were weak, insn->imm/off would be stored as 0, which the verifier would reject as an invalid kfunc call.
Quoted 4 lines
The test added later in the series (
selftests/bpf/progs/exceptions_cleanup_light.c) writes 'callbpf_unwind_resume' directly in inline asm, so it uses the kernel name and never exercises the mapping.
Indeed, later patch in selftests/bpf/progs/exceptions_cleanup_light.c fixed the issue with bpf_undiwn_resume properly. So this is not really a bug. But we can do better than this. We can change from ext->name to bpf_unwind_resume in file selftests/bpf/progs/exceptions_cleanup_light.c for this patch. This will make things easier to understand.
Quoted 29 lines
Quoted 13 lines
lib/bpf/libbpf.c +2
-2@@ -8807,17 +8816,19 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,struct module_btf *mod_btf =NULL ;const struct btf_type *kern_func;struct btf *kern_btf =NULL ; +const char *kern_name;int ret; local_func_proto_id = ext->ksym.type_id; - kfunc_id =find_ksym_btf_id (obj, ext->essent_name ?: ext->name,BTF_KIND_FUNC , &kern_btf, - &mod_btf); + kern_name =kern_extern_name (ext->essent_name ?: ext->name);lib/bpf/libbpf.c static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj, · line 8833
Quoted 10 lines
lib/bpf/libbpf.c +3
-1+ + kfunc_id =find_ksym_btf_id (obj, kern_name,BTF_KIND_FUNC , &kern_btf, &mod_btf);if (kfunc_id <0 ) {if (kfunc_id == -ESRCH && ext->is_weak)return 0 ;pr_warn ("extern (func ksym) '%s': not found in kernel or module BTFs\n" , - ext->name); +strcmp (kern_name,"bpf_unwind_resume" ) ? ext->name : kern_name);return kfunc_id; }---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35188619605
Yonghong Song
reviewerPATCH 14/20 libbpf: Add cleanup_info to bpf_prog_load_opts
In reply to Yonghong SongLet a caller hand the kernel an exception cleanup table. Extend struct bpf_prog_load_opts with cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, pass them through to BPF_PROG_LOAD, and grow the attr size bpf_prog_load() computes to cover the new fields.
--- tools/lib/bpf/bpf.c | 6 +++++- tools/lib/bpf/bpf.h | 7 ++++++- 2 files changed, 11 insertions(+), 2 deletions(-)
tools/lib/bpf/bpf.c +5 -1
@@ -295,7 +295,7 @@ int bpf_prog_load(enum bpf_prog_type prog_type, const struct bpf_insn *insns, size_t insn_cnt, struct bpf_prog_load_opts *opts) { - const size_t attr_sz = offsetofend(union bpf_attr, keyring_id); + const size_t attr_sz = offsetofend(union bpf_attr, cleanup_info_cnt);void *finfo =NULL , *linfo =NULL ;const char *func_info, *line_info;__u32 log_size, log_level, attach_prog_fd, attach_btf_obj_fd;
@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type, attr.fd_array =ptr_to_u64 (OPTS_GET (opts, fd_array,NULL )); attr.fd_array_cnt =OPTS_GET (opts, fd_array_cnt,0 ); + attr.cleanup_info =ptr_to_u64 (OPTS_GET (opts, cleanup_info,NULL )); + attr.cleanup_info_cnt =OPTS_GET (opts, cleanup_info_cnt,0 ); + attr.cleanup_info_rec_size =OPTS_GET (opts, cleanup_info_rec_size,0 ); +if (log_level) { attr.log_buf =ptr_to_u64 (log_buf); attr.log_size = log_size;
tools/lib/bpf/bpf.h +6 -1
@@ -128,9 +128,14 @@ struct bpf_prog_load_opts {/* if set, provides the length of fd_array */ __u32 fd_array_cnt; + +/* exception cleanup table, from the .bpf_cleanup section */ +const void *cleanup_info; +__u32 cleanup_info_cnt; +__u32 cleanup_info_rec_size; size_t :0; }; -#define bpf_prog_load_opts__last_field fd_array_cnt +#define bpf_prog_load_opts__last_field cleanup_info_rec_size LIBBPF_API int bpf_prog_load(enum bpf_prog_type prog_type, const char *prog_name, const char *license,
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH 15/20 libbpf: Collect .bpf_cleanup records and pass them to the kernel
In reply to Yonghong SongParse the compiler-emitted .bpf_cleanup section and hand the resulting table to BPF_PROG_LOAD.
Each record is three 4-byte fields, and each field is a byte offset into some code section named by a matching .rel.bpf_cleanup relocation. bpf_object__init_cleanup_info() resolves both halves once at open time and keeps (section, instruction index) pairs; it rejects a record whose field has no relocation, whose relocation is not one of the two 32-bit types that spell a data reference to a code section -- R_BPF_64_NODYLD32 from LLVM, R_BPF_64_ABS32 from GNU as -- or whose offset is not instruction aligned.
The records are sorted by begin_off once the offsets are final: the kernel wants the table sorted with disjoint ranges so that it can find the record covering a call site with a binary search, and records arrive in .bpf_cleanup order, which says nothing about where the subprograms they describe were appended. Overlapping ranges are reported here, where the program name and both regions are still at hand.
bpf_object_load_prog() then passes the per-program table through the bpf_prog_load() options added in the previous patch, with the record size carried on the program the way func_info and line_info carry theirs rather than taken from a sizeof() at the call site. bpf_program__clone() carries it too. That is a second load path -- the one veristat uses -- and without the table the kernel sees landing pads nothing reaches and refuses the program with "unreachable insn".
--- tools/lib/bpf/libbpf.c | 284 ++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf_internal.h | 3 + 2 files changed, 287 insertions(+)
tools/lib/bpf/libbpf.c +284 -0
@@ -514,6 +514,11 @@ struct bpf_program {void *line_info;__u32 line_info_rec_size;__u32 line_info_cnt; + +struct bpf_cleanup_info *cleanup_info; +__u32 cleanup_info_rec_size; +__u32 cleanup_info_cnt; +__u32 prog_flags;__u8 hash[SHA256_DIGEST_LENGTH ];
@@ -549,6 +554,7 @@ struct bpf_struct_ops {#define STRUCT_OPS_SEC ".struct_ops"#define STRUCT_OPS_LINK_SEC ".struct_ops.link"#define ARENA_SEC ".addr_space.1" +#define CLEANUP_SEC ".bpf_cleanup"enum libbpf_map_type {LIBBPF_MAP_UNSPEC ,
@@ -677,6 +683,25 @@ struct elf_sec_desc {Elf_Data *data; }; +#define CLEANUP_REC_FIELDS (sizeof(struct bpf_cleanup_info) / sizeof(__u32)) + +/* Index of each field of struct bpf_cleanup_info, read as an array of __u32. */ +enum { +CLEANUP_REC_BEGIN , +CLEANUP_REC_END , +CLEANUP_REC_PAD , +}; + +/* One (begin, end, landing_pad) triple from .bpf_cleanup, with each field +* resolved from its relocation to an ELF section plus a section-relative +* instruction index. The mapping to final program instruction indices can only +* happen after subprogram placement, which differs per main program. +*/ +struct cleanup_raw_rec { +int sec_idx[CLEANUP_REC_FIELDS ]; +size_t insn_idx[CLEANUP_REC_FIELDS ]; +}; +struct elf_state {int fd;const void *obj_buf;
@@ -696,6 +721,8 @@ struct elf_state {bool has_st_ops;int arena_data_shndx;int jumptables_data_shndx; +Elf_Data *cleanup_data; +int cleanup_shndx; };struct usdt_manager ;
@@ -771,6 +798,9 @@ struct bpf_object {void *jumptables_data;size_t jumptables_data_sz; +struct cleanup_raw_rec *cleanup_recs; +size_t cleanup_rec_cnt; +struct {struct bpf_program *prog;unsigned int sym_off;
@@ -817,7 +847,10 @@ static void bpf_program__exit(struct bpf_program *prog)zfree (&prog->sec_name);zfree (&prog->insns);zfree (&prog->reloc_desc); +zfree (&prog->cleanup_info); + prog->cleanup_info_rec_size =0 ; + prog->cleanup_info_cnt =0 ; prog->nr_reloc =0 ; prog->insns_cnt =0 ; prog->sec_idx =-1 ;
@@ -1554,6 +1587,7 @@ static struct bpf_object *bpf_object__new(const char *path, obj->efile.obj_buf = obj_buf; obj->efile.obj_buf_sz = obj_buf_sz; obj->efile.btf_maps_shndx =-1 ; + obj->efile.cleanup_shndx =-1 ; obj->kconfig_map_idx =-1 ; obj->arena_map_idx =-1 ;
@@ -4040,6 +4074,9 @@ static int bpf_object__elf_collect(struct bpf_object *obj) sec_desc->shdr = sh; sec_desc->data = data; obj->efile.has_st_ops = true; + }else if (strcmp (name,CLEANUP_SEC ) ==0 ) { + obj->efile.cleanup_data = data; + obj->efile.cleanup_shndx = idx; }else if (strcmp (name,ARENA_SEC ) ==0 ) { obj->efile.arena_data = data; obj->efile.arena_data_shndx = idx;
@@ -4067,6 +4104,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj) strcmp(name, ".rel" STRUCT_OPS_LINK_SEC) && strcmp(name, ".rel?" STRUCT_OPS_SEC) && strcmp(name, ".rel?" STRUCT_OPS_LINK_SEC) && + strcmp(name, ".rel" CLEANUP_SEC) && strcmp(name, ".rel" MAPS_ELF_SEC)) { pr_info("elf: skipping relo section(%d) %s for section(%d) %s\n", idx, name, targ_sec_idx,
@@ -4847,6 +4885,214 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,return NULL ; } +static int bpf_object__init_cleanup_info (struct bpf_object *obj) +{ +Elf_Data *data = obj->efile.cleanup_data; +Elf_Data *relo =NULL ; +size_t i, nrels, nslots, nrecs; +struct cleanup_raw_rec *recs; +int *slot_sec, ret =0 ; +size_t *slot_val; +const __u32 *vals; +bool native; + +if (!data || obj->efile.cleanup_shndx <0 ) +return 0 ; + + native =is_native_endianness (obj); + +for (i =0 ; i < obj->efile.sec_cnt; i++) { +struct elf_sec_desc *sd = &obj->efile.secs[i]; + +if (sd->sec_type ==SEC_RELO && sd->shdr && + sd->shdr->sh_info == (Elf64_Word )obj->efile.cleanup_shndx) { + relo = sd->data; +break ; + } + } +if (!relo) { +pr_warn ("%s present without relocations\n" ,CLEANUP_SEC ); +return -LIBBPF_ERRNO__FORMAT ; + } +if (data->d_size %sizeof (struct bpf_cleanup_info )) { +pr_warn ("%s size %zu is not a multiple of the record size %zu\n" , +CLEANUP_SEC , data->d_size,sizeof (struct bpf_cleanup_info )); +return -LIBBPF_ERRNO__FORMAT ; + } + + vals = data->d_buf; + nslots = data->d_size /sizeof (__u32); + nrecs = data->d_size /sizeof (struct bpf_cleanup_info ); + + slot_sec =calloc (nslots,sizeof (*slot_sec)); + slot_val =calloc (nslots,sizeof (*slot_val)); + recs =calloc (nrecs ?:1 ,sizeof (*recs)); +if (!slot_sec || !slot_val || !recs) { + ret = -ENOMEM ; + goto out; + } +for (i =0 ; i < nslots; i++) + slot_sec[i] =-1 ; + +/* One relocation per 4-byte field, naming the section it points into. */ + nrels = relo->d_size /sizeof (Elf64_Rel); +for (i =0 ; i < nrels; i++) { +Elf64_Rel *rel =elf_rel_by_idx (relo, i); +Elf64_Sym *sym =elf_sym_by_idx (obj,ELF64_R_SYM (rel->r_info)); +size_t type =ELF64_R_TYPE (rel->r_info); +size_t slot = rel->r_offset /sizeof (__u32); + +if (type !=R_BPF_64_NODYLD32 && type !=R_BPF_64_ABS32 ) { +pr_warn ("%s: relocation %zu has unexpected type %zu\n" , +CLEANUP_SEC , i, type); + ret = -LIBBPF_ERRNO__FORMAT ; + goto out; + } +if (!sym || slot >= nslots || rel->r_offset %sizeof (__u32)) { +pr_warn ("%s: bad relocation %zu\n" ,CLEANUP_SEC , i); + ret = -LIBBPF_ERRNO__FORMAT ; + goto out; + } + slot_sec[slot] = sym->st_shndx; +/* The addend lives in the section data, which libelf leaves in +* the object's byte order; a non-section symbol additionally +* contributes its own value. +*/ + slot_val[slot] = (native ? vals[slot] :bswap_32 (vals[slot])) + + sym->st_value; + } + +for (i =0 ; i < nslots; i++) { +struct cleanup_raw_rec *rec = &recs[i /CLEANUP_REC_FIELDS ]; +size_t field = i %CLEANUP_REC_FIELDS ; + +if (slot_sec[i] <0 ) { +pr_warn ("%s: field %zu has no relocation\n" ,CLEANUP_SEC , i); + ret = -LIBBPF_ERRNO__FORMAT ; + goto out; + } +if (slot_val[i] %BPF_INSN_SZ ) { +pr_warn ("%s: field %zu offset %zu is not instruction aligned\n" , +CLEANUP_SEC , i, slot_val[i]); + ret = -LIBBPF_ERRNO__FORMAT ; + goto out; + } + rec->sec_idx[field] = slot_sec[i]; + rec->insn_idx[field] = slot_val[i] /BPF_INSN_SZ ; + } + + obj->cleanup_recs = recs; + obj->cleanup_rec_cnt = nrecs; + recs =NULL ; +out: +free (recs); +free (slot_val); +free (slot_sec); +return ret; +} + +static int cmp_cleanup_info (const void *a,const void *b) +{ +const struct bpf_cleanup_info *x = a, *y = b; + +if (x->begin_off == y->begin_off) +return 0 ; +return x->begin_off < y->begin_off ?-1 :1 ; +} + +static int bpf_prog_collect_cleanup_info (struct bpf_object *obj, +struct bpf_program *prog) +{ +size_t i; +int j; + +for (i =0 ; i < obj->cleanup_rec_cnt; i++) { +struct cleanup_raw_rec *raw = &obj->cleanup_recs[i]; +struct bpf_program *owner =NULL ; +struct bpf_cleanup_info ci = {}; +__u32 *fields = (__u32 *)&ci; +void *tmp; + +for (j =0 ; j <CLEANUP_REC_FIELDS ; j++) { +size_t idx = raw->insn_idx[j], final; +struct bpf_program *p; + +/* The end of a range is exclusive, so it may name the +* instruction just past the last one of a function, +* which belongs to the next function or to nothing at +* all. Ask about the last instruction the range covers, +* the way the kernel does. +*/ +if (j ==CLEANUP_REC_END ) { +if (!idx) { +pr_warn ("%s: record %zu is an empty range\n" , +CLEANUP_SEC , i); +return -LIBBPF_ERRNO__FORMAT ; + } + idx--; + } + + p =find_prog_by_sec_insn (obj, raw->sec_idx[j], idx); +if (!p) { +pr_warn ("%s: record %zu field %d is not inside a function\n" , +CLEANUP_SEC , i, j); +return -LIBBPF_ERRNO__FORMAT ; + } +if (!owner) { + owner = p; + }else if (owner != p) { +pr_warn ("%s: record %zu spans functions '%s' and '%s'\n" , +CLEANUP_SEC , i, owner->name, p->name); +return -LIBBPF_ERRNO__FORMAT ; + } + +if (owner == prog) { + final = raw->insn_idx[j] - prog->sec_insn_off; + }else if (prog_is_subprog (obj, owner) && owner->sub_insn_off) { +/* sub_insn_off is where this subprogram was +* appended to the main program being relocated; +* zero means it is not part of it. +*/ + final = owner->sub_insn_off + + raw->insn_idx[j] - owner->sec_insn_off; + }else { + owner =NULL ; +break ; + } + fields[j] = final; + } +if (!owner) +continue ; + + tmp =libbpf_reallocarray (prog->cleanup_info, prog->cleanup_info_cnt +1 , +sizeof (*prog->cleanup_info)); +if (!tmp) +return -ENOMEM ; + prog->cleanup_info = tmp; + prog->cleanup_info_rec_size =sizeof (struct bpf_cleanup_info ); + prog->cleanup_info[prog->cleanup_info_cnt++] = ci; + +pr_debug ("prog '%s': cleanup region [%u,%u) -> landing pad %u\n" , + prog->name, ci.begin_off, ci.end_off, ci.landing_pad_off); + } + +qsort (prog->cleanup_info, prog->cleanup_info_cnt, +sizeof (*prog->cleanup_info), cmp_cleanup_info); +for (i =1 ; i < prog->cleanup_info_cnt; i++) { +struct bpf_cleanup_info *prev = &prog->cleanup_info[i -1 ]; +struct bpf_cleanup_info *cur = &prog->cleanup_info[i]; + +if (cur->begin_off < prev->end_off) { +pr_warn ("prog '%s': overlapping cleanup regions [%u,%u) and [%u,%u)\n" , + prog->name, prev->begin_off, prev->end_off, + cur->begin_off, cur->end_off); +return -LIBBPF_ERRNO__FORMAT ; + } + } + +return 0 ; +} +static int bpf_object__collect_prog_relos (struct bpf_object *obj,Elf64_Shdr *shdr,Elf_Data *data) {
@@ -7556,6 +7802,13 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_patreturn err; } } + + err =bpf_prog_collect_cleanup_info (obj, prog); +if (err) { +pr_warn ("prog '%s': failed to collect cleanup info: %s\n" , + prog->name,errstr (err)); +return err; + } }for (i =0 ; i < obj->nr_programs; i++) { prog = &obj->programs[i];
@@ -7746,6 +7999,9 @@ static int bpf_object__collect_relos(struct bpf_object *obj)return -LIBBPF_ERRNO__INTERNAL ; } +if (idx == obj->efile.cleanup_shndx) +continue ; +if (obj->efile.secs[idx].sec_type ==SEC_ST_OPS ) err =bpf_object__collect_st_ops_relos (obj, shdr, data);else if (idx == obj->efile.btf_maps_shndx)
@@ -8018,6 +8274,11 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog load_attr.line_info_rec_size = prog->line_info_rec_size; load_attr.line_info_cnt = prog->line_info_cnt; } +if (prog->cleanup_info_cnt) { + load_attr.cleanup_info = prog->cleanup_info; + load_attr.cleanup_info_cnt = prog->cleanup_info_cnt; + load_attr.cleanup_info_rec_size = prog->cleanup_info_rec_size; + } load_attr.log_level = log_level; load_attr.prog_flags = prog->prog_flags; load_attr.fd_array = obj->fd_array;
@@ -8574,6 +8835,7 @@ static struct bpf_object *bpf_object_open(const char *path, const void *obj_buf, err = err ? :bpf_object__init_maps (obj, opts); err = err ? :bpf_object_init_progs (obj, opts); err = err ? :bpf_object__collect_relos (obj); + err = err ? :bpf_object__init_cleanup_info (obj);if (err) goto out;
@@ -9687,6 +9949,9 @@ void bpf_object__close(struct bpf_object *obj)zfree (&obj->jumptables_data); obj->jumptables_data_sz =0 ; +zfree (&obj->cleanup_recs); + obj->cleanup_rec_cnt =0 ; +for (i =0 ; i < obj->jumptable_map_cnt; i++)close (obj->jumptable_maps[i].fd);zfree (&obj->jumptable_maps);
@@ -10079,6 +10344,25 @@ int bpf_program__clone(struct bpf_program *prog, const struct bpf_prog_load_opts attr.line_info_rec_size = info ? info_rec_size : prog->line_info_rec_size; } +/* exception cleanup table */ + info =OPTS_GET (opts, cleanup_info,NULL ); + info_cnt =OPTS_GET (opts, cleanup_info_cnt,0 ); + info_rec_size =OPTS_GET (opts, cleanup_info_rec_size,0 ); +if (!!info != !!info_cnt || !!info != !!info_rec_size) { +pr_warn ("prog '%s': cleanup_info, cleanup_info_cnt, and cleanup_info_rec_size must all be specified or all omitted\n" , + prog->name); +return libbpf_err (-EINVAL ); + } +if (info) { + attr.cleanup_info = info; + attr.cleanup_info_cnt = info_cnt; + attr.cleanup_info_rec_size = info_rec_size; + }else if (prog->cleanup_info_cnt) { + attr.cleanup_info = prog->cleanup_info; + attr.cleanup_info_cnt = prog->cleanup_info_cnt; + attr.cleanup_info_rec_size = prog->cleanup_info_rec_size; + } +/* Logging is caller-controlled; no fallback to prog/obj log settings */ attr.log_buf =OPTS_GET (opts, log_buf,NULL ); attr.log_size =OPTS_GET (opts, log_size,0 );
tools/lib/bpf/libbpf_internal.h +3 -0
@@ -56,6 +56,9 @@#ifndef R_BPF_64_ABS32 #define R_BPF_64_ABS32 3#endif +#ifndef R_BPF_64_NODYLD32 +#define R_BPF_64_NODYLD32 4 +#endif #ifndef R_BPF_64_32 #define R_BPF_64_32 10#endif
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 15/20] libbpf: Collect .bpf_cleanup records and pass them to the kernel
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] bpf: undefined behavior when calling qsort with a NULL pointer - [Low] bpf: array bounds…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] bpf: undefined behavior when calling qsort with a NULL pointer
- [Low] bpf: array bounds violation via type-punned pointer arithmetic
--
commit 1e591245d7ec663b872eae44c8fa7f02cfd956a4
Author: Yonghong Song <[email protected]>
libbpf: Collect .bpf_cleanup records and pass them to the kernel
This commit parses the compiler-emitted .bpf_cleanup section and hands the resulting table to BPF_PROG_LOAD so the kernel can resolve exceptions.
Quoted 4 lines
tools/lib/bpf/libbpf.c +0
-0
[ ... ]
Quoted 1 line
tools/lib/bpf/libbpf.c +0
-0@@ -4847,6 +4885,214 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,
[ ... ]
Quoted 16 lines
tools/lib/bpf/libbpf.c +16
-0+static int bpf_prog_collect_cleanup_info (struct bpf_object *obj, +struct bpf_program *prog) +{ +size_t i; +int j; + +for (i =0 ; i < obj->cleanup_rec_cnt; i++) { +struct cleanup_raw_rec *raw = &obj->cleanup_recs[i]; +struct bpf_program *owner =NULL ; +struct bpf_cleanup_info ci = {}; +__u32 *fields = (__u32 *)&ci; +void *tmp; + +for (j =0 ; j <CLEANUP_REC_FIELDS ; j++) { +size_t idx = raw->insn_idx[j], final; +struct bpf_program *p;
[ ... ]
Quoted 4 lines
tools/lib/bpf/libbpf.c +4
-0+if (owner == prog) { + final = raw->insn_idx[j] - prog->sec_insn_off; + }else if (prog_is_subprog (obj, owner) && owner->sub_insn_off) { +/* sub_insn_off is where this subprogram was
tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 5062
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+* appended to the main program being relocated; +* zero means it is not part of it. +*/ + final = owner->sub_insn_off + + raw->insn_idx[j] - owner->sec_insn_off; + }else { + owner =NULL ; +break ; + } + fields[j] = final;
[Severity: Low]
Is it safe to cast a pointer to a struct field into an array pointer and access subsequent members via pointer arithmetic in this loop?
The struct bpf_cleanup_info fields are populated by casting a pointer to the struct to a __u32 * array and indexing into it in a loop.
According to C11 6.5.6, pointer arithmetic on a pointer to a scalar (the first struct field) only allows accessing up to one element past it, making fields[1] and fields[2] undefined behavior.
This can trigger compiler warnings (-Warray-bounds) and potentially cause aggressive optimizers to generate incorrect code.
Quoted 8 lines
tools/lib/bpf/libbpf.c +8
-0+ } +if (!owner) +continue ; + + tmp =libbpf_reallocarray (prog->cleanup_info, prog->cleanup_info_cnt +1 , +sizeof (*prog->cleanup_info)); +if (!tmp) +return -ENOMEM ;
tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 5080
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+ prog->cleanup_info = tmp; + prog->cleanup_info_rec_size =sizeof (struct bpf_cleanup_info ); + prog->cleanup_info[prog->cleanup_info_cnt++] = ci; + +pr_debug ("prog '%s': cleanup region [%u,%u) -> landing pad %u\n" , + prog->name, ci.begin_off, ci.end_off, ci.landing_pad_off); + } + +qsort (prog->cleanup_info, prog->cleanup_info_cnt, +sizeof (*prog->cleanup_info), cmp_cleanup_info);
[Severity: Medium]
Can this unconditionally call qsort() with a NULL base pointer?
If obj->cleanup_rec_cnt is 0, or if none of the records match the program, prog->cleanup_info_cnt remains 0 and prog->cleanup_info will be NULL.
Calling qsort() with a NULL pointer invokes undefined behavior under C11 7.1.4, even if the element count is 0. This triggers UBSAN errors (nonnull attribute violation in libc headers) and can fail runtime checks.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
Yonghong Song
reviewerRe: [PATCH bpf-next 15/20] libbpf: Collect .bpf_cleanup records and pass them to the kernel
Quoted 61 lines · On 9/16/26 11:12 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] bpf: undefined behavior when calling qsort with a NULL pointer
- [Low] bpf: array bounds violation via type-punned pointer arithmetic
--commit
1e591245d7ec663b872eae44c8fa7f02cfd956a4
Author: Yonghong Song <[email protected]>libbpf: Collect .bpf_cleanup records and pass them to the kernel
This commit parses the compiler-emitted .bpf_cleanup section and hands the resulting table to
BPF_PROG_LOADso the kernel can resolve exceptions.Quoted 4 lines
tools/lib/bpf/libbpf.c +0
-0[ ... ]
Quoted 1 line
tools/lib/bpf/libbpf.c +0
-0@@ -4847,6 +4885,214 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,[ ... ]
Quoted 16 lines
tools/lib/bpf/libbpf.c +16
-0+static int bpf_prog_collect_cleanup_info (struct bpf_object *obj, +struct bpf_program *prog) +{ +size_t i; +int j; + +for (i =0 ; i < obj->cleanup_rec_cnt; i++) { +struct cleanup_raw_rec *raw = &obj->cleanup_recs[i]; +struct bpf_program *owner =NULL ; +struct bpf_cleanup_info ci = {}; +__u32 *fields = (__u32 *)&ci; +void *tmp; + +for (j =0 ; j <CLEANUP_REC_FIELDS ; j++) { +size_t idx = raw->insn_idx[j], final; +struct bpf_program *p;[ ... ]
Quoted 4 lines
tools/lib/bpf/libbpf.c +4
-0+if (owner == prog) { + final = raw->insn_idx[j] - prog->sec_insn_off; + }else if (prog_is_subprog (obj, owner) && owner->sub_insn_off) { +/* sub_insn_off is where this subprogram was tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 5062
Quoted 10 lines
tools/lib/bpf/libbpf.c +10
-0+* appended to the main program being relocated; +* zero means it is not part of it. +*/ + final = owner->sub_insn_off + + raw->insn_idx[j] - owner->sec_insn_off; + }else { + owner =NULL ; +break ; + } + fields[j] = final;[Severity: Low]
Is it safe to cast a pointer to a struct field into an array pointer and access subsequent members via pointer arithmetic in this loop?The struct
bpf_cleanup_infofields are populated by casting a pointer to the struct to a__u32* array and indexing into it in a loop.
Quoted 6 lines
According to C11 6.5.6, pointer arithmetic on a pointer to a scalar (the first struct field) only allows accessing up to one element past it, making fields[1] and fields[2] undefined behavior.
This can trigger compiler warnings (-Warray-bounds) and potentially cause aggressive optimizers to generate incorrect code.
Yes, the code is correct but it is awkward and my trigger compiler warnings with additional compiler flag. Will make it explicit with bpf_cleanup_info fields.
Quoted 13 lines
Quoted 13 lines
tools/lib/bpf/libbpf.c +12
-0+ } +if (!owner) +continue ; + + tmp =libbpf_reallocarray (prog->cleanup_info, prog->cleanup_info_cnt +1 , +sizeof (*prog->cleanup_info)); +if (!tmp) +return -ENOMEM ; + prog->cleanup_info = tmp; + prog->cleanup_info_rec_size =sizeof (struct bpf_cleanup_info ); + prog->cleanup_info[prog->cleanup_info_cnt++] = ci; +
Quoted 8 lines
tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 5080
Quoted 6 lines
tools/lib/bpf/libbpf.c +6
-0+pr_debug ("prog '%s': cleanup region [%u,%u) -> landing pad %u\n" , + prog->name, ci.begin_off, ci.end_off, ci.landing_pad_off); + } + +qsort (prog->cleanup_info, prog->cleanup_info_cnt, +sizeof (*prog->cleanup_info), cmp_cleanup_info);[Severity: Medium]
Can this unconditionally callqsort()with a NULL base pointer?
You are right. It is possible prog->cleanup_info_cnt is 0 and in such cases, cleanup_info will be NULL. Will add a condition like 'prog->cleanup_info_cnt > 0'.
Quoted 8 lines
If
obj->cleanup_rec_cntis 0, or if none of the records match the program,prog->cleanup_info_cntremains 0 andprog->cleanup_infowill be NULL.Calling
qsort()with a NULL pointer invokes undefined behavior under C11 7.1.4, even if the element count is 0. This triggers UBSAN errors (nonnull attribute violation in libc headers) and can fail runtime checks.
Yonghong Song
reviewerPATCH 16/20 libbpf: Carry the exception cleanup table through the light skeleton
In reply to Yonghong SongA light skeleton does not call bpf_prog_load(). bpf_gen__prog_load() builds its own union bpf_attr field by field, and the loader program it emits is what issues BPF_PROG_LOAD when the skeleton runs -- so a program loaded this way reached the kernel without the table the previous patch collected for it. The load then fails, because the landing pads are code nothing reaches and the verifier says so. It says "unreachable insn", which names neither the skeleton nor the table.
Carry it the way func_info and line_info are carried: the records go into the loader's blob of bytes, the count and record size into the attr, and a relocation stores the blob's address into attr.cleanup_info once that address is known. The attr grows to its new last field, cleanup_info_cnt. Records are 4-byte fields like the other info blobs, so a cross-endian build has to swap them too.
The kfunc names take the same detour. The light skeleton writes the name into the blob for the loader program to resolve when it runs, rather than resolving it against the kernel's BTF while libbpf runs, so the name recorded there has to go through kern_extern_name() as well.
--- tools/lib/bpf/gen_loader.c | 27 +++++++++++++++++++++++---- tools/lib/bpf/libbpf.c | 4 ++-- tools/lib/bpf/libbpf_internal.h | 7 +++++++ 3 files changed, 32 insertions(+), 6 deletions(-)
tools/lib/bpf/gen_loader.c +23 -4
@@ -981,13 +981,15 @@ static void cleanup_relos(struct bpf_gen *gen, int insns)cleanup_core_relo (gen); } -/* Convert func, line, and core relo info blobs to target endianness */ +/* Convert func, line, core relo and cleanup info blobs to target endianness */ static void info_blob_bswap (struct bpf_gen *gen,int func_info,int line_info, -int core_relos,struct bpf_prog_load_opts *load_attr) +int core_relos,int cleanup_info, +struct bpf_prog_load_opts *load_attr) {struct bpf_func_info *fi = gen->data_start + func_info;struct bpf_line_info *li = gen->data_start + line_info;struct bpf_core_relo *cr = gen->data_start + core_relos; +struct bpf_cleanup_info *ci = gen->data_start + cleanup_info;int i;for (i =0 ; i < load_attr->func_info_cnt; i++)
@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,for (i =0 ; i < gen->core_relo_cnt; i++)bpf_core_relo_bswap (cr++); + +for (i =0 ; i < load_attr->cleanup_info_cnt; i++) +bpf_cleanup_info_bswap (ci++); } void bpf_gen__prog_load(struct bpf_gen *gen,
@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen, load_attr->line_info_rec_size;int core_relo_tot_sz = gen->core_relo_cnt *sizeof (struct bpf_core_relo ); +int cleanup_info_tot_sz = load_attr->cleanup_info_cnt * + load_attr->cleanup_info_rec_size;int prog_load_attr, license_off, insns_off, func_info, line_info, core_relos; - int attr_size = offsetofend(union bpf_attr, core_relo_rec_size); + int attr_size = offsetofend(union bpf_attr, cleanup_info_cnt); +int cleanup_info;union bpf_attr attr;memset (&attr,0 , attr_size);
@@ -1061,9 +1069,17 @@ void bpf_gen__prog_load(struct bpf_gen *gen, core_relos, gen->core_relo_cnt, sizeof(struct bpf_core_relo)); + attr.cleanup_info_rec_size =tgt_endian (load_attr->cleanup_info_rec_size); + attr.cleanup_info_cnt =tgt_endian (load_attr->cleanup_info_cnt); + cleanup_info =add_data (gen, load_attr->cleanup_info, cleanup_info_tot_sz); +pr_debug ("gen: prog_load: cleanup_info: off %d cnt %u rec size %u\n" , + cleanup_info, load_attr->cleanup_info_cnt, + load_attr->cleanup_info_rec_size); +/* convert all info blobs to target endianness */ if (gen->swapped_endian && !gen->error) -info_blob_bswap (gen, func_info, line_info, core_relos, load_attr); +info_blob_bswap (gen, func_info, line_info, core_relos, cleanup_info, + load_attr);libbpf_strlcpy (attr.prog_name, prog_name,sizeof (attr.prog_name)); prog_load_attr =add_data (gen, &attr, attr_size);
@@ -1085,6 +1101,9 @@ void bpf_gen__prog_load(struct bpf_gen *gen,/* populate union bpf_attr with a pointer to core_relos */ emit_rel_store (gen,attr_field (prog_load_attr, core_relos), core_relos); +/* populate union bpf_attr with a pointer to cleanup_info */ +emit_rel_store (gen,attr_field (prog_load_attr, cleanup_info), cleanup_info); +/* populate union bpf_attr fd_array with a pointer to data where map_fds are saved */ emit_rel_store (gen,attr_field (prog_load_attr, fd_array), gen->fd_array);
tools/lib/bpf/libbpf.c +2 -2
@@ -8632,12 +8632,12 @@ static int bpf_program_record_relos(struct bpf_program *prog)continue ; kind =btf_is_var (btf__type_by_id (obj->btf, ext->btf_id)) ?BTF_KIND_VAR :BTF_KIND_FUNC ; -bpf_gen__record_extern (obj->gen_loader, ext->name, +bpf_gen__record_extern (obj->gen_loader,kern_extern_name (ext->name), ext->is_weak, !ext->ksym.type_id, true, kind, relo->insn_idx);break ;case RELO_EXTERN_CALL : -bpf_gen__record_extern (obj->gen_loader, ext->name, +bpf_gen__record_extern (obj->gen_loader,kern_extern_name (ext->name), ext->is_weak, false, false,BTF_KIND_FUNC , relo->insn_idx);break ;
tools/lib/bpf/libbpf_internal.h +7 -0
@@ -572,6 +572,13 @@ static inline void bpf_core_relo_bswap(struct bpf_core_relo *i) i->kind =bswap_32 (i->kind); } +static inline void bpf_cleanup_info_bswap (struct bpf_cleanup_info *i) +{ + i->begin_off =bswap_32 (i->begin_off); + i->end_off =bswap_32 (i->end_off); + i->landing_pad_off =bswap_32 (i->landing_pad_off); +} +enum btf_field_iter_kind {BTF_FIELD_ITER_IDS ,BTF_FIELD_ITER_STRS ,
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 16/20] libbpf: Carry the exception cleanup table through the light skeleton
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] libbpf: unconditional cleanup_info emission breaks backward compatibility on older kernels…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] libbpf: unconditional cleanup_info emission breaks backward compatibility on older kernels
- [Medium] libbpf: integer overflow in cleanup_info size calculation leads to OOB read/write --
commit 076a2794e54df3a939c1a2aebf4467a8a0af5e41
Author: Yonghong Song <[email protected]>
libbpf: Carry the exception cleanup table through the light skeleton
This commit adds support for carrying the exception cleanup table through the light skeleton during BPF program loads. It expands the bpf_attr union to include cleanup_info fields and handles endianness byte-swapping for the records.
Quoted 4 lines
tools/lib/bpf/gen_loader.c +0
-0
[ ... ]
tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1020
Quoted 6 lines
tools/lib/bpf/gen_loader.c +2
-0@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen, load_attr->line_info_rec_size;int core_relo_tot_sz = gen->core_relo_cnt *sizeof (struct bpf_core_relo ); +int cleanup_info_tot_sz = load_attr->cleanup_info_cnt * + load_attr->cleanup_info_rec_size;
[Severity: Medium]
Is there a risk of integer overflow here when parsing untrusted ELF files?
If load_attr->cleanup_info_cnt is large enough, the multiplication with load_attr->cleanup_info_rec_size could overflow the 32-bit signed integer cleanup_info_tot_sz, resulting in a truncated heap allocation.
Later in info_blob_bswap(), the loop executes load_attr->cleanup_info_cnt times:
info_blob_bswap() {
...
for (i = 0; i < load_attr->cleanup_info_cnt; i++)
bpf_cleanup_info_bswap(ci++);
}Could this lead to an out-of-bounds memory read and write over the truncated buffer, potentially causing memory corruption in userspace tooling like bpftool?
[ ... ]
tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1105
Quoted 6 lines
tools/lib/bpf/gen_loader.c +2
-0@@ -1085,6 +1101,9 @@ void bpf_gen__prog_load(struct bpf_gen *gen,/* populate union bpf_attr with a pointer to core_relos */ emit_rel_store (gen,attr_field (prog_load_attr, core_relos), core_relos); +/* populate union bpf_attr with a pointer to cleanup_info */ +emit_rel_store (gen,attr_field (prog_load_attr, cleanup_info), cleanup_info);
[Severity: High]
Will this unconditional emission cause light skeletons to fail to load on older kernels?
By unconditionally writing a non-zero pointer (the offset in the data blob) into the tail of the BPF attribute structure, older target kernels that do not support the cleanup_info field might reject the BPF_PROG_LOAD syscall.
Because sys_bpf verifies that all fields beyond its known sizeof(union bpf_attr) are zero using bpf_check_uarg_tail_zero(), it seems this could fail with -E2BIG. Does this break backward compatibility for all light skeletons running on older kernels, even those that do not use exceptions?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
Yonghong Song
reviewerRe: [PATCH bpf-next 16/20] libbpf: Carry the exception cleanup table through the light skeleton
Quoted 29 lines · On 9/16/26 11:18 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] libbpf: unconditionalcleanup_infoemission breaks backward compatibility on older kernels
- [Medium] libbpf: integer overflow incleanup_infosize calculation leads to OOB read/write --commit
076a2794e54df3a939c1a2aebf4467a8a0af5e41
Author: Yonghong Song <[email protected]>libbpf: Carry the exception cleanup table through the light skeleton
This commit adds support for carrying the exception cleanup table through the light skeleton during BPF program loads. It expands the
bpf_attrunion to includecleanup_infofields and handles endianness byte-swapping for the records.Quoted 4 lines
tools/lib/bpf/gen_loader.c +0
-0[ ... ]
tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1020
Quoted 6 lines
tools/lib/bpf/gen_loader.c +2
-0@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen, load_attr->line_info_rec_size;int core_relo_tot_sz = gen->core_relo_cnt *sizeof (struct bpf_core_relo ); +int cleanup_info_tot_sz = load_attr->cleanup_info_cnt * + load_attr->cleanup_info_rec_size;[Severity: Medium]
Is there a risk of integer overflow here when parsing untrusted ELF files?
Quoted 3 lines
If
load_attr->cleanup_info_cntis large enough, the multiplication withload_attr->cleanup_info_rec_sizecould overflow the 32-bit signed integercleanup_info_tot_sz, resulting in a truncated heap allocation.
We should be okay here. In add_data, we will see:
static int add_data(struct bpf_gen *gen, const void *data, __u32 size)
{
__u64 zero = 0;
__u32 size8;
void *prev;
if (size > INT32_MAX) {
gen->error = -ERANGE;
return 0;
}
size8 = roundup(size, 8);
...
}So size guarding above prevents integer overflow. add_data() also applies to func_info_tot_sz, line_info_tot_sz, core_relo_tot_sz and here cleanup_info_tot_sz.
Quoted 19 lines
Later in
info_blob_bswap(), the loop executesload_attr->cleanup_info_cnttimes:info_blob_bswap() { ... for (i = 0; i < load_attr->cleanup_info_cnt; i++) bpf_cleanup_info_bswap(ci++); }Could this lead to an out-of-bounds memory read and write over the truncated buffer, potentially causing memory corruption in userspace tooling like bpftool?
[ ... ]
Quoted 4 lines
tools/lib/bpf/gen_loader.c +0
-0@@ -1085,6 +1101,9 @@ void bpf_gen__prog_load(struct bpf_gen *gen,/* populate union bpf_attr with a pointer to core_relos */ emit_rel_store (gen,attr_field (prog_load_attr, core_relos), core_relos);
Quoted 5 lines
tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1105
Quoted 2 lines
tools/lib/bpf/gen_loader.c +2
-0+/* populate union bpf_attr with a pointer to cleanup_info */ +emit_rel_store (gen,attr_field (prog_load_attr, cleanup_info), cleanup_info);[Severity: High]
Will this unconditional emission cause light skeletons to fail to load on older kernels?
You are right. We cannot unconditional emit. We can only do emit if cleanup_info_cnt > 0.
Quoted 10 lines
By unconditionally writing a non-zero pointer (the offset in the data blob) into the tail of the BPF attribute structure, older target kernels that do not support the
cleanup_infofield might reject theBPF_PROG_LOADsyscall.Because
sys_bpfverifies that all fields beyond its knownsizeof(union bpf_attr)are zero usingbpf_check_uarg_tail_zero(), it seems this could fail with-E2BIG. Does this break backward compatibility for all light skeletons running on older kernels, even those that do not use exceptions?
Yonghong Song
reviewerPATCH 17/20 libbpf: Let the static linker carry .bpf_cleanup relocations
In reply to Yonghong SongAn object that carries a compiler-emitted exception cleanup table cannot be linked today. The table's fields are byte offsets into a code section, materialised by a 32-bit relocation against that section's symbol with the offset itself as the implicit addend, and the linker rejects both halves of
that: the relocation type is not in the list it accepts, and a relocation against an STT_SECTION symbol from a non-executable section is an outright error.
Both spellings of that relocation have to be taken. LLVM emits R_BPF_64_NODYLD32 for a .long against a section symbol; GNU as emits R_BPF_64_ABS32, which is what bpf_reloc_type_lookup() maps BFD_RELOC_32 to. They describe the same value, and the selftests are built with both compilers.
--- tools/lib/bpf/linker.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)
tools/lib/bpf/linker.c +18 -1
@@ -1036,7 +1036,8 @@ static int linker_sanity_check_elf_relos(struct src_obj *obj, struct src_sec *sesize_t sym_type =ELF64_R_TYPE (relo->r_info);if (sym_type !=R_BPF_64_64 && sym_type !=R_BPF_64_32 && - sym_type !=R_BPF_64_ABS64 && sym_type !=R_BPF_64_ABS32 ) { + sym_type !=R_BPF_64_ABS64 && sym_type !=R_BPF_64_ABS32 && + sym_type !=R_BPF_64_NODYLD32 ) {pr_warn ("ELF relo #%d in section #%zu has unexpected type %zu in %s\n" , i, sec->sec_idx, sym_type, obj->filename);return -EINVAL ;
@@ -2274,6 +2275,22 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; + }else if (sym_type ==R_BPF_64_NODYLD32 || + sym_type ==R_BPF_64_ABS32 ) { +__u32 *val; + +/* Two spellings of the one thing: LLVM +* emits NODYLD32 for a .long against a +* section symbol, GNU as emits ABS32 +* (bpf_reloc_type_lookup() maps +* BFD_RELOC_32 to it), and the value +* they describe is the same. +*/ + val = dst_linked_sec->raw_data + dst_rel->r_offset; +if (linker->swapped_endian) + *val =bswap_32 (bswap_32 (*val) + sec->dst_off); +else + *val += sec->dst_off; }else {pr_warn ("relocation against STT_SECTION in non-exec section is not supported!\n" );return -EINVAL ;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] libbpf: unaligned memory access in linker for 32-bit relocations --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] libbpf: unaligned memory access in linker for 32-bit relocations --
commit 408d36d153c1766697801912b5eb3e8087995ef9
Author: Yonghong Song <[email protected]>
libbpf: Let the static linker carry .bpf_cleanup relocations
This commit allows the BPF static linker to process 32-bit relocations against non-executable sections by handling R_BPF_64_NODYLD32 and R_BPF_64_ABS32 relocation types. It updates the 32-bit values directly in the raw data of the relocated section.
Quoted 4 lines
tools/lib/bpf/linker.c +0
-0
[ ... ]
Quoted 10 lines
tools/lib/bpf/linker.c +6
-0@@ -2274,6 +2275,22 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; + }else if (sym_type ==R_BPF_64_NODYLD32 || + sym_type ==R_BPF_64_ABS32 ) { +__u32 *val; + +/* Two spellings of the one thing: LLVM +* emits NODYLD32 for a .long against a
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2293
Quoted 10 lines
tools/lib/bpf/linker.c +10
-0+* section symbol, GNU as emits ABS32 +* (bpf_reloc_type_lookup() maps +* BFD_RELOC_32 to it), and the value +* they describe is the same. +*/ + val = dst_linked_sec->raw_data + dst_rel->r_offset; +if (linker->swapped_endian) + *val =bswap_32 (bswap_32 (*val) + sec->dst_off); +else + *val += sec->dst_off;
[Severity: Medium]
Can this lead to an unaligned memory access crash?
The static linker processes these 32-bit relocations against non-executable sections by casting the relocated offset to a 32-bit integer pointer and directly dereferencing it.
If a user supplies a malformed BPF object file with an unaligned relocation offset, this could result in undefined behavior and a crash (SIGBUS) on host architectures with strict alignment requirements.
Should linker_sanity_check_elf_relos() be updated to ensure the relocation offset is a multiple of 4 for non-executable sections, similar to how it checks alignment for SHF_EXECINSTR sections?
tools/lib/bpf/linker.c:linker_sanity_check_elf_relos() {
if (link_sec->shdr->sh_flags & SHF_EXECINSTR) {
if (relo->r_offset % sizeof(struct bpf_insn) != 0) {
pr_warn("ELF relo #%d in section #%zu points to missing symbol #%zu in %s\n",
i, sec->sec_idx, sym_idx, obj->filename);
return -EINVAL;
}
}
}Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
Yonghong Song
reviewerRe: [PATCH bpf-next 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations
Quoted 43 lines · On 9/16/26 11:16 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] libbpf: unaligned memory access in linker for 32-bit relocations --commit
408d36d153c1766697801912b5eb3e8087995ef9
Author: Yonghong Song <[email protected]>libbpf: Let the static linker carry .bpf_cleanup relocations
This commit allows the BPF static linker to process 32-bit relocations against non-executable sections by handling
R_BPF_64_NODYLD32andR_BPF_64_ABS32relocation types. It updates the 32-bit values directly in the raw data of the relocated section.Quoted 4 lines
tools/lib/bpf/linker.c +0
-0[ ... ]
Quoted 10 lines
tools/lib/bpf/linker.c +6
-0@@ -2274,6 +2275,22 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; + }else if (sym_type ==R_BPF_64_NODYLD32 || + sym_type ==R_BPF_64_ABS32 ) { +__u32 *val; + +/* Two spellings of the one thing: LLVM +* emits NODYLD32 for a .long against a tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2293
Quoted 10 lines
tools/lib/bpf/linker.c +10
-0+* section symbol, GNU as emits ABS32 +* (bpf_reloc_type_lookup() maps +* BFD_RELOC_32 to it), and the value +* they describe is the same. +*/ + val = dst_linked_sec->raw_data + dst_rel->r_offset; +if (linker->swapped_endian) + *val =bswap_32 (bswap_32 (*val) + sec->dst_off); +else + *val += sec->dst_off;[Severity: Medium]
Can this lead to an unaligned memory access crash?
Quoted 7 lines
The static linker processes these 32-bit relocations against non-executable sections by casting the relocated offset to a 32-bit integer pointer and directly dereferencing it.
If a user supplies a malformed BPF object file with an unaligned relocation offset, this could result in undefined behavior and a crash (SIGBUS) on host architectures with strict alignment requirements.
This only happen for a malformed BPF object file and we don't care if the BPF object file is indeed malformed.
Quoted 15 lines
Should
linker_sanity_check_elf_relos()be updated to ensure the relocation offset is a multiple of 4 for non-executable sections, similar to how it checks alignment forSHF_EXECINSTRsections?tools/lib/bpf/linker.c:linker_sanity_check_elf_relos() { if (link_sec->shdr->sh_flags & SHF_EXECINSTR) { if (relo->r_offset % sizeof(struct bpf_insn) != 0) { pr_warn("ELF relo #%d in section #%zu points to missing symbol #%zu in %s\n", i, sec->sec_idx, sym_idx, obj->filename); return -EINVAL; } } }
Yonghong Song
reviewerPATCH 18/20 selftests/bpf: Add an end-to-end .bpf_cleanup exception test
In reply to Yonghong SongC has no unwinding, so nothing here comes out of the frontend: the frames that own a resource are written as __naked inline assembly, which spells out by hand exactly what a frontend emits -- a call site bracketed by two labels, a landing pad unreachable in the compiler's CFG, and a .bpf_cleanup record tying them together. The assembler turns ".long <text label>" into the same R_BPF_64_NODYLD32 relocation the BPF AsmPrinter emits, so libbpf and the kernel see an object indistinguishable from a compiler-generated one.
Call chain: entry -> foo1 -> foo1v -> foo2 -> foo3. foo3 holds a non-preemptible section and throws inside it; foo2 holds an RCU read lock and has two call sites sharing one pad, one of them its own throw; foo1v is a void frame whose pad ends in a jump to a resume block placed after an unrelated block that ends in a plain exit; foo1 owns nothing and gets no record; entry is the boundary.
There are also some shapes the kernel refuses -- the ones with no correct answer, and the ones a pad running on the bpf_throw() walker's stack cannot express -- plus the return-value rejection that delivering an exception at the boundary of a program type which constrains its return value produces. The test skips rather than fails where the JIT cannot dispatch a landing pad at all.
--- .../selftests/bpf/exceptions_cleanup.h | 29 + .../bpf/prog_tests/exceptions_cleanup.c | 79 +++ .../selftests/bpf/progs/exceptions_cleanup.c | 152 +++++ .../bpf/progs/exceptions_cleanup_fail.c | 600 ++++++++++++++++++ 4 files changed, 860 insertions(+) create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
tools/testing/selftests/bpf/exceptions_cleanup.h new file +29 -0
@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#ifndef __EXCEPTIONS_CLEANUP_H__ +#define __EXCEPTIONS_CLEANUP_H__ + +#define THROW_COOKIE 0x100 + +/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */ +#define RAN_FOO3_PREEMPT 0x1 +#define RAN_FOO2_RCU 0x2 +#define RAN_FOO1V_PREEMPT 0x4 +#define RAN_FOO2_DROP 0x8 +#define RAN_BUMP 0x10 + +#define CLEANUP_REC (begin, end, landing_pad) \ + ".pushsection .bpf_cleanup,\"a\",@progbits;" \ + ".long " begin ";" \ + ".long " end ";" \ + ".long " landing_pad ";" \ + ".popsection;" + +/* Set a bit in @pads_ran. */ +#define PAD_RAN (bit) \ + "r1 = %[pads_ran] ll;" \ + "r2 = *(u64 *)(r1 + 0);" \ + "r2 |= " bit ";" \ + "*(u64 *)(r1 + 0) = r2;" + +#endif /* __EXCEPTIONS_CLEANUP_H__ */
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +79 -0
79 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_cleanup.c new file +152 -0
152 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +600 -0
600 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 18/20] selftests/bpf: Add an end-to-end .bpf_cleanup exception test
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] bpf: multi-line comments do not follow subsystem style --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: multi-line comments do not follow subsystem style
--
commit 5918cbd599a9ba7808857475f6ca49093862b54a
Author: Yonghong Song <[email protected]>
selftests/bpf: Add an end-to-end .bpf_cleanup exception test
This commit adds an end-to-end test for .bpf_cleanup exception handling, using inline assembly to construct landing pads and cleanup records. It validates both successful exception propagation scenarios and various edge cases that the kernel is expected to refuse.
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +0
-0
[ ... ]
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c line 72
Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +7
-0+/* foo3 throws; every pad runs and the cookie is delivered at entry. */ +if (test__start_subtest ("throw_from_foo3" )) +run (skel,101 ,THROW_COOKIE ,PADS_FOO3_THREW ); + +/* foo3 returns 2 ^ 1 == 3, so foo2 throws from its own second region; +* foo3's frame is long gone, so its pad must not run. +*/
[Severity: Low]
This isn't a bug, but does this multi-line comment follow the BPF subsystem style? The guidelines prefer placing the opening /* on its own line for multi-line comments.
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +4
-0+if (test__start_subtest ("throw_from_foo2" )) +run (skel,2 ,THROW_COOKIE ,PADS_FOO2_THREW ); + +exceptions_cleanup__destroy (skel);
[ ... ]
Quoted 4 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c +0
-0
[ ... ]
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c line 586
Quoted 9 lines
+"3:" /* landing pad */
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c +8
-0+"r1 = jt0_%= ll;" +"r1 += 8;" +"r2 = *(u64 *)(r1 + 0);" +/* gotox r2. Spelled as a raw insn on purpose: the "gotox" mnemonic +* only reached the LLVM assembler in llvm 22, and BPF_RAW_INSN() +* needs <linux/bpf.h>, which this file cannot have -- vmlinux.h +* already defines the uapi enums. +*/
[Severity: Low]
This is also not a bug, but does this comment formatting comply with the subsystem guidelines? Like the previous instance, the preferred BPF style places the opening /* on a separate line.
Quoted 4 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c +4
-0+".8byte 0x20d;" +"l0_%=:" +"call bpf_unwind_resume;" +"exit;"
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
Yonghong Song
reviewerPATCH 19/20 selftests/bpf: Cover the exception cleanup shapes the chain does not reach
In reply to Yonghong SongThe end-to-end test walks one call chain with a pad in most of its frames. This adds the shapes that chain does not reach: a callee called from both a covered and an uncovered site, a pad that reads its frame's callee-saved registers, a tail-call-reachable callee, a tail call that is taken, an extension standing in for a covered call, a pad in the main program's own frame, a record covering bpf_throw() itself, a record covering only a nounwind call, a throwing subprogram named by a BPF_PSEUDO_FUNC, a region ending on a 16-byte instruction, a pad that reloads from and writes to its own frame, a pad two frames up, a pad that calls a subprogram which tail calls, an extension carrying a table of its own, and a pad terminated by _Unwind_Resume rather than bpf_unwind_resume.
--- .../selftests/bpf/exceptions_cleanup.h | 20 + .../bpf/prog_tests/exceptions_cleanup.c | 314 +++++++ .../bpf/progs/exceptions_cleanup_ext_table.c | 48 + .../bpf/progs/exceptions_cleanup_freplace.c | 17 + .../progs/exceptions_cleanup_pad_freplace.c | 17 + .../bpf/progs/exceptions_cleanup_shapes.c | 863 ++++++++++++++++++ 6 files changed, 1279 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
tools/testing/selftests/bpf/exceptions_cleanup.h +20 -0
@@ -4,6 +4,7 @@#define __EXCEPTIONS_CLEANUP_H__#define THROW_COOKIE 0x100 +#define INNER_COOKIE 0x200/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */ #define RAN_FOO3_PREEMPT 0x1
@@ -12,6 +13,25 @@#define RAN_FOO2_DROP 0x8#define RAN_BUMP 0x10 +/* progs/exceptions_cleanup_shapes.c: one bit per shape, numbered its own way. */ +#define RAN_SWEEP 0x1 +#define RAN_SHARED 0x2 +#define RAN_REGS 0x4 +#define RAN_TAIL_CALL 0x8 +#define RAN_MAIN_PAD 0x10 +#define RAN_TC_TAKEN 0x20 +#define RAN_FREPLACE 0x40 +#define RAN_ADDR_TAKEN 0x80 +#define RAN_NO_SUBPROG 0x100 +#define RAN_PAD_CALLS 0x200 +#define RAN_PAD_FIRST 0x400 +#define RAN_WIDE_REC 0x800 +#define RAN_PAD_STACK 0x1000 +#define RAN_DEEP_PAD 0x2000 +#define RAN_NOUNWIND_REC 0x4000 +#define RAN_RESUME_ALIAS 0x8000 +#define RAN_PAD_TAIL_CALL 0x10000 +#define CLEANUP_REC (begin, end, landing_pad) \ ".pushsection .bpf_cleanup,\"a\",@progbits;" \ ".long " begin ";" \
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +314 -0
331 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c new file +48 -0
48 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "exceptions_cleanup.h" + +/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */ +int btf_int_anchor; + +SEC("freplace/fr_callee") +__u64 new_fr_callee (__u64 x) +{ +bpf_throw (THROW_COOKIE ); +return 0 ; +} + +char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "exceptions_cleanup.h" + +/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */ +int btf_int_anchor; + +SEC("freplace/pad_callee") +__u64 new_pad_callee (__u64 x) +{ +bpf_throw (INNER_COOKIE ); +return 0 ; +} + +char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +863 -0
863 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 19/20] selftests/bpf: Cover the exception cleanup shapes the chain does not reach
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] bpf: multi-line comments violate BPF subsystem style --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: multi-line comments violate BPF subsystem style
--
commit 621c13df81280f1c0c32e09255ce8c64860bd768
Author: Yonghong Song <[email protected]>
selftests/bpf: Cover the exception cleanup shapes the chain does not reach
The end-to-end test walks one call chain with a pad in most of its frames. This adds the shapes that chain does not reach: a callee called from both a covered and an uncovered site, a pad that reads its frame's callee-saved registers, a tail-call-reachable callee, a tail call that is taken, an extension standing in for a covered call, a pad in the main program's own frame, a record covering bpf_throw() itself, a record covering only a nounwind call, a throwing subprogram named by a BPF_PSEUDO_FUNC, a region ending on a 16-byte instruction, a pad that reloads from and writes to its own frame, a pad two frames up, a pad that calls a subprogram which tail calls, an extension carrying a table of its own, and a pad terminated by _Unwind_Resume rather than bpf_unwind_resume.
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +0
-0@@ -212,6 +212,9 @@ static void test_shapes(void)
[ ... ]
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 215
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +3
-0+/* The same call site either way: the subprogram's throw unwinds into +* this frame and runs its pad, an extension's stops at its own boundary. +*/
[Severity: Low]
Does this multi-line comment in test_shapes() violate the BPF subsystem style?
The style guidelines prefer the opening /* on its own line:
/* * The same call site either way: the subprogram's throw unwinds into * this frame and runs its pad, an extension's stops at its own boundary. */
[ ... ]
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 224
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +3
-0+/* A tail call that is taken: the walk ends at the target, so the cookie +* comes back from there and this frame's pad does not run. +*/
[Severity: Low]
Could this multi-line comment in test_shapes() also be updated to start the text on the line following the opening /* to match the preferred BPF style?
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c static __used __noinline __u64 tc_thrower(void *ctx) · line 217
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +3
-0@@ -214,6 +214,9 @@ static __used __noinline __u64 tc_thrower(void *ctx) { +/* Never taken; its presence is what makes this frame, whose spill the +* walker reads, tail-call-reachable. +*/
[Severity: Low]
Is this multi-line comment in tc_thrower() formatted correctly?
The BPF subsystem style requires the opening /* on its own line for multi-line comments.
[ ... ]
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c static __used __noinline __u64 tc_taken_callee(void *ctx, __u64 x) · line 314
Quoted 5 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +3
-0@@ -311,6 +311,9 @@ static __used __noinline __u64 tc_taken_callee(void *ctx, __u64 x) { +/* Never true at run time; the verifier cannot know that, and its +* unwind out of here is what keeps the caller's pad alive. +*/
[Severity: Low]
Should this multi-line comment in tc_taken_callee() also be adjusted so the text starts on the next line after the opening /* marker?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19
Yonghong Song
reviewerPATCH 20/20 selftests/bpf: Load an exception cleanup program from a light skeleton
In reply to Yonghong SongFor a light skeleton, libbpf hands the records to bpf_gen__prog_load(), which writes them into a blob and emits a loader program that issues BPF_PROG_LOAD from inside the kernel. Nothing about that is shared with the ordinary path: the attr is built field by field, and the kfunc names are resolved by the loader program when it runs.
exceptions_cleanup_light.c is the smallest program that can tell whether a table survives that trip: one record covering the bpf_throw() call itself, one pad that sets a bit and resumes. If the table arrives, the pad runs and the cookie comes back; if it does not, the program does not load at all.
--- tools/testing/selftests/bpf/Makefile | 2 +- .../selftests/bpf/exceptions_cleanup.h | 3 ++ .../bpf/prog_tests/exceptions_cleanup.c | 28 +++++++++++++ .../bpf/progs/exceptions_cleanup_light.c | 39 +++++++++++++++++++ 4 files changed, 71 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c
tools/testing/selftests/bpf/Makefile +1 -1
@@ -523,7 +523,7 @@ LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ LSKELS := fexit_sleep.c trace_printk.c trace_vprintk.c map_ptr_kern.c \ core_kern.c core_kern_overflow.c test_ringbuf.c \ test_ringbuf_n.c test_ringbuf_map_key.c test_ringbuf_write.c \ - test_ringbuf_overwrite.c + test_ringbuf_overwrite.c exceptions_cleanup_light.c LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
tools/testing/selftests/bpf/exceptions_cleanup.h +3 -0
@@ -32,6 +32,9 @@#define RAN_RESUME_ALIAS 0x8000#define RAN_PAD_TAIL_CALL 0x10000 +/* progs/exceptions_cleanup_light.c: the one pad it has. */ +#define RAN_LIGHT 0x1 +#define CLEANUP_REC (begin, end, landing_pad) \ ".pushsection .bpf_cleanup,\"a\",@progbits;" \ ".long " begin ";" \
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +28 -0
@@ -8,6 +8,7 @@#include "exceptions_cleanup_freplace.skel.h" #include "exceptions_cleanup_pad_freplace.skel.h" #include "exceptions_cleanup_ext_table.skel.h" +#include "exceptions_cleanup_light.lskel.h" /* foo3 threw: every frame that has a pad ran it. */ #define PADS_FOO3_THREW \
@@ -170,6 +171,30 @@ static void test_ext_table(struct exceptions_cleanup_shapes *skel)exceptions_cleanup_ext_table__destroy (fr); } +static void test_light_skeleton (void ) +{ +struct exceptions_cleanup_light_lskel *skel; +__u64 ctx =0 ; +int err; + +LIBBPF_OPTS (bpf_test_run_opts, topts, + .ctx_in = &ctx, + .ctx_size_in = sizeof(ctx), + ); + + skel =exceptions_cleanup_light_lskel__open_and_load (); +if (!ASSERT_OK_PTR (skel,"light open_and_load" )) +return ; + + err =bpf_prog_test_run_opts (skel->progs.entry_light.prog_fd, &topts); +if (!ASSERT_OK (err,"run" )) + goto out; +ASSERT_EQ (topts.retval,THROW_COOKIE ,"retval" ); +ASSERT_EQ (skel->bss->pads_ran,RAN_LIGHT ,"pads_ran" ); +out: +exceptions_cleanup_light_lskel__destroy (skel); +} +static void test_shapes (void ) {struct exceptions_cleanup_shapes *skel;
@@ -387,6 +412,9 @@ void test_exceptions_cleanup(void)exceptions_cleanup__destroy (skel); +if (test__start_subtest ("light_skeleton" )) +test_light_skeleton (); +test_shapes ();RUN_TESTS (exceptions_cleanup_fail);
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +39 -0
@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" +#include "exceptions_cleanup.h" + +static __used __noinline void __kfunc_btf_anchor(void) +{ +bpf_throw (0 ); +bpf_preempt_disable (); +bpf_preempt_enable (); +bpf_unwind_resume (); +} + +__u64 pads_ran =0 ; + +SEC("syscall") +__naked int entry_light (void ) +{ + asmvolatile ( +"call bpf_preempt_disable;" +"r1 = %[cookie];" +"1:" "call bpf_throw;" /* cleanup region */ +"2:" +"exit;" +"3:" /* landing pad */ +"call bpf_preempt_enable;" + PAD_RAN("%[ran]") +"call bpf_unwind_resume;" +"exit;" +CLEANUP_REC ("1b" ,"2b" ,"3b" ) + : + : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_LIGHT), +__imm_addr (pads_ran) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.53.0-Meta