bpf: Run exception cleanup landing pads when bpf_throw() unwinds
Yonghong Song
reviewerv3 cover bpf: Run exception cleanup landing pads when bpf_throw() unwinds
bpf_throw() walks the BPF call stack to the exception boundary and discards every frame in between. A frame that owns something -- an RCU read lock, a preemption-disabled section, a referenced kptr -- never gets to give it back, so the verifier refuses to let such a frame throw at all. That is the whole reason a Rust program cannot use bpf_throw() as its panic path right now: Rust's Drop glue *is* that give-back, and there is nowhere to run it.
LLVM 23 added the compiler half ([1]). A Rust function that owns a value across a call that can unwind
fn foo() {
let _guard = RcuReadGuard::new(); /* bpf_rcu_read_lock() */
may_throw(); /* extern "C-unwind" */
} /* Drop: rcu_read_unlock */lowers to an invoke with a cleanup landing pad holding the Drop call, and the BPF backend writes one record per invoke region into a .bpf_cleanup
section: a flat table of 12-byte (begin, end, landing_pad) triples, each field a byte offset into the code section. The rule is "if a frame unwinds with its return address in [begin, end), run landing_pad before discarding it". A pad ends with a call to _Unwind_Resume(), which the kernel provides as the bpf_unwind_resume() kfunc.
This series is the kernel half: take that table at BPF_PROG_LOAD, teach the verifier that a covered call can also go to its landing pad, and have bpf_throw() run the pads as it walks.
C has no unwinding, so the selftests spell out by hand what a frontend emits -- a call site bracketed by two labels, a landing pad, and a record tying them together. The frame above, written that way:
"call bpf_rcu_read_lock;"
"1:" "call foo3;" /* cleanup region */
"2:"
... normal path, ends in bpf_rcu_read_unlock ...
"6:" /* landing pad */
"call bpf_rcu_read_unlock;"
"call bpf_unwind_resume;" CLEANUP_REC("1b", "2b", "6b")
Right now that program does not load: the throw inside foo3 is reported as "bpf_throw cannot be used inside bpf_rcu_read_lock-ed region", because as far as the verifier is concerned nothing will ever unlock. With the series the verifier walks the unwind the same way the run time will -- into the pad, which unlocks, then on to the next frame -- and the program both loads and releases the lock when it throws.
Design
======
A pad is run, not lowered. bpf_throw() already walks the frames with arch_bpf_stack_walk(); it now looks each frame's return address up in that (sub)program's table and calls the pad as a subroutine of the walker, with the unwinding frame's frame pointer and its callee-saved registers restored from the spill its callee's prologue left. The pad therefore sees its own frame but runs on the walker's stack, far below it, so nothing it calls can disturb the frame it is cleaning up after. The JIT turns its bpf_unwind_resume() into the way back to the walker.
The verifier walks the same thing, step for step, so the resource rules are unchanged: whatever a pad releases is released in the verifier state too, and check_resource_leak() simply moves from "a throw was seen" to the end of the walk.
1-2 uapi: cleanup_info in BPF_PROG_LOAD, struct bpf_cleanup_info,
and the bpf_unwind_resume() kfunc
3-9 verifier: mark the covered call sites, give them an edge to the
pad, walk the unwind, and refuse the shapes that cannot be dispatched
10 bpf_throw(): dispatch pads while walking 11-12 x86-64 and arm64 JITs 13-17 libbpf: collect .bpf_cleanup, pass it to the kernel, resolve
_Unwind_Resume, carry it through the light skeleton and the static linker
18-20 selftests
Limitations
===========
- Cleanup pads only. A catch pad -- one that ends in a plain exit rather than a resume, which is what Rust's catch_unwind would need -- is
refused: the walker calls a pad as a subroutine and cannot hand a frame back its own execution. LLVM refuses type-specific catches and filters on its side as well.
- A JIT that can dispatch pads is required: x86-64 (with CONFIG_UNWINDER_ORC, which bpf_throw() already needs there) and arm64. Anywhere else the load fails with -EOPNOTSUPP rather than silently doing nothing.
- No offloaded programs, no private stack, and no combining a table with an exception callback.
- In a pad body: no tail call, no BPF_LD_[ABS|IND], no indirect jump, and no on-stack call arguments -- each of them reads or writes a stack the pad does not own.
- The Rust toolchain does not properly support BPF exception handling yet. The tables the selftests use are hand-written inline asm, which the assembler turns into the same relocations the BPF AsmPrinter emits, so libbpf and the kernel see an object indistinguishable from a compiler-generated one.
[1] https://github.com/llvm/llvm-project/pull/192164
llvm commit 9d51c891b719 ("[BPF] Add exception handling support with .bpf_cleanup section")
Changelog
=========
v2 -> v3:
- v2: https://lore.kernel.org/bpf/[email protected]/
- Keep a landing pad's record when opt_remove_nops() deletes a pad that is a nop, instead of dropping it after the verifier has already checked the call site against it.
- Refuse a BPF_LD_[ABS|IND] in a pad body.
- Teach mark_chain_precision() about the throw-to-pad edge, which crosses frames with no instruction to account for them.
- Refuse a bpf_unwind_resume() in any frame but the one whose landing pad the walker entered.
- Check raw_data, alignment and bounds before the static linker writes through a relocation in a non-executable section, which may be SHT_NOBITS.
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Consolidate all usages of kern_extern_name() in a single patch in libbpf.
- Avoid compiler warning and add proper cleanup_info_cnt guard in libbpf when collecting .bpf_cleanup records.
- Add cleanup_info_cnt condition for emit_rel_store() with cleanup_info.
Yonghong Song (20):
bpf: Accept the compiler's exception cleanup table at program load
bpf: Add the bpf_unwind_resume() kfunc
bpf: Add lookups for exception cleanup resumes and landing pads
bpf: Prepare for an exception cleanup table before the CFG walk
bpf: Make exception landing pads reachable in the CFG
bpf: Explore the landing pads no call site reaches
bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch
bpf: Walk the exception unwind in the verifier
bpf: Refuse a private stack for a program with an exception cleanup
table
bpf: Dispatch exception cleanup pads from bpf_throw()
bpf, x86: Dispatch exception cleanup pads at run time
bpf, arm64: Dispatch exception cleanup pads at run time
libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
libbpf: Add cleanup_info to bpf_prog_load_opts
libbpf: Collect .bpf_cleanup records and pass them to the kernel
libbpf: Carry the exception cleanup table through the light skeleton
libbpf: Let the static linker carry .bpf_cleanup relocations selftests/bpf: Add an end-to-end .bpf_cleanup exception test selftests/bpf: Cover the exception cleanup shapes the chain does not
reach
selftests/bpf: Load an exception cleanup program from a light skeleton
arch/arm64/net/Makefile | 2 +- arch/arm64/net/bpf_cleanup_pad.S | 95 ++ arch/arm64/net/bpf_jit_comp.c | 91 +- arch/x86/net/Makefile | 3 + arch/x86/net/bpf_cleanup_pad.S | 80 ++ arch/x86/net/bpf_jit_comp.c | 79 +- include/linux/bpf.h | 84 ++ include/linux/bpf_cleanup_abi.h | 16 + include/linux/bpf_verifier.h | 17 +- include/linux/filter.h | 2 + include/uapi/linux/bpf.h | 9 + kernel/bpf/Makefile | 2 +- kernel/bpf/backtrack.c | 21 +- kernel/bpf/cfg.c | 71 +- kernel/bpf/check_btf.c | 147 +++ kernel/bpf/core.c | 35 +- kernel/bpf/exception.c | 664 ++++++++++++ kernel/bpf/exception.h | 22 + kernel/bpf/fixups.c | 166 ++- kernel/bpf/helpers.c | 46 + kernel/bpf/liveness.c | 20 + kernel/bpf/states.c | 6 + kernel/bpf/syscall.c | 2 +- kernel/bpf/verifier.c | 166 ++- tools/include/uapi/linux/bpf.h | 9 + tools/lib/bpf/bpf.c | 6 +- tools/lib/bpf/bpf.h | 7 +- tools/lib/bpf/gen_loader.c | 29 +- tools/lib/bpf/libbpf.c | 320 +++++- tools/lib/bpf/libbpf_internal.h | 10 + tools/lib/bpf/linker.c | 36 +- tools/testing/selftests/bpf/Makefile | 2 +- .../selftests/bpf/exceptions_cleanup.h | 55 + .../bpf/prog_tests/exceptions_cleanup.c | 454 ++++++++ .../selftests/bpf/progs/exceptions_cleanup.c | 152 +++ .../bpf/progs/exceptions_cleanup_ext_table.c | 48 + .../bpf/progs/exceptions_cleanup_fail.c | 662 ++++++++++++ .../bpf/progs/exceptions_cleanup_freplace.c | 17 + .../bpf/progs/exceptions_cleanup_light.c | 39 + .../progs/exceptions_cleanup_pad_freplace.c | 17 + .../bpf/progs/exceptions_cleanup_shapes.c | 984 ++++++++++++++++++ 41 files changed, 4621 insertions(+), 72 deletions(-) create mode 100644 arch/arm64/net/bpf_cleanup_pad.S create mode 100644 arch/x86/net/bpf_cleanup_pad.S create mode 100644 include/linux/bpf_cleanup_abi.h create mode 100644 kernel/bpf/exception.c create mode 100644 kernel/bpf/exception.h create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 1/20 bpf: Accept the compiler's exception cleanup table at program load
LLVM 23 added exception handling support for BPF with the .bpf_cleanup section ([1]). Rust code compiled with panic=unwind runs cleanup code (Drop glue) when bpf_throw() fires, and the LLVM BPF backend emits that section from the landing pads the frontend produced. Plain C cannot generate .bpf_cleanup unless inline asm is used. The Rust compiler does not *properly* support BPF exception handling yet, but the kernel can support the table today, and inline assembly is enough to test it.
Add the UAPI to carry the .bpf_cleanup table into the kernel. BPF_PROG_LOAD grows cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, and struct bpf_cleanup_info describes one record as a triple of instruction indices:
the half-open call-site range [begin_off, end_off) and the landing_pad_off the frame resumes at. check_cleanup_info() validates the table a program is loaded with, so the rest of the kernel can rely on it.
--- include/linux/bpf_verifier.h | 2 + include/uapi/linux/bpf.h | 9 ++ kernel/bpf/check_btf.c | 147 +++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 2 +- kernel/bpf/verifier.c | 1 + tools/include/uapi/linux/bpf.h | 9 ++ 6 files changed, 169 insertions(+), 1 deletion(-)
include/linux/bpf_verifier.h +2 -0
@@ -987,6 +987,8 @@ struct bpf_verifier_env {struct arg_track **callsite_at_stack;u32 pass_cnt;/* number of times do_check() was called */ u32 subprog_cnt; +struct bpf_cleanup_info *cleanup_info; +u32 cleanup_info_cnt;/* number of instructions analyzed by the verifier */ u32 prev_insn_processed, insn_processed;/* number of jmps, calls, exits analyzed so far */
include/uapi/linux/bpf.h +9 -0
@@ -1669,6 +1669,9 @@ union bpf_attr {* verification. */ __s32 keyring_id; +__aligned_u64 cleanup_info;/* exception cleanup table */ +__u32 cleanup_info_rec_size;/* userspace bpf_cleanup_info size */ +__u32 cleanup_info_cnt;/* number of bpf_cleanup_info records */ };struct {/* anonymous struct used by BPF_OBJ_* commands */
@@ -7599,6 +7602,12 @@ struct bpf_line_info {__u32 line_col; }; +struct bpf_cleanup_info { +__u32 begin_off; +__u32 end_off; +__u32 landing_pad_off; +}; +struct bpf_spin_lock {__u32 val; };
kernel/bpf/check_btf.c +147 -0
159 lines not on this page · raw message
kernel/bpf/syscall.c +1 -1
@@ -2915,7 +2915,7 @@ int __init __used bpf_multi_func(void) { return 0; }BTF_ID_LIST_GLOBAL_SINGLE (bpf_multi_func_btf_id, func, bpf_multi_func) /* last field in 'union bpf_attr' used by this command */ -#define BPF_PROG_LOAD_LAST_FIELD keyring_id +#define BPF_PROG_LOAD_LAST_FIELD cleanup_info_cntstatic int bpf_prog_load (union bpf_attr *attr,bpfptr_t uattr,struct bpf_log_attr *attr_log) {
kernel/bpf/verifier.c +1 -0
@@ -21884,6 +21884,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,kvfree (env->succ);kvfree (env->gotox_tmp_buf);bpf_diag_free (env); +kvfree (env->cleanup_info);kvfree (env);return ret; }
tools/include/uapi/linux/bpf.h +9 -0
@@ -1669,6 +1669,9 @@ union bpf_attr {* verification. */ __s32 keyring_id; +__aligned_u64 cleanup_info;/* exception cleanup table */ +__u32 cleanup_info_rec_size;/* userspace bpf_cleanup_info size */ +__u32 cleanup_info_cnt;/* number of bpf_cleanup_info records */ };struct {/* anonymous struct used by BPF_OBJ_* commands */
@@ -7599,6 +7602,12 @@ struct bpf_line_info {__u32 line_col; }; +struct bpf_cleanup_info { +__u32 begin_off; +__u32 end_off; +__u32 landing_pad_off; +}; +struct bpf_spin_lock {__u32 val; };
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v3 01/20] bpf: Accept the compiler's exception cleanup table at program load
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] bpf: unbounded memory allocation in check_cleanup_info via cleanup_info_cnt --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] bpf: unbounded memory allocation in check_cleanup_info via cleanup_info_cnt --
commit 18ab7230f0d26780fd3b40453aca2155a679f967
Author: Yonghong Song <[email protected]>
bpf: Accept the compiler's exception cleanup table at program load
This commit adds support for BPF exception handling cleanup tables generated by LLVM. It introduces a new UAPI to pass the .bpf_cleanup table into the kernel via BPF_PROG_LOAD, along with the necessary validation logic in check_cleanup_info().
Quoted 4 lines
kernel/bpf/check_btf.c +0
-0
[ ... ]
Quoted 6 lines
kernel/bpf/check_btf.c +2
-0@@ -407,6 +407,149 @@ static int check_core_relo(struct bpf_verifier_env *env,return err; } +static int cleanup_insn_subprog (struct bpf_verifier_env *env,u32 off) +{
[ ... ]
Quoted 5 lines
kernel/bpf/check_btf.c +5
-0+static int check_cleanup_info (struct bpf_verifier_env *env, +const union bpf_attr *attr, +bpfptr_t uattr) +{ +u32 krec_size =sizeof (struct bpf_cleanup_info );
kernel/bpf/check_btf.c static int check_core_relo(struct bpf_verifier_env *env, · line 437
Quoted 10 lines
kernel/bpf/check_btf.c +10
-0+u32 i, nrec, urec_size, min_size, prev_end =0 ; +struct bpf_cleanup_info *krecord; +bpfptr_t urecord; +int ret = -EINVAL ; + + nrec = attr->cleanup_info_cnt; +if (!nrec) +return 0 ; +if (nrec >INT_MAX / krec_size) +return -EINVAL ;
[Severity: Medium]
Can an unprivileged user trigger a multi-gigabyte memory allocation here by passing an artificially large attr->cleanup_info_cnt?
While the allocation below uses GFP_KERNEL_ACCOUNT and will not cause a CPU lockup, kvcalloc() still attempts to allocate and zero up to ~2.1 GB of memory.
Since each cleanup info record must cover at least one instruction and they cannot overlap, is there a case where nrec could legitimately exceed env->prog->len? Would it be safer to add a bounds check like if (nrec > env->prog->len) before the allocation to prevent this resource churn?
Quoted 10 lines
kernel/bpf/check_btf.c +10
-0+ + urec_size = attr->cleanup_info_rec_size; +if (urec_size <MIN_BPF_CLEANUP_INFO_SIZE || + urec_size >MAX_CLEANUP_INFO_REC_SIZE || + urec_size %sizeof (u32)) { +verbose (env,"invalid cleanup info rec size %u\n" , urec_size); +return -EINVAL ; + } + + krecord =kvcalloc (nrec, krec_size,GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Yonghong Song
reviewerRe: [PATCH bpf-next v3 01/20] bpf: Accept the compiler's exception cleanup table at program load
Quoted 53 lines · On 9/19/26 10:56 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] bpf: unbounded memory allocation incheck_cleanup_infoviacleanup_info_cnt--commit
18ab7230f0d26780fd3b40453aca2155a679f967
Author: Yonghong Song <[email protected]>bpf: Accept the compiler's exception cleanup table at program load
This commit adds support for BPF exception handling cleanup tables generated by LLVM. It introduces a new UAPI to pass the .bpf_cleanup table into the kernel via
BPF_PROG_LOAD, along with the necessary validation logic incheck_cleanup_info().Quoted 4 lines
kernel/bpf/check_btf.c +0
-0[ ... ]
Quoted 6 lines
kernel/bpf/check_btf.c +2
-0@@ -407,6 +407,149 @@ static int check_core_relo(struct bpf_verifier_env *env,return err; } +static int cleanup_insn_subprog (struct bpf_verifier_env *env,u32 off) +{[ ... ]
Quoted 5 lines
kernel/bpf/check_btf.c +5
-0+static int check_cleanup_info (struct bpf_verifier_env *env, +const union bpf_attr *attr, +bpfptr_t uattr) +{ +u32 krec_size =sizeof (struct bpf_cleanup_info );kernel/bpf/check_btf.c static int check_core_relo(struct bpf_verifier_env *env, · line 437
Quoted 10 lines
kernel/bpf/check_btf.c +10
-0+u32 i, nrec, urec_size, min_size, prev_end =0 ; +struct bpf_cleanup_info *krecord; +bpfptr_t urecord; +int ret = -EINVAL ; + + nrec = attr->cleanup_info_cnt; +if (!nrec) +return 0 ; +if (nrec >INT_MAX / krec_size) +return -EINVAL ;[Severity: Medium]
Can an unprivileged user trigger a multi-gigabyte memory allocation here by passing an artificially largeattr->cleanup_info_cnt?While the allocation below uses
GFP_KERNEL_ACCOUNTand will not cause a CPU lockup,kvcalloc()still attempts to allocate and zero up to ~2.1 GB of memory.
Quoted 5 lines
Since each cleanup info record must cover at least one instruction and they cannot overlap, is there a case where nrec could legitimately exceed
env->prog->len? Would it be safer to add a bounds check like if (nrec >env->prog->len) before the allocation to prevent this resource churn?
Okay, I will use the above condition to limit the upper bound of memory allocation.
Quoted 13 lines
Quoted 11 lines
kernel/bpf/check_btf.c +10
-0+ + urec_size = attr->cleanup_info_rec_size; +if (urec_size <MIN_BPF_CLEANUP_INFO_SIZE || + urec_size >MAX_CLEANUP_INFO_REC_SIZE || + urec_size %sizeof (u32)) { +verbose (env,"invalid cleanup info rec size %u\n" , urec_size); +return -EINVAL ; + } + + krecord =kvcalloc (nrec, krec_size,GFP_KERNEL_ACCOUNT | __GFP_NOWARN);[ ... ]
[email protected]
AI reviewRe: [PATCH bpf-next v3 01/20] bpf: Accept the compiler's exception cleanup table at program load
In reply to Yonghong SongThis isn't a bug, but would it be worth a short comment on struct bpf_cleanup_info spelling out that the offsets are instruction indices, that the range is [begin_off, end_off), and that records…
Quoted 15 lines
include/uapi/linux/bpf.h +3
-0@@ -1669,6 +1669,9 @@ union bpf_attr {* verification. */ __s32 keyring_id; +__aligned_u64 cleanup_info;/* exception cleanup table */ +__u32 cleanup_info_rec_size;/* userspace bpf_cleanup_info size */ +__u32 cleanup_info_cnt;/* number of bpf_cleanup_info records */ };struct {/* anonymous struct used by BPF_OBJ_* commands */
include/uapi/linux/bpf.h struct bpf_line_info { · line 7610
Quoted 9 lines
include/uapi/linux/bpf.h +6
-0@@ -7599,6 +7602,12 @@ struct bpf_line_info {__u32 line_col; }; +struct bpf_cleanup_info { +__u32 begin_off; +__u32 end_off; +__u32 landing_pad_off; +}; +
This isn't a bug, but would it be worth a short comment on struct bpf_cleanup_info spelling out that the offsets are instruction indices, that the range is [begin_off, end_off), and that records must arrive sorted and non-overlapping with the landing pad outside every range?
A producer other than LLVM has to read check_cleanup_info() to discover the rules, and insn_off in struct bpf_core_relo in the same header is documented as a byte offset (line 7750: "instruction offset (in bytes)"), so the unit is genuinely ambiguous at a glance.
[ ... ]
Quoted 5 lines
tools/include/uapi/linux/bpf.h +0
-0
tools/include/uapi/linux/bpf.h struct bpf_line_info { · line 7610
Quoted 9 lines
tools/include/uapi/linux/bpf.h +6
-0@@ -7599,6 +7602,12 @@ struct bpf_line_info {__u32 line_col; }; +struct bpf_cleanup_info { +__u32 begin_off; +__u32 end_off; +__u32 landing_pad_off; +}; +
Same here.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerRe: [PATCH bpf-next v3 01/20] bpf: Accept the compiler's exception cleanup table at program load
Quoted 18 lines · On 9/19/26 11:32 PM, [email protected] wrote:
Quoted 18 lines
include/uapi/linux/bpf.h +3
-0@@ -1669,6 +1669,9 @@ union bpf_attr {* verification. */ __s32 keyring_id; +__aligned_u64 cleanup_info;/* exception cleanup table */ +__u32 cleanup_info_rec_size;/* userspace bpf_cleanup_info size */ +__u32 cleanup_info_cnt;/* number of bpf_cleanup_info records */ };struct {/* anonymous struct used by BPF_OBJ_* commands */ @@ -7599,6 +7602,12 @@ struct bpf_line_info {__u32 line_col; };
Quoted 10 lines
include/uapi/linux/bpf.h struct bpf_line_info { · line 7610
Quoted 6 lines
include/uapi/linux/bpf.h +6
-0+struct bpf_cleanup_info { +__u32 begin_off; +__u32 end_off; +__u32 landing_pad_off; +}; +This isn't a bug, but would it be worth a short comment on struct
bpf_cleanup_infospelling out that the offsets are instruction indices, that the range is [begin_off,end_off), and that records must arrive sorted and non-overlapping with the landing pad outside every range?
The above bpf_attr already has related information.
Quoted 30 lines
A producer other than LLVM has to read
check_cleanup_info()to discover the rules, andinsn_offin structbpf_core_reloin the same header is documented as a byte offset (line 7750: "instruction offset (in bytes)"), so the unit is genuinely ambiguous at a glance.[ ... ]
Quoted 5 lines
tools/include/uapi/linux/bpf.h +0
-0tools/include/uapi/linux/bpf.h struct bpf_line_info { · line 7610
Quoted 9 lines
tools/include/uapi/linux/bpf.h +6
-0@@ -7599,6 +7602,12 @@ struct bpf_line_info {__u32 line_col; }; +struct bpf_cleanup_info { +__u32 begin_off; +__u32 end_off; +__u32 landing_pad_off; +}; +Same here.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerPATCH v3 2/20 bpf: Add the bpf_unwind_resume() kfunc
In reply to Yonghong SongA compiler-emitted cleanup landing pad ends with a call to _Unwind_Resume(), which carries the unwind on once that frame's cleanups have run. The kernel provides the same terminator as a kfunc, named bpf_unwind_resume() to keep the 'bpf_' prefix kfunc convention; a later libbpf patch resolves the compiler's name to it.
The body never runs. A JIT emits the way back out of a landing pad in its place.
--- kernel/bpf/helpers.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/bpf/helpers.c +12 -0
@@ -3445,6 +3445,17 @@ __bpf_kfunc void bpf_throw(u64 cookie)WARN (1 ,"A call to BPF exception callback should never return\n" ); } +/* +* Terminator of a compiler-emitted cleanup landing pad. The compiler names +* this _Unwind_Resume, the base unwind ABI's entry point for carrying an +* unwind on once a frame's cleanups have run. To match kernel kfunc +* convention, the kernel calls it bpf_unwind_resume and libbpf maps the +* compiler's name onto it. +*/ +__bpf_kfunc void bpf_unwind_resume(void) +{ +} + __bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, unsigned int flags) {struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
@@ -4981,6 +4992,7 @@ BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_throw) +BTF_ID_FLAGS(func, bpf_unwind_resume)#ifdef CONFIG_BPF_EVENTS BTF_ID_FLAGS(func, bpf_send_signal_task)#endif
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 3/20 bpf: Add lookups for exception cleanup resumes and landing pads
In reply to Yonghong SongAdd two new files, exception.h and exception.c, to host the exception handling code. Only a few helpers so far: recognising a call to bpf_unwind_resume(), and asking which landing pad, if any, a call site unwinds to.
The pad of a call site is kept in insn_aux_data, so the two places that move instructions around -- bpf_patch_insn_data() and verifier_remove_insns() -- learn to keep it in step.
--- include/linux/bpf_verifier.h | 6 ++++++ kernel/bpf/Makefile | 2 +- kernel/bpf/exception.c | 36 ++++++++++++++++++++++++++++++++++++ kernel/bpf/exception.h | 12 ++++++++++++ kernel/bpf/fixups.c | 24 +++++++++++++++++++++--- 5 files changed, 76 insertions(+), 4 deletions(-) create mode 100644 kernel/bpf/exception.c create mode 100644 kernel/bpf/exception.h
include/linux/bpf_verifier.h +6 -0
@@ -681,6 +681,11 @@ struct bpf_insn_aux_data {bool needs_zext;/* alu op needs to clear upper bits */ bool non_sleepable;/* helper/kfunc may be called from non-sleepable context */ bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ +/* +* 1 + the instruction index of the exception cleanup landing pad this +* call site unwinds to, or 0 for none. +*/ +u32 cleanup_pad;bool call_with_percpu_alloc_ptr;/* {this,per}_cpu_ptr() with prog percpu alloc */ u8 alu_state;/* used in combination with alu_limit */ /* true if STX or LDX instruction is a part of a spill/fill
@@ -1518,6 +1523,7 @@ u32 btf_func_arg_align(const struct btf *btf, const struct btf_type *t);int bpf_find_subprog (struct bpf_verifier_env *env,int off);bool bpf_is_throw_kfunc (struct bpf_insn *insn); +bool bpf_is_unwind_resume_kfunc (const struct bpf_insn *insn);int bpf_compute_const_regs (struct bpf_verifier_env *env);int bpf_prune_dead_branches (struct bpf_verifier_env *env);int bpf_check_cfg (struct bpf_verifier_env *env);
kernel/bpf/Makefile +1 -1
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_ obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o -obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o +obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o exception.o obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o obj-$(CONFIG_BPF_JIT) += trampoline.o
kernel/bpf/exception.c new file +36 -0
@@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <linux/bpf.h> +#include <linux/bpf_verifier.h> +#include <linux/btf.h> +#include <linux/btf_ids.h> +#include <linux/filter.h> +#include <linux/slab.h> +#include "exception.h" + +#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args) + +enum exc_kfunc { + EXC_KF_bpf_unwind_resume, +}; + +BTF_ID_LIST (exc_kfunc_list) +BTF_ID (func, bpf_unwind_resume) + +static bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf) +{ + return bpf_pseudo_kfunc_call(insn) && insn->off == 0 && + insn->imm == exc_kfunc_list[kf]; +} + +bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn) +{ +return insn_is_exc_kfunc (insn, EXC_KF_bpf_unwind_resume); +} + +int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx) +{ +u32 pad = env->insn_aux_data[idx].cleanup_pad; + +return pad ? (int )pad -1 :-1 ; +}
kernel/bpf/exception.h new file +12 -0
@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#ifndef _LINUX_BPF_EXCEPTION_H +#define _LINUX_BPF_EXCEPTION_H + +#include <linux/types.h> + +struct bpf_verifier_env ; + +int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx); + +#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +21 -3
@@ -261,6 +261,11 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env, } } +if (env->cleanup_info_cnt) +for (i =0 ; i < prog_len; i++) +if (data[i].cleanup_pad > off +1 ) + data[i].cleanup_pad += cnt -1 ; +/* * Last slot instruction could be a newly generated * BPF_ST/BPF_LDX/BPF_STX, systematically mark it for non-stack access
@@ -544,11 +549,13 @@ void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len) } } -static int verifier_remove_insns (struct bpf_verifier_env *env,u32 off,u32 cnt) +static int verifier_remove_insns (struct bpf_verifier_env *env,u32 off,u32 cnt, +bool falls_through) {struct bpf_insn_aux_data *aux_data = env->insn_aux_data;unsigned int orig_prog_len = env->prog->len;int err; +u32 i;if (bpf_prog_is_offloaded (env->prog->aux))bpf_prog_offload_remove_insns (env, off, cnt);
@@ -573,6 +580,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) sizeof(*aux_data) * (orig_prog_len - off - cnt)); env->insn_aux_data_len -= cnt; +if (env->cleanup_info_cnt) { +for (i =0 ; i < env->insn_aux_data_len; i++) { +u32 pad = aux_data[i].cleanup_pad; + +if (pad > off + cnt) + aux_data[i].cleanup_pad = pad - cnt; +else if (pad > off) + aux_data[i].cleanup_pad = falls_through ? off +1 :0 ; + } + } +return 0 ; }
@@ -634,7 +652,7 @@ int bpf_opt_remove_dead_code(struct bpf_verifier_env *env)if (!j)continue ; - err =verifier_remove_insns (env, i, j); + err =verifier_remove_insns (env, i, j, false);if (err)return err; insn_cnt = env->prog->len;
@@ -657,7 +675,7 @@ int bpf_opt_remove_nops(struct bpf_verifier_env *env)if (!is_may_goto_0 && !is_ja)continue ; - err =verifier_remove_insns (env, i,1 ); + err =verifier_remove_insns (env, i,1 , true);if (err)return err; insn_cnt--;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v3 03/20] bpf: Add lookups for exception cleanup resumes and landing pads
Does this miss the case where the original instruction is not placed last in the patch buffer? Looking at adjust_insn_aux_data(), there's logic earlier in the function that handles non_stack_access…
Quoted 16 lines
include/linux/bpf_verifier.h +5
-0@@ -681,6 +681,11 @@ struct bpf_insn_aux_data {bool needs_zext;/* alu op needs to clear upper bits */ bool non_sleepable;/* helper/kfunc may be called from non-sleepable context */ bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ +/* +* 1 + the instruction index of the exception cleanup landing pad this +* call site unwinds to, or 0 for none. +*/ +u32 cleanup_pad;bool call_with_percpu_alloc_ptr;/* {this,per}_cpu_ptr() with prog percpu alloc */ u8 alu_state;/* used in combination with alu_limit */ /* true if STX or LDX instruction is a part of a spill/fill
[ ... ]
kernel/bpf/fixups.c static void adjust_insn_aux_data(struct bpf_verifier_env *env, · line 268
Quoted 13 lines
kernel/bpf/fixups.c +5
-0@@ -261,6 +261,11 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env, } } +if (env->cleanup_info_cnt) +for (i =0 ; i < prog_len; i++) +if (data[i].cleanup_pad > off +1 ) + data[i].cleanup_pad += cnt -1 ; +
Does this miss the case where the original instruction is not placed last in the patch buffer? Looking at adjust_insn_aux_data(), there's logic earlier in the function that handles non_stack_access when the patched instructions contain the original:
kernel/bpf/fixups.c:adjust_insn_aux_data() {
for (i = off; i < off + cnt - 1; i++) {
/* Expand insni[off]'s seen count to the patched range. */
data[i].seen = old_seen;
data[i].zext_dst = bpf_insn_def32(new_prog, insn + i) >= 0;
if (!memcmp(insn + i, original_insn, sizeof(struct bpf_insn))) {
data[i].non_stack_access =
data[off + cnt - 1].non_stack_access;
data[off + cnt - 1].non_stack_access = false;
} else if (bpf_is_mem_insn(insn + i)) {
data[i].non_stack_access = true;
}
}
}The memcmp branch transfers non_stack_access from data[off + cnt - 1] to data[i] when it finds the original instruction. A bit further down, indirect_target gets similar treatment.
cleanup_pad is a property of one specific call instruction (the landing pad this call site unwinds to), not of an instruction slot. When bpf_patch_insn_data() parks the original instruction's aux record in data[off + cnt - 1] but the patch buffer places the original instruction at position i (not last), shouldn't cleanup_pad be transferred the same way?
Without this transfer, after such a patch the surviving call instruction would have cleanup_pad == 0 (no landing pad) and an unrelated newly generated instruction would carry the pad index instead. The CFG edge from the call to its pad would disappear, and an exception unwinding through that frame would silently skip its cleanup.
A later commit (d710bf82c5e9, "bpf: Dispatch exception cleanup pads from bpf_throw()") adds these lines to the memcmp branch:
data[i].cleanup_pad = data[off + cnt - 1].cleanup_pad;
data[off + cnt - 1].cleanup_pad = 0;Should that transfer belong in this commit, which introduces the field and claims to keep it in step?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerRe: [PATCH bpf-next v3 03/20] bpf: Add lookups for exception cleanup resumes and landing pads
Quoted 68 lines · On 9/19/26 11:31 PM, [email protected] wrote:
Quoted 16 lines
include/linux/bpf_verifier.h +5
-0@@ -681,6 +681,11 @@ struct bpf_insn_aux_data {bool needs_zext;/* alu op needs to clear upper bits */ bool non_sleepable;/* helper/kfunc may be called from non-sleepable context */ bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ +/* +* 1 + the instruction index of the exception cleanup landing pad this +* call site unwinds to, or 0 for none. +*/ +u32 cleanup_pad;bool call_with_percpu_alloc_ptr;/* {this,per}_cpu_ptr() with prog percpu alloc */ u8 alu_state;/* used in combination with alu_limit */ /* true if STX or LDX instruction is a part of a spill/fill [ ... ]
kernel/bpf/fixups.c static void adjust_insn_aux_data(struct bpf_verifier_env *env, · line 268
Quoted 13 lines
kernel/bpf/fixups.c +5
-0@@ -261,6 +261,11 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env, } } +if (env->cleanup_info_cnt) +for (i =0 ; i < prog_len; i++) +if (data[i].cleanup_pad > off +1 ) + data[i].cleanup_pad += cnt -1 ; +Does this miss the case where the original instruction is not placed last in the patch buffer? Looking at
adjust_insn_aux_data(), there's logic earlier in the function that handlesnon_stack_accesswhen the patched instructions contain the original:kernel/bpf/fixups.c:adjust_insn_aux_data() { for (i = off; i < off + cnt - 1; i++) { /* Expand insni[off]'s seen count to the patched range. */ data[i].seen = old_seen; data[i].zext_dst = bpf_insn_def32(new_prog, insn + i) >= 0; if (!memcmp(insn + i, original_insn, sizeof(struct bpf_insn))) { data[i].non_stack_access = data[off + cnt - 1].non_stack_access; data[off + cnt - 1].non_stack_access = false; } else if (bpf_is_mem_insn(insn + i)) { data[i].non_stack_access = true; } } }The memcmp branch transfers
non_stack_accessfrom data[off + cnt - 1] to data[i] when it finds the original instruction. A bit further down,indirect_targetgets similar treatment.
cleanup_padis a property of one specific call instruction (the landing pad this call site unwinds to), not of an instruction slot. Whenbpf_patch_insn_data()parks the original instruction's aux record in data[off + cnt - 1] but the patch buffer places the original instruction at position i (not last), shouldn'tcleanup_padbe transferred the same way?Without this transfer, after such a patch the surviving call instruction would have
cleanup_pad== 0 (no landing pad) and an unrelated newly generated instruction would carry the pad index instead. The CFG edge from the call to its pad would disappear, and an exception unwinding through that frame would silently skip its cleanup.
Quoted 8 lines
A later commit (
d710bf82c5e9, "bpf: Dispatch exception cleanup pads frombpf_throw()") adds these lines to the memcmp branch:data[i].cleanup_pad = data[off + cnt - 1].cleanup_pad; data[off + cnt - 1].cleanup_pad = 0;Should that transfer belong in this commit, which introduces the field and claims to keep it in step?
You are right. We can transfer the above cleanup_pad in this commit to keep the field consistent.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerPATCH v3 4/20 bpf: Prepare for an exception cleanup table before the CFG walk
In reply to Yonghong SongSome plumbing work is done before bpf_check_cfg(). More specifically, insn_aux_data records cleanup_throw_site for every bpf_throw() and cleanup_resume_site for every bpf_unwind_resume() -- the two calls a JIT lowers its own way rather than as calls -- and cleanup_pad, the landing pad a frame resumes at, for every call within the [begin_off, end_off) range of a cleanup record. Subsequent commits consume all three.
Marking the two calls here, rather than recognising them in the JIT, is what makes the recognition exact: by the time a JIT runs, bpf_fixup_kfunc_call() has rewritten every other kfunc's imm into an offset from __bpf_call_base, and a BTF id compared against one of those offsets could match an unrelated call.
bpf_prepare_cleanup_exceptions() runs before bpf_check_cfg(), because what it produces is what the CFG walk consumes. It refuses a table on an offloaded program, on a program whose JIT cannot dispatch landing pads or which the JIT was not asked to compile, and on a program that also installs an exception callback -- two different answers to what runs on the way out. bpf_jit_supports_cleanup_pads() is weak here and says no; the arch patches provide the real ones.
--- include/linux/bpf_verifier.h | 2 ++ include/linux/filter.h | 1 + kernel/bpf/core.c | 5 +++ kernel/bpf/exception.c | 60 ++++++++++++++++++++++++++++++++++++ kernel/bpf/exception.h | 1 + kernel/bpf/verifier.c | 6 ++++ 6 files changed, 75 insertions(+)
include/linux/bpf_verifier.h +2 -0
@@ -681,6 +681,8 @@ struct bpf_insn_aux_data {bool needs_zext;/* alu op needs to clear upper bits */ bool non_sleepable;/* helper/kfunc may be called from non-sleepable context */ bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ +bool cleanup_throw_site;/* call to bpf_throw() */ +bool cleanup_resume_site;/* call to bpf_unwind_resume() */ /* * 1 + the instruction index of the exception cleanup landing pad this * call site unwinds to, or 0 for none.
include/linux/filter.h +1 -0
@@ -1242,6 +1242,7 @@ bool bpf_jit_supports_stack_args(void);bool bpf_jit_supports_arena_args (void );bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void ); +bool bpf_jit_supports_cleanup_pads (void );bool bpf_jit_supports_ptr_xchg (void );bool bpf_jit_supports_arena (void );bool bpf_jit_supports_insn (struct bpf_insn *insn,bool in_arena);
kernel/bpf/core.c +5 -0
@@ -3475,6 +3475,11 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, { } +bool __weak bpf_jit_supports_cleanup_pads (void ) +{ +return false; +} +bool __weak bpf_jit_supports_timed_may_goto (void ) {return false;
kernel/bpf/exception.c +60 -0
66 lines not on this page · raw message
kernel/bpf/exception.h +1 -0
@@ -7,6 +7,7 @@struct bpf_verifier_env ; +int bpf_prepare_cleanup_exceptions (struct bpf_verifier_env *env);int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx);#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/verifier.c +6 -0
@@ -37,6 +37,7 @@#include "diagnostics.h" #include "disasm.h" +#include "exception.h" static const struct bpf_verifier_ops *const bpf_verifier_ops[] = { #define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
@@ -21677,6 +21678,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,if (ret <0 ) goto skip_full_check; +/* The CFG needs an edge from a call in a cleanup range to its pad. */ + ret =bpf_prepare_cleanup_exceptions (env); +if (ret <0 ) + goto skip_full_check; +/* Validate instructions and resolve the program's referenced resources. */ ret =check_and_resolve_insns (env);if (ret <0 )
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 5/20 bpf: Make exception landing pads reachable in the CFG
In reply to Yonghong SongBoth the CFG walk and liveness are involved. A bpf_throw() or a bpf2bpf call inside the [begin_off, end_off) range of a cleanup record can reach that record's landing pad, so both grow that edge; and a bpf_unwind_resume() reaches nothing after it, so it has no successors.
Liveness needs one more thing. bpf_stack_slot_alive() decides whether an outer frame's stack slot is still read after the call the frame is suspended at by looking at the instruction after the call. A slot whose only remaining reader is the landing pad -- which is every slot a compiler-generated pad reloads, since only four registers survive a call -- is dead by that measure, so clean_verifier_state() poisons it while the callee runs and the pad is then rejected for reading it. Ask about the pad as well when the call site names one.
--- kernel/bpf/cfg.c | 54 ++++++++++++++++++++++++++++++++++++++++--- kernel/bpf/liveness.c | 20 ++++++++++++++++ 2 files changed, 71 insertions(+), 3 deletions(-)
kernel/bpf/cfg.c +51 -3
@@ -6,6 +6,7 @@#include <linux/sort.h> #include "diagnostics.h" +#include "exception.h" #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@@ -158,17 +159,64 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)return DONE_EXPLORING ; } +static int visit_cleanup_pad_edge (int t,struct bpf_verifier_env *env) +{ +int *insn_stack = env->cfg.insn_stack; +int *insn_state = env->cfg.insn_state; +int w; + +if (!env->cleanup_info_cnt) +return DONE_EXPLORING ; + w =bpf_cleanup_pad_of_call (env, t); +if (w <0 ) +return DONE_EXPLORING ; + +/* +* @t is a call that may branch here, and @w is the target of that +* branch, so both are prune points. @w especially: every covered call +* site in a region unwinds to the same pad, and without a prune point +* at its head the verifier walks the pad again for each of them. +*/ +mark_prune_point (env, t); +mark_prune_point (env, w); +mark_jmp_point (env, w); +mark_jump_target (env, w); + +if (insn_state[w]) +return DONE_EXPLORING ; +if (env->cfg.cur_stack >= env->prog->len) +return -E2BIG ; + insn_stack[env->cfg.cur_stack++] = w; + insn_state[w] |=DISCOVERED ; +return KEEP_EXPLORING ; +} + +static int merge_visit_ret (int a,int b) +{ +if (a <0 ) +return a; +if (b <0 ) +return b; +if (a ==KEEP_EXPLORING || b ==KEEP_EXPLORING ) +return KEEP_EXPLORING ; +return DONE_EXPLORING ; +} +static int visit_func_call_insn (int t,struct bpf_insn *insns,struct bpf_verifier_env *env,bool visit_callee) { -int ret, insn_sz; +int ret, insn_sz, pad_ret;int w; + pad_ret =visit_cleanup_pad_edge (t, env); +if (pad_ret <0 ) +return pad_ret; + insn_sz =bpf_is_ldimm64 (&insns[t]) ?2 :1 ; ret =push_insn (t, t + insn_sz,FALLTHROUGH , env);if (ret) -return ret; +return merge_visit_ret (pad_ret, ret);mark_prune_point (env, t + insn_sz);/* when we exit from subprog, we need to record non-linear history */
@@ -180,7 +228,7 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,merge_callee_effects (env, t, w); ret =push_insn (t, w,BRANCH , env); } -return ret; +return merge_visit_ret (pad_ret, ret); }struct bpf_iarray *bpf_iarray_realloc (struct bpf_iarray *old,size_t n_elem)
kernel/bpf/liveness.c +20 -0
@@ -8,6 +8,8 @@#include <linux/slab.h> #include <linux/sort.h> +#include "exception.h" + #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)struct per_frame_masks {
@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) succ = env->succ; succ->cnt =0 ; +if (unlikely (bpf_is_unwind_resume_kfunc (insn))) +return succ; + opcode_info = &opcode_info_tbl[BPF_CLASS (insn->code) |BPF_OP (insn->code)]; insn_sz =bpf_is_ldimm64 (insn) ?2 :1 ;if (opcode_info->can_fallthrough)
@@ -264,6 +269,13 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)if (opcode_info->can_jump) succ->items[succ->cnt++] = idx +bpf_jmp_offset (insn) +1 ; +if (unlikely (env->cleanup_info_cnt)) { +int pad =bpf_cleanup_pad_of_call (env, idx); + +if (pad >=0 ) + succ->items[succ->cnt++] = pad; + } +return succ; }
@@ -397,6 +409,14 @@ bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp alive =bpf_calls_callback (env, callsite) ?is_live_before (instance, callsite, rel, half_spi) :is_live_before (instance, callsite +1 , rel, half_spi); + +/* Control may also go to the landing pad. */ +if (!alive &&unlikely (env->cleanup_info_cnt)) { +int pad =bpf_cleanup_pad_of_call (env, callsite); + +if (pad >=0 ) + alive =is_live_before (instance, pad, rel, half_spi); + }if (alive)return true; }
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 6/20 bpf: Explore the landing pads no call site reaches
In reply to Yonghong SongA cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving bpf_check_cfg() to refuse the program over code its own frontend had no way not to emit:
0: call bpf_preempt_disable
1: call bpf_preempt_enable record = { begin = 1, end = 2, pad = 4 }
2: r0 = 0
3: exit
4: r1 = pads_ran ll landing pad
6: r2 = *(u64 *)(r1 + 0)
7: r2 |= RAN_NOUNWIND_REC
8: *(u64 *)(r1 + 0) = r2
9: call bpf_unwind_resume
10: exit
The range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it. cleanup_mark_call_sites() marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.
Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other: do_check() never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.
--- kernel/bpf/cfg.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
kernel/bpf/cfg.c +17 -0
@@ -640,6 +640,7 @@ int bpf_check_cfg(struct bpf_verifier_env *env)int insn_cnt = env->prog->len;int *insn_stack, *insn_state;int ex_insn_beg, i, ret =0 ; +u32 pad_idx =0 ; insn_state = env->cfg.insn_state =kvzalloc_objs (int, insn_cnt,GFP_KERNEL_ACCOUNT );
@@ -695,6 +696,22 @@ int bpf_check_cfg(struct bpf_verifier_env *env) goto walk_cfg; } +/* +* A landing pad no call site was marked with -- a record whose range +* holds no call an exception can unwind out of -- is reached by +* nothing. Walk it from here, and let the dead code sweep remove it. +*/ +while (pad_idx < env->cleanup_info_cnt) { +u32 pad = env->cleanup_info[pad_idx++].landing_pad_off; + +if (insn_state[pad] !=EXPLORED ) { + insn_state[pad] =DISCOVERED ; + insn_stack[0 ] = pad; + env->cfg.cur_stack =1 ; + goto walk_cfg; + } + } +for (i =0 ; i < insn_cnt; i++) {struct bpf_insn *insn = &env->prog->insnsi[i];
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 7/20 bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch
In reply to Yonghong SongThe table is on the instructions and the landing pads are in the control flow graph. What is left before bpf_throw() can be taught to dispatch them is to work out what that walk will need, and refuse the shapes it could not handle.
bpf_check_cleanup_exceptions() runs after bpf_check_cfg(). Everything it needs is control flow, so it reads what that walk has already worked out rather than working it out again:
subprog_info.might_throw which subprograms an exception may leave,
closed over the call graph by merge_callee_effects() as the walk pops each callee
cleanup_reachability() what each instruction can reach -- a resume, a
plain exit, a throw, an indirect jump
cleanup_mark_pad_bodies() which instructions only ever run with an
exception already in flight
What it refuses:
- a pad that reaches both a resume and a plain exit, or neither: nothing says whether it is a cleanup pad or a catch pad
- a catch pad, which ends in a plain exit: the walker calls a pad as a subroutine and cannot hand a frame back its own execution
- a throw in a pad, or a call from a pad to a subprogram that can throw:
a second unwind over frames the first is still discarding
- a bpf_unwind_resume() outside a pad body
- a subprogram that may unwind used as a helper callback: the helper's own kernel frame would end the walk before it found a boundary
- a tail call, an indirect jump, or an outgoing on-stack call argument in a pad body, all of which touch a stack the pad does not own
- a BPF_LD_[ABS|IND] in a pad body: a failed load leaves the subprogram through the hidden exit gen_ld_abs() patches in, and an exit in a pad is the epilogue, which pops off the walker's stack and returns through the frame the walk is discarding
--- include/linux/bpf_verifier.h | 1 + kernel/bpf/exception.c | 405 +++++++++++++++++++++++++++++++++++ kernel/bpf/exception.h | 2 + kernel/bpf/fixups.c | 1 + kernel/bpf/verifier.c | 8 + 5 files changed, 417 insertions(+)
include/linux/bpf_verifier.h +1 -0
@@ -683,6 +683,7 @@ struct bpf_insn_aux_data {bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ bool cleanup_throw_site;/* call to bpf_throw() */ bool cleanup_resume_site;/* call to bpf_unwind_resume() */ +bool in_cleanup_pad;/* only runs with an exception in flight */ /* * 1 + the instruction index of the exception cleanup landing pad this * call site unwinds to, or 0 for none.
kernel/bpf/exception.c +405 -0
429 lines not on this page · raw message
kernel/bpf/exception.h +2 -0
@@ -8,6 +8,8 @@struct bpf_verifier_env ;int bpf_prepare_cleanup_exceptions (struct bpf_verifier_env *env); +int bpf_check_cleanup_exceptions (struct bpf_verifier_env *env); +int bpf_cleanup_check_callback (struct bpf_verifier_env *env,int subprog);int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx);#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +1 -0
@@ -252,6 +252,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,/* Expand insni[off]'s seen count to the patched range. */ data[i].seen = old_seen; data[i].zext_dst =bpf_insn_def32 (new_prog, insn + i) >=0 ; + data[i].in_cleanup_pad = data[off + cnt -1 ].in_cleanup_pad;if (!memcmp (insn + i, original_insn,sizeof (struct bpf_insn ))) { data[i].non_stack_access = data[off + cnt -1 ].non_stack_access;
kernel/bpf/verifier.c +8 -0
@@ -10503,6 +10503,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins* callbacks */ env->subprog_info[subprog].is_cb = true; + err =bpf_cleanup_check_callback (env, subprog); +if (err) +return err; +if (bpf_pseudo_kfunc_call (insn) && !is_callback_calling_kfunc (insn->imm)) { verifier_bug(env, "kfunc %s#%d not marked as callback-calling",
@@ -21703,6 +21707,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,if (ret <0 ) goto skip_full_check; + ret =bpf_check_cleanup_exceptions (env); +if (ret <0 ) + goto skip_full_check; + ret =bpf_compute_postorder (env);if (ret <0 ) goto skip_full_check;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v3 07/20] bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch
Does the in_cleanup_pad comment accurately describe what the field represents at this commit? The comment says "only runs with an exception in flight", but cleanup_mark_pad_bodies() marks…
include/linux/bpf_verifier.h struct bpf_insn_aux_data { · line 689
Quoted 12 lines
include/linux/bpf_verifier.h +1
-0@@ -683,6 +683,7 @@ struct bpf_insn_aux_data {bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ bool cleanup_throw_site;/* call to bpf_throw() */ bool cleanup_resume_site;/* call to bpf_unwind_resume() */ +bool in_cleanup_pad;/* only runs with an exception in flight */ /* * 1 + the instruction index of the exception cleanup landing pad this * call site unwinds to, or 0 for none.
Does the in_cleanup_pad comment accurately describe what the field represents at this commit? The comment says "only runs with an exception in flight", but cleanup_mark_pad_bodies() marks reachability from a landing pad without checking whether the pad is unreachable by ordinary control flow.
For example, with a record {begin_off=0, end_off=1, landing_pad_off=1}:
0: call pcall foo /* covered by the record */
1: r0 = 0 /* landing pad, also insn 0's fall-through */
2: call bpf_unwind_resume
3: exit
check_cleanup_info() permits this (it only rejects a pad that lies *inside* a call-site range). cleanup_reachability() gives pad 1 only CLEANUP_REACH_RESUME, cleanup_pad_is_catch() accepts it, and insns 1-2 get in_cleanup_pad = true. Yet on normal return from foo, execution falls into insns 1-2 with no exception in flight.
The next commit (c046fa312ea4 "bpf: Walk the exception unwind in the verifier") adds the state->unwinding check that actually refuses this shape ("bpf_unwind_resume() at insn %d reached without an exception in flight"). Should the field comment say "reachable from a landing pad" rather than "only runs with an exception in flight" until that enforcement is in place, or does the check belong in this commit since the changelog promises to refuse every cleanup shape bpf_throw() cannot dispatch?
Quoted 4 lines
kernel/bpf/exception.c +0
-0
[ ... ]
Quoted 1 line
kernel/bpf/exception.c +0
-0@@ -37,6 +162,255 @@ static void cleanup_mark_kfunc_sites(struct bpf_verifier_env *env)
[ ... ]
Quoted 25 lines
kernel/bpf/exception.c +25
-0+/* What every instruction can reach along intra-subprog edges, for +* cleanup_pad_is_catch(). One backward walk over a predecessor index, rather +* than a forward walk from each landing pad, which would be quadratic. +*/ +static int cleanup_reachability (struct cleanup_ctx *ctx) +{ +struct bpf_verifier_env *env = ctx->env; +u32 len = env->prog->len; +struct cleanup_cursor c =CLEANUP_CURSOR_INIT ; +u32 *head =NULL , *link =NULL ; +bool *queued =NULL ; +u32 i, sp =0 ; +void *scratch; +const struct cleanup_alloc_req tab[] = { + { (void **)&head, len,sizeof (*head) }, + { (void **)&link,2 * (size_t )len,sizeof (*link) }, + { (void **)&queued, len,sizeof (*queued) }, + }; + + scratch =cleanup_alloc (tab,ARRAY_SIZE (tab)); +if (!scratch) +return -ENOMEM ; + +/* Index the predecessors, and seed the walk at the terminators. */ +for (i =0 ; i < len; i++) {
[ ... ]
kernel/bpf/exception.c static void cleanup_mark_kfunc_sites(struct bpf_verifier_env *env) · line 247
Quoted 6 lines
kernel/bpf/exception.c +6
-0+ } + +/* Each instruction re-enters the worklist at most once per bit it +* gains, so this is linear in the number of edges. +*/ +while (sp) {
This isn't a bug, but could the two multi-line comments in cleanup_reachability() follow the BPF comment style with the opening /* on its own line? The other multi-line comments added by this series use that format, and the guide requires it for files under kernel/bpf/.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerRe: [PATCH bpf-next v3 07/20] bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch
Quoted 31 lines · On 9/19/26 11:32 PM, [email protected] wrote:
include/linux/bpf_verifier.h struct bpf_insn_aux_data { · line 689
Quoted 12 lines
include/linux/bpf_verifier.h +1
-0@@ -683,6 +683,7 @@ struct bpf_insn_aux_data {bool is_iter_next;/* bpf_iter_<type>_next() kfunc call */ bool cleanup_throw_site;/* call to bpf_throw() */ bool cleanup_resume_site;/* call to bpf_unwind_resume() */ +bool in_cleanup_pad;/* only runs with an exception in flight */ /* * 1 + the instruction index of the exception cleanup landing pad this * call site unwinds to, or 0 for none. Does the
in_cleanup_padcomment accurately describe what the field represents at this commit? The comment says "only runs with an exception in flight", butcleanup_mark_pad_bodies()marks reachability from a landing pad without checking whether the pad is unreachable by ordinary control flow.For example, with a record {begin_off=0, end_off=1, landing_pad_off=1}:
0: call pcall foo /* covered by the record */
1: r0 = 0 /* landing pad, also insn 0's fall-through */
2: callbpf_unwind_resume
3: exit
check_cleanup_info()permits this (it only rejects a pad that lies *inside* a call-site range).cleanup_reachability()gives pad 1 onlyCLEANUP_REACH_RESUME,cleanup_pad_is_catch()accepts it, and insns 1-2 getin_cleanup_pad= true. Yet on normal return from foo, execution falls into insns 1-2 with no exception in flight.
Quoted 7 lines
The next commit (
c046fa312ea4"bpf: Walk the exception unwind in the verifier") adds thestate->unwindingcheck that actually refuses this shape ("bpf_unwind_resume()at insn %d reached without an exception in flight"). Should the field comment say "reachable from a landing pad" rather than "only runs with an exception in flight" until that enforcement is in place, or does the check belong in this commit since the changelog promises to refuse every cleanup shapebpf_throw()cannot dispatch?
You are right. The comment is not good. Let us do:
bool in_cleanup_pad; /* reachable from an exception cleanup landing pad */
Quoted 54 lines
Quoted 5 lines
kernel/bpf/exception.c +0
-0[ ... ]
Quoted 1 line
kernel/bpf/exception.c +0
-0@@ -37,6 +162,255 @@ static void cleanup_mark_kfunc_sites(struct bpf_verifier_env *env)[ ... ]
Quoted 25 lines
kernel/bpf/exception.c +25
-0+/* What every instruction can reach along intra-subprog edges, for +* cleanup_pad_is_catch(). One backward walk over a predecessor index, rather +* than a forward walk from each landing pad, which would be quadratic. +*/ +static int cleanup_reachability (struct cleanup_ctx *ctx) +{ +struct bpf_verifier_env *env = ctx->env; +u32 len = env->prog->len; +struct cleanup_cursor c =CLEANUP_CURSOR_INIT ; +u32 *head =NULL , *link =NULL ; +bool *queued =NULL ; +u32 i, sp =0 ; +void *scratch; +const struct cleanup_alloc_req tab[] = { + { (void **)&head, len,sizeof (*head) }, + { (void **)&link,2 * (size_t )len,sizeof (*link) }, + { (void **)&queued, len,sizeof (*queued) }, + }; + + scratch =cleanup_alloc (tab,ARRAY_SIZE (tab)); +if (!scratch) +return -ENOMEM ; + +/* Index the predecessors, and seed the walk at the terminators. */ +for (i =0 ; i < len; i++) {[ ... ]
kernel/bpf/exception.c static void cleanup_mark_kfunc_sites(struct bpf_verifier_env *env) · line 247
Quoted 6 lines
kernel/bpf/exception.c +6
-0+ } + +/* Each instruction re-enters the worklist at most once per bit it +* gains, so this is linear in the number of edges. +*/ +while (sp) {This isn't a bug, but could the two multi-line comments in
cleanup_reachability()follow the BPF comment style with the opening /* on its own line? The other multi-line comments added by this series use that format, and the guide requires it for files underkernel/bpf/.---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerPATCH v3 8/20 bpf: Walk the exception unwind in the verifier
In reply to Yonghong Songbpf_throw() is about to stop discarding the frames it unwinds and start running their landing pads instead. Have the verifier walk the same thing, step for step:
at a throw, with the frame chain in env->cur_state->frame[]
if a record covers the call control is at
run that record's landing pad in this frame on reaching its resume, pop the frame and carry on
else
pop the frame and carry on
at the boundary, deliver
Doing it this way is what keeps the resource rules unchanged. Whatever a pad releases is released in the verifier state too, so by the time the walk reaches the boundary the state says exactly what the program will really hold there -- and check_resource_leak(), which used to fire the moment a throw was seen, simply moves to the end of the walk. A frame that no record covers contributes nothing, so anything it held is still held when the walk ends, and that is what gets reported.
Two things about that walk have to be said out loud, because the instruction stream does not say them.
A resume belongs to the frame whose landing pad the walker called. Each JIT lowers bpf_unwind_resume() as the way back out of a pad -- a bare return on x86-64, a branch to the saved address on arm64 -- and that reaches the walker only from there. An exception being in flight is not enough to tell:
a pad may call a subprogram that has a landing pad of its own and reaches it by ordinary control flow, and the resume in it is in a pad body by every static measure. So the state records the frame the unwind entered a pad in, and a resume in any other frame is refused.
The edge from a throw to a pad crosses frames with no instruction in between to account for them, and mark_chain_precision() walks that history backwards. Left alone it stays in the pad's frame while it reads the callee's instructions: a request for the pad frame's r6 is cleared by the callee's own write to r6 -- so the caller's definition never becomes precise -- or reaches the call instruction still set and trips "static subprog unexpected regs". The history entry for a pad therefore records how many frames the unwind popped, and the backtrack enters that many, the way it enters one at a time for BPF_EXIT. A throwing global subprogram gets no frame of its own, and its exception arrives at the landing pad rather than at the next instruction, which the global-call check has to allow for.
--- include/linux/bpf_cleanup_abi.h | 16 ++++ include/linux/bpf_verifier.h | 6 +- kernel/bpf/backtrack.c | 21 ++++- kernel/bpf/states.c | 6 ++ kernel/bpf/verifier.c | 143 ++++++++++++++++++++++++++------ 5 files changed, 163 insertions(+), 29 deletions(-) create mode 100644 include/linux/bpf_cleanup_abi.h
include/linux/bpf_cleanup_abi.h new file +16 -0
@@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#ifndef _LINUX_BPF_CLEANUP_ABI_H +#define _LINUX_BPF_CLEANUP_ABI_H + +/* +* Value arch_bpf_run_cleanup_pad() leaves in r0 on the way into a landing pad. +* It has to be a constant the verifier knows: LLVM names r0 as both the +* exception pointer and the exception selector register, so every pad reads it +* before anything else and is free to store what it read. Kept on its own +* because the verifier and the arch dispatchers, which are assembly, have to +* agree on it. +*/ +#define BPF_PAD_ENTRY_R0 1 + +#endif /* _LINUX_BPF_CLEANUP_ABI_H */
include/linux/bpf_verifier.h +5 -1
@@ -429,7 +429,8 @@ struct bpf_jmp_history_entry { u32 prev_idx : 20;/* special INSN_F_xxx flags */ u32 flags : 4; - u32 :8 ; + u32 unwind_frames : 4; /* frames the unwind popped to get here */ + u32 :4 ;/* * additional registers that need precision tracking when this * jump is backtracked, vector of five 11-bit records
@@ -509,6 +510,8 @@ struct bpf_verifier_state {bool speculative;bool in_sleepable; +bool unwinding;/* an exception is in flight */ +u8 unwind_frameno;/* the frame whose landing pad it entered */ /* first and last insn idx of this verifier state */ u32 first_insn_idx;
@@ -991,6 +994,7 @@ struct bpf_verifier_env { } cfg;struct backtrack_state bt;struct bpf_jmp_history_entry *cur_hist_ent; +u8 unwind_frames;/* scratch: pops for the insn about to be recorded */ /* Per-callsite copy of parent's converged at_stack_in for cross-frame fills. */ struct arg_track **callsite_at_stack;u32 pass_cnt;/* number of times do_check() was called */
kernel/bpf/backtrack.c +17 -4
@@ -4,6 +4,7 @@#include <linux/bpf_verifier.h> #include <linux/filter.h> #include <linux/bitmap.h> +#include "exception.h" #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@@ -47,6 +48,7 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state p->flags = insn_flags; p->spi = spi; p->frame = frame; + p->unwind_frames =0 ; p->linked_regs = linked_regs; cur->jmp_history_cnt = cnt; env->cur_hist_ent = p;
@@ -419,10 +421,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,* extra instructions from subprog; the next * instruction after call to global subprog * should be literally next instruction in -* caller program +* caller program -- or, if the callee threw, +* the landing pad of this call site */ -verifier_bug_if (idx +1 != subseq_idx, env, -"extra insn from subprog" ); +verifier_bug_if (idx +1 != subseq_idx && +bpf_cleanup_pad_of_call (env, idx) != subseq_idx, + env,"extra insn from subprog" );/* global subprog always sets R0 */ bt_clear_reg (bt,BPF_REG_0 );/* and if it does not set R2, main pass would catch it */
@@ -888,11 +892,11 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env, }for (i = last_idx;;) { + hist =get_jmp_hist_entry (st, history, i);if (skip_first) { err =0 ; skip_first = false; }else { - hist =get_jmp_hist_entry (st, history, i); err =backtrack_insn (env, i, subseq_idx, hist, bt); }if (err == -ENOTSUPP ) {
@@ -909,6 +913,15 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,*/ return 0 ; subseq_idx = i; +/* An exception unwind reached this insn, a landing +* pad, from a throw or a resume that many frames +* deeper. There is no insn in between to backtrack +* over, so enter those frames here, the way BPF_EXIT +* does one at a time. +*/ +for (fr =0 ; hist && fr < hist->unwind_frames; fr++) +if (bt_subprog_enter (bt)) +return -EFAULT ; i =get_prev_insn_idx (st, i, &history);if (i == -ENOENT )break ;
kernel/bpf/states.c +6 -0
@@ -996,6 +996,12 @@ static bool states_equal(struct bpf_verifier_env *env,if (old->in_sleepable != cur->in_sleepable)return false; +if (old->unwinding != cur->unwinding) +return false; + +if (old->unwinding && old->unwind_frameno != cur->unwind_frameno) +return false; +if (!refsafe (old, cur, &env->idmap_scratch))return false;
kernel/bpf/verifier.c +119 -24
220 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 9/20 bpf: Refuse a private stack for a program with an exception cleanup table
In reply to Yonghong SongA private stack is hard to support together with an exception cleanup table on x86-64: a landing pad runs on the bpf_throw() walker's stack with its frame's registers put back from a spill area, and carrying the private stack's frame pointer through there as well would cost a register that architecture cannot spare.
Refuse the combination in check_max_stack_depth(), where the choice is made, and refuse it on every architecture for now rather than just x86-64.
--- kernel/bpf/verifier.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/bpf/verifier.c +8 -0
@@ -5592,6 +5592,14 @@ static int check_max_stack_depth(struct bpf_verifier_env *env) } } +/* +* A pad rebuilds its frame from a spill area, and on x86-64 a private +* stack's frame pointer lives in r9, which no spill area holds. +* Refused on every arch rather than just that one. +*/ +if (env->cleanup_info_cnt) + priv_stack_mode =NO_PRIV_STACK ; +if (priv_stack_mode ==PRIV_STACK_UNKNOWN ) priv_stack_mode =bpf_enable_priv_stack (env->prog);
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 10/20 bpf: Dispatch exception cleanup pads from bpf_throw()
In reply to Yonghong SongStop discarding the frames an exception unwinds through. bpf_throw() already walks the BPF call stack with arch_bpf_stack_walk() to find the exception boundary; have it look each frame's return address up in that (sub)program's cleanup table on the way and run the landing pad a matching record names.
A pad is run as a subroutine of the walker, not jumped to. It executes with the unwinding frame's frame pointer and BPF callee-saved registers, so everything it reads is that frame's, but on the current stack far below it, so nothing it calls can disturb the frame it is cleaning up after. It ends in what the JIT emits for its bpf_unwind_resume(), which hands control back to the walker rather than to the frame's caller. That is exactly the procedure the LLVM commit emitting the section describes.
Restoring the frame's r6-r9 is what makes this work, and it is only possible because the callee about to be discarded spilled them in its own prologue. bpf_cleanup_force_spill() tells a JIT to make that spill unconditional and of a known shape for every subprogram of a program carrying a cleanup table, and aux->exc->spill_off records where it starts, so the walker needs no per-frame metadata. The frame that called bpf_throw() has no callee to have spilled anything and never runs its own epilogue, so the JIT spills that frame's registers at the throw site instead, in the area aux->exc->throw_spill_off names.
The main program needs the table handed to it rather than built for it. jit_subprogs() compiles it as func[0], but the ksym covering that image is the one bpf_prog_load() registers for the outer bpf_prog, and that is what the walker finds -- so without the handover a landing pad in the main program's own frame is never dispatched, silently, since the exception still reaches the boundary and the cookie still comes back.
Which instructions a JIT has to lower its own way reach it as indices into the (sub)program, aux->exc->throw_at and ->resume_at, carried there from the insn_aux_data marks so that patching keeps them pointing at the right instruction.
All of it hangs off bpf_prog_aux by a single pointer. bpf_prog_aux is on every BPF program and this is a niche feature, so struct bpf_exception_info exists only for a program that carries a table, and aux->exc being set is what says the program carries one at all.
--- include/linux/bpf.h | 84 +++++++++++++++++++++ include/linux/filter.h | 1 + kernel/bpf/core.c | 30 +++++++- kernel/bpf/exception.c | 163 +++++++++++++++++++++++++++++++++++++++++ kernel/bpf/exception.h | 7 ++ kernel/bpf/fixups.c | 141 +++++++++++++++++++++++++++++++++++ kernel/bpf/helpers.c | 34 +++++++++ 7 files changed, 457 insertions(+), 3 deletions(-)
include/linux/bpf.h +84 -0
@@ -1774,6 +1774,89 @@ enum bpf_sig_keyring { BPF_SIG_KEYRING_BPF, }; +/* +* One cleanup region of a JITed (sub)program: @pad is the landing pad to run +* for a return address in (begin, end], the native code of its call sites. +*/ +struct bpf_cleanup_range { +u64 begin; +u64 end; +u64 pad; +}; + +struct bpf_exception_info { +struct bpf_cleanup_info *info; +struct bpf_cleanup_range *ranges; +/* Landing pad instruction indices, sorted and deduplicated. */ +u32 *pad_at; +/* bpf_throw() call instruction indices, sorted. */ +u32 *throw_at; +/* Likewise for bpf_unwind_resume(), the way back out of a pad. */ +u32 *resume_at; +/* One bit per instruction that only runs while unwinding. */ +unsigned long *pad_body; +u32 nr_info; +u32 nr_ranges; +u32 nr_pad_at; +u32 nr_throw_at; +u32 nr_resume_at; +u32 nr_pad_body; +/* Offset from a frame's FP to the caller's spilled r6-r9. */ +s32 spill_off; +/* Likewise, to the registers a frame spills before calling bpf_throw(). */ +s32 throw_spill_off; +}; + +#ifdef CONFIG_BPF_SYSCALL +bool bpf_cleanup_force_spill (const struct bpf_prog *prog); +bool bpf_cleanup_insn_is_pad (const struct bpf_prog *prog,u32 idx); +bool bpf_cleanup_insn_in_pad (const struct bpf_prog *prog,u32 idx); +bool bpf_cleanup_insn_is_throw (const struct bpf_prog *prog,u32 idx); +bool bpf_cleanup_insn_is_resume (const struct bpf_prog *prog,u32 idx); +int bpf_cleanup_attach_main_prog (struct bpf_verifier_env *env,struct bpf_prog *prog); +void bpf_cleanup_fill_native_ranges (struct bpf_prog *prog,u32 *addrs,void *image); +void bpf_cleanup_free_info (struct bpf_prog_aux *aux); +#else +static inline bool bpf_cleanup_force_spill (const struct bpf_prog *prog) +{ +return false; +} + +static inline bool bpf_cleanup_insn_is_pad (const struct bpf_prog *prog,u32 idx) +{ +return false; +} + +static inline bool bpf_cleanup_insn_in_pad (const struct bpf_prog *prog,u32 idx) +{ +return false; +} + +static inline bool bpf_cleanup_insn_is_throw (const struct bpf_prog *prog,u32 idx) +{ +return false; +} + +static inline bool bpf_cleanup_insn_is_resume (const struct bpf_prog *prog,u32 idx) +{ +return false; +} + +static inline int bpf_cleanup_attach_main_prog (struct bpf_verifier_env *env, +struct bpf_prog *prog) +{ +return 0 ; +} + +static inline void bpf_cleanup_fill_native_ranges (struct bpf_prog *prog,u32 *addrs,void *image) +{ +} + +static inline void bpf_cleanup_free_info (struct bpf_prog_aux *aux) +{ +} +#endif +struct bpf_prog_aux {atomic64_t refcnt;u32 used_map_cnt;
@@ -1854,6 +1937,7 @@ struct bpf_prog_aux {char name[BPF_OBJ_NAME_LEN ];u64 (*bpf_exception_cb)(u64 cookie,u64 sp,u64 bp,u64 ,u64 );u16 stack_arg_sp_adjust; +struct bpf_exception_info *exc;#ifdef CONFIG_SECURITY void *security;#endif
include/linux/filter.h +1 -0
@@ -1243,6 +1243,7 @@ bool bpf_jit_supports_arena_args(void);bool bpf_jit_supports_far_kfunc_call (void );bool bpf_jit_supports_exceptions (void );bool bpf_jit_supports_cleanup_pads (void ); +void arch_bpf_run_cleanup_pad (u64 pad,u64 frame_fp,u64 spill_base);bool bpf_jit_supports_ptr_xchg (void );bool bpf_jit_supports_arena (void );bool bpf_jit_supports_insn (struct bpf_insn *insn,bool in_arena);
kernel/bpf/core.c +27 -3
@@ -292,6 +292,7 @@ void __bpf_prog_free(struct bpf_prog *fp)mutex_destroy (&fp->aux->dst_mutex);mutex_destroy (&fp->aux->st_ops_assoc_mutex);kfree (fp->aux->poke_tab); +bpf_cleanup_free_info (fp->aux);kfree (fp->aux); }free_percpu (fp->stats);
@@ -2625,13 +2626,21 @@ static bool bpf_prog_select_interpreter(struct bpf_prog *fp)return select_interpreter; } -static struct bpf_prog *bpf_prog_jit_compile (struct bpf_verifier_env *env,struct bpf_prog *prog) +static struct bpf_prog *bpf_prog_jit_compile (struct bpf_verifier_env *env,struct bpf_prog *prog, +int *err) {#ifdef CONFIG_BPF_JIT struct bpf_prog *orig_prog; +int ret; -if (!bpf_prog_need_blind (prog)) +if (!bpf_prog_need_blind (prog)) { + ret =bpf_cleanup_attach_main_prog (env, prog); +if (ret) { + *err = ret; +return prog; + }return bpf_int_jit_compile (env, prog); + } orig_prog = prog; prog =bpf_jit_blind_constants (env, prog);
@@ -2642,6 +2651,13 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, strucif (IS_ERR (prog)) goto out_restore; + ret =bpf_cleanup_attach_main_prog (env, prog); +if (ret) { + *err = ret; +bpf_jit_prog_release_other (orig_prog, prog); + goto out_restore; + } + prog =bpf_int_jit_compile (env, prog);if (prog->jited) {bpf_jit_prog_release_other (prog, orig_prog);
@@ -2681,8 +2697,10 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, structif (*err)return fp; - fp =bpf_prog_jit_compile (env, fp); + fp =bpf_prog_jit_compile (env, fp, err);bpf_prog_jit_attempt_done (fp); +if (*err) +return fp;if (!fp->jited && jit_needed) { *err = -ENOTSUPP ;return fp;
@@ -3480,6 +3498,12 @@ bool __weak bpf_jit_supports_cleanup_pads(void)return false; } +/* Call @pad with the frame pointer @frame_fp and r6-r9 spilled at @spill_base. */ +void __weak arch_bpf_run_cleanup_pad (u64 pad,u64 frame_fp,u64 spill_base) +{ +WARN_ON_ONCE (1 ); +} +bool __weak bpf_jit_supports_timed_may_goto (void ) {return false;
kernel/bpf/exception.c +163 -0
173 lines not on this page · raw message
kernel/bpf/exception.h +7 -0
@@ -5,11 +5,18 @@#include <linux/types.h> +struct bpf_cleanup_info ; +struct bpf_cleanup_range ; +struct bpf_prog ; +struct bpf_prog_aux ;struct bpf_verifier_env ;int bpf_prepare_cleanup_exceptions (struct bpf_verifier_env *env);int bpf_check_cleanup_exceptions (struct bpf_verifier_env *env);int bpf_cleanup_check_callback (struct bpf_verifier_env *env,int subprog);int bpf_cleanup_pad_of_call (struct bpf_verifier_env *env,u32 idx); +int bpf_cleanup_alloc_info (struct bpf_prog_aux *aux); +int bpf_cleanup_attach_info (struct bpf_prog_aux *aux,struct bpf_cleanup_info *recs,u32 cnt); +const struct bpf_cleanup_range *bpf_cleanup_pad_for_ip (const struct bpf_prog *prog,u64 ip);#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +141 -0
182 lines not on this page · raw message
kernel/bpf/helpers.c +34 -0
@@ -31,6 +31,7 @@#include <linux/buildid.h> #include "../../lib/kstrtox.h" +#include "exception.h" /* If kernel subsystem is allowing eBPF programs to call this function, * inside its own verifier_ops->get_func_proto() callback it should return
@@ -3398,8 +3399,36 @@ struct bpf_throw_ctx {u64 sp;u64 bp;int cnt; +const struct bpf_prog *callee; +u64 callee_fp; }; +static void bpf_run_cleanup_pad (struct bpf_throw_ctx *ctx,const struct bpf_prog *prog, +u64 ip,u64 fp) +{ +const struct bpf_exception_info *exc = prog->aux->exc; +const struct bpf_cleanup_range *rec; +u64 spill_base; + +if (!exc || !exc->nr_ranges) +return ; + rec =bpf_cleanup_pad_for_ip (prog, ip); +if (!rec) +return ; + +/* +* The callee is always another subprogram of this program -- the walk +* ends at any frame that is not one -- so its prologue spilled these +* registers and its exc is there to say where. +*/ +if (ctx->callee) + spill_base = ctx->callee_fp + ctx->callee->aux->exc->spill_off; +else + spill_base = fp + exc->throw_spill_off; + +arch_bpf_run_cleanup_pad (rec->pad, fp, spill_base); +} +static bool bpf_stack_walker (void *cookie,u64 ip,u64 sp,u64 bp) {struct bpf_throw_ctx *ctx = cookie;
@@ -3416,6 +3445,11 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)if (!prog)return !ctx->cnt; ctx->cnt++; + +bpf_run_cleanup_pad (ctx, prog, ip, bp); + ctx->callee = prog; + ctx->callee_fp = bp; +if (bpf_is_subprog (prog))return true; ctx->aux = prog->aux;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v3 10/20] bpf: Dispatch exception cleanup pads from bpf_throw()
This isn't a bug, but bpf_cleanup_attach_info()'s only caller already returns early when cnt is zero - is the `if (!cnt)` arm here reachable, or would dropping it (and letting the caller keep owning…
Quoted 4 lines
include/linux/bpf.h +0
-0
[ ... ]
Quoted 4 lines
kernel/bpf/exception.c +0
-0
[ ... ]
Quoted 13 lines
kernel/bpf/exception.c +9
-0@@ -499,3 +501,164 @@ int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx)return pad ? (int )pad -1 :-1 ; } + +int bpf_cleanup_alloc_info (struct bpf_prog_aux *aux) +{ +if (aux->exc) +return 0 ; + aux->exc = kzalloc_obj(struct bpf_exception_info, GFP_KERNEL_ACCOUNT | __GFP_NOWARN); +return aux->exc ?0 : -ENOMEM ; +} +
kernel/bpf/exception.c int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx) · line 522
Quoted 10 lines
kernel/bpf/exception.c +10
-0+int bpf_cleanup_attach_info (struct bpf_prog_aux *aux,struct bpf_cleanup_info *recs,u32 cnt) +{ +struct bpf_exception_info *exc = aux->exc; +struct bpf_cleanup_range *ranges; +u32 i, n_at, *at; + +if (!cnt) { +kvfree (recs); +return 0 ; + }
This isn't a bug, but bpf_cleanup_attach_info()'s only caller already returns early when cnt is zero - is the if (!cnt) arm here reachable, or would dropping it (and letting the caller keep owning that case) make the recs ownership easier to follow?
Quoted 2 lines
kernel/bpf/exception.c +2
-0+ + ranges =kvcalloc (cnt,sizeof (*ranges),GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
[ ... ]
Quoted 4 lines
kernel/bpf/fixups.c +0
-0
[ ... ]
Quoted 18 lines
kernel/bpf/fixups.c +14
-0@@ -1114,6 +1124,129 @@ static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_s env->subprog_info[env->subprog_cnt].start = env->prog->len; } +static int cleanup_pad_body_for_subprog (struct bpf_verifier_env *env,struct bpf_prog *sub, +u32 start,u32 end) +{ +unsigned long *bits; +u32 i, cnt =0 ; + +for (i = start; i < end; i++) +if (env->insn_aux_data[i].in_cleanup_pad) + cnt++; +if (!cnt) +return 0 ; + + bits =bitmap_zalloc (end - start,GFP_KERNEL_ACCOUNT | __GFP_NOWARN); +if (!bits)
kernel/bpf/fixups.c static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_s · line 1150
Quoted 10 lines
kernel/bpf/fixups.c +10
-0+return -ENOMEM ; + +for (i = start; i < end; i++) +if (env->insn_aux_data[i].in_cleanup_pad) +__set_bit (i - start, bits); + + sub->aux->exc->pad_body = bits; + sub->aux->exc->nr_pad_body = end - start; +return 0 ; +}
This isn't a bug, but would naming this field after the bitmap size rather than nr_pad_body make it clearer that it is the bit count and not the number of pad-body instructions, given the nr_throw_at / nr_resume_at siblings are true counts?
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerRe: [PATCH bpf-next v3 10/20] bpf: Dispatch exception cleanup pads from bpf_throw()
Quoted 31 lines · On 9/19/26 11:32 PM, [email protected] wrote:
Quoted 4 lines
include/linux/bpf.h +0
-0[ ... ]
Quoted 4 lines
kernel/bpf/exception.c +0
-0[ ... ]
Quoted 19 lines
kernel/bpf/exception.c +15
-0@@ -499,3 +501,164 @@ int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx)return pad ? (int )pad -1 :-1 ; } + +int bpf_cleanup_alloc_info (struct bpf_prog_aux *aux) +{ +if (aux->exc) +return 0 ; + aux->exc = kzalloc_obj(struct bpf_exception_info, GFP_KERNEL_ACCOUNT | __GFP_NOWARN); +return aux->exc ?0 : -ENOMEM ; +} + +int bpf_cleanup_attach_info (struct bpf_prog_aux *aux,struct bpf_cleanup_info *recs,u32 cnt) +{ +struct bpf_exception_info *exc = aux->exc; +struct bpf_cleanup_range *ranges; +u32 i, n_at, *at; +
Quoted 8 lines
kernel/bpf/exception.c int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx) · line 560
Quoted 4 lines
kernel/bpf/exception.c +4
-0+if (!cnt) { +kvfree (recs); +return 0 ; + }This isn't a bug, but bpf_cleanup_attach_info()'s only caller already returns early when cnt is zero - is the
if (!cnt)arm here reachable, or would dropping it (and letting the caller keep owning that case) make the recs ownership easier to follow?
Will drop the above if condition.
Quoted 35 lines
Quoted 3 lines
kernel/bpf/exception.c +2
-0+ + ranges =kvcalloc (cnt,sizeof (*ranges),GFP_KERNEL_ACCOUNT | __GFP_NOWARN);[ ... ]
Quoted 4 lines
kernel/bpf/fixups.c +0
-0[ ... ]
Quoted 24 lines
kernel/bpf/fixups.c +20
-0@@ -1114,6 +1124,129 @@ static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_s env->subprog_info[env->subprog_cnt].start = env->prog->len; } +static int cleanup_pad_body_for_subprog (struct bpf_verifier_env *env,struct bpf_prog *sub, +u32 start,u32 end) +{ +unsigned long *bits; +u32 i, cnt =0 ; + +for (i = start; i < end; i++) +if (env->insn_aux_data[i].in_cleanup_pad) + cnt++; +if (!cnt) +return 0 ; + + bits =bitmap_zalloc (end - start,GFP_KERNEL_ACCOUNT | __GFP_NOWARN); +if (!bits) +return -ENOMEM ; + +for (i = start; i < end; i++) +if (env->insn_aux_data[i].in_cleanup_pad) +__set_bit (i - start, bits); +
Quoted 8 lines
kernel/bpf/fixups.c static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_s · line 1181
Quoted 4 lines
kernel/bpf/fixups.c +4
-0+ sub->aux->exc->pad_body = bits; + sub->aux->exc->nr_pad_body = end - start; +return 0 ; +}This isn't a bug, but would naming this field after the bitmap size rather than
nr_pad_bodymake it clearer that it is the bit count and not the number of pad-body instructions, given thenr_throw_at/nr_resume_atsiblings are true counts?
Will rename to pad_body_bits.
Quoted 9 lines
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerPATCH v3 11/20 bpf, x86: Dispatch exception cleanup pads at run time
In reply to Yonghong SongProvide the arch half for x86-64: force the full callee-saved spill for a program carrying a cleanup table so the walker can find a frame's r6-r9 in its callee's prologue, record where that spill area starts, build the native cleanup table from the JIT's addrs[], emit a bare return for a pad's bpf_unwind_resume(), and hand control to a pad from arch_bpf_run_cleanup_pad().
Support is gated on CONFIG_UNWINDER_ORC, the same requirement arch_bpf_stack_walk() and therefore bpf_throw() already have here, and bpf_cleanup_pad.o is built only there. Nothing else can call it, and a frame-pointer build would have objtool validate a routine that has to carry BPF r10 in rbp rather than a frame pointer ("call without frame pointer save/setup", an error under CONFIG_OBJTOOL_WERROR).
--- arch/x86/net/Makefile | 3 ++ arch/x86/net/bpf_cleanup_pad.S | 80 ++++++++++++++++++++++++++++++++++ arch/x86/net/bpf_jit_comp.c | 79 ++++++++++++++++++++++++++++----- 3 files changed, 150 insertions(+), 12 deletions(-) create mode 100644 arch/x86/net/bpf_cleanup_pad.S
arch/x86/net/Makefile +3 -0
@@ -7,4 +7,7 @@ ifeq ($(CONFIG_X86_32),y) obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o else obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o + ifdef CONFIG_UNWINDER_ORC + obj-$(CONFIG_BPF_JIT) += bpf_cleanup_pad.o + endif endif
arch/x86/net/bpf_cleanup_pad.S new file +80 -0
80 lines not on this page · raw message
arch/x86/net/bpf_jit_comp.c +67 -12
171 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 12/20 bpf, arm64: Dispatch exception cleanup pads at run time
In reply to Yonghong SongThe same arch half for arm64: force the full callee-saved spill, record where it starts, build the native table from the JIT's byte offsets, and hand control to a pad from arch_bpf_run_cleanup_pad().
Unlike x86-64 a pad cannot simply return: every call it makes clobbers x30, so no return address survives to its resume. x23 carries it instead. bpf2a64[] maps nothing to x23 or x24 -- they are pushed only to keep the frame shape the exception callback expects -- so nothing else in generated code touches them, and being callee-saved they survive every kfunc the pad calls. The JIT emits "br x23" for a pad's bpf_unwind_resume().
x24 is the other one, and it is what makes a pad able to touch its own frame at all. Generated code addresses the BPF frame through the stack pointer -- the frame sits directly on top of it, which turns every offset positive and each access into one instruction -- and in a pad the stack pointer is the walker's. So the JIT has a pad recompute the equivalent of its frame's stack pointer from BPF r10 on entry, into x24, and addresses the frame off x24 for every instruction the previous patches marked as running only while unwinding. One instruction per pad, and the accesses keep the shape and the immediate range they have everywhere else.
--- arch/arm64/net/Makefile | 2 +- arch/arm64/net/bpf_cleanup_pad.S | 95 ++++++++++++++++++++++++++++++++ arch/arm64/net/bpf_jit_comp.c | 91 ++++++++++++++++++++++++++++-- 3 files changed, 181 insertions(+), 7 deletions(-) create mode 100644 arch/arm64/net/bpf_cleanup_pad.S
arch/arm64/net/Makefile +1 -1
@@ -2,4 +2,4 @@ # # ARM64 networking code # -obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o +obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o
arch/arm64/net/bpf_cleanup_pad.S new file +95 -0
95 lines not on this page · raw message
arch/arm64/net/bpf_jit_comp.c +85 -6
189 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 13/20 libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
In reply to Yonghong SongLLVM terminates a cleanup landing pad with a call to _Unwind_Resume: the base unwind ABI's entry point for carrying an unwind on once a frame's cleanups have run. The kernel provides that terminator as a kfunc, but not under that name: claiming _Unwind_Resume in the kernel's own symbol table, for a function whose body never runs, would be needlessly confusing, so it is called bpf_unwind_resume.
Both load paths take the detour. A direct load resolves the name against the kernel's BTF while libbpf runs. A light skeleton instead writes the name into the loader program's blob of bytes, for that program to resolve when it runs, so the name recorded there has to be translated as well.
--- tools/lib/bpf/libbpf.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
tools/lib/bpf/libbpf.c +16 -5
@@ -8351,6 +8351,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s } } +/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the +* base unwind ABI's entry point for carrying an unwind on once a frame's +* cleanups have run. The kernel knows it as bpf_unwind_resume. +*/ +static const char *kern_extern_name (const char *name) +{ +return strcmp (name,"_Unwind_Resume" ) ? name :"bpf_unwind_resume" ; +} +static int bpf_program_record_relos (struct bpf_program *prog) {struct bpf_object *obj = prog->obj;
@@ -8367,12 +8376,12 @@ static int bpf_program_record_relos(struct bpf_program *prog)continue ; kind =btf_is_var (btf__type_by_id (obj->btf, ext->btf_id)) ?BTF_KIND_VAR :BTF_KIND_FUNC ; -bpf_gen__record_extern (obj->gen_loader, ext->name, +bpf_gen__record_extern (obj->gen_loader,kern_extern_name (ext->name), ext->is_weak, !ext->ksym.type_id, true, kind, relo->insn_idx);break ;case RELO_EXTERN_CALL : -bpf_gen__record_extern (obj->gen_loader, ext->name, +bpf_gen__record_extern (obj->gen_loader,kern_extern_name (ext->name), ext->is_weak, false, false,BTF_KIND_FUNC , relo->insn_idx);break ;
@@ -8812,17 +8821,19 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,struct module_btf *mod_btf =NULL ;const struct btf_type *kern_func;struct btf *kern_btf =NULL ; +const char *kern_name;int ret; local_func_proto_id = ext->ksym.type_id; - kfunc_id =find_ksym_btf_id (obj, ext->essent_name ?: ext->name,BTF_KIND_FUNC , &kern_btf, - &mod_btf); + kern_name =kern_extern_name (ext->essent_name ?: ext->name); + + kfunc_id =find_ksym_btf_id (obj, kern_name,BTF_KIND_FUNC , &kern_btf, &mod_btf);if (kfunc_id <0 ) {if (kfunc_id == -ESRCH && ext->is_weak)return 0 ;pr_warn ("extern (func ksym) '%s': not found in kernel or module BTFs\n" , - ext->name); +strcmp (kern_name,"bpf_unwind_resume" ) ? ext->name : kern_name);return kfunc_id; }
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 14/20 libbpf: Add cleanup_info to bpf_prog_load_opts
In reply to Yonghong SongLet a caller hand the kernel an exception cleanup table. Extend struct bpf_prog_load_opts with cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, pass them through to BPF_PROG_LOAD, and grow the attr size bpf_prog_load() computes to cover the new fields.
--- tools/lib/bpf/bpf.c | 6 +++++- tools/lib/bpf/bpf.h | 7 ++++++- 2 files changed, 11 insertions(+), 2 deletions(-)
tools/lib/bpf/bpf.c +5 -1
@@ -295,7 +295,7 @@ int bpf_prog_load(enum bpf_prog_type prog_type, const struct bpf_insn *insns, size_t insn_cnt, struct bpf_prog_load_opts *opts) { - const size_t attr_sz = offsetofend(union bpf_attr, keyring_id); + const size_t attr_sz = offsetofend(union bpf_attr, cleanup_info_cnt);void *finfo =NULL , *linfo =NULL ;const char *func_info, *line_info;__u32 log_size, log_level, attach_prog_fd, attach_btf_obj_fd;
@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type, attr.fd_array =ptr_to_u64 (OPTS_GET (opts, fd_array,NULL )); attr.fd_array_cnt =OPTS_GET (opts, fd_array_cnt,0 ); + attr.cleanup_info =ptr_to_u64 (OPTS_GET (opts, cleanup_info,NULL )); + attr.cleanup_info_cnt =OPTS_GET (opts, cleanup_info_cnt,0 ); + attr.cleanup_info_rec_size =OPTS_GET (opts, cleanup_info_rec_size,0 ); +if (log_level) { attr.log_buf =ptr_to_u64 (log_buf); attr.log_size = log_size;
tools/lib/bpf/bpf.h +6 -1
@@ -128,9 +128,14 @@ struct bpf_prog_load_opts {/* if set, provides the length of fd_array */ __u32 fd_array_cnt; + +/* exception cleanup table, from the .bpf_cleanup section */ +const void *cleanup_info; +__u32 cleanup_info_cnt; +__u32 cleanup_info_rec_size; size_t :0; }; -#define bpf_prog_load_opts__last_field fd_array_cnt +#define bpf_prog_load_opts__last_field cleanup_info_rec_size LIBBPF_API int bpf_prog_load(enum bpf_prog_type prog_type, const char *prog_name, const char *license,
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v3 14/20] libbpf: Add cleanup_info to bpf_prog_load_opts
This isn't a bug, but would it help a future reader if the changelog noted that cleanup_info_cnt is the last field of the BPF_PROG_LOAD attr struct, so offsetofend() on it covers all three new…
Quoted 5 lines
Let a caller hand the kernel an exception cleanup table. Extend struct
bpf_prog_load_optswithcleanup_info,cleanup_info_cntandcleanup_info_rec_size, pass them through toBPF_PROG_LOAD, and grow the attr sizebpf_prog_load()computes to cover the new fields.Signed-off-by Yonghong Song <[email protected]>
This isn't a bug, but would it help a future reader if the changelog noted that cleanup_info_cnt is the last field of the BPF_PROG_LOAD attr struct, so offsetofend() on it covers all three new fields, rather than restating the three edits?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerRe: [PATCH bpf-next v3 14/20] libbpf: Add cleanup_info to bpf_prog_load_opts
Quoted 9 lines · On 9/19/26 11:31 PM, [email protected] wrote:
Quoted 5 lines
Let a caller hand the kernel an exception cleanup table. Extend struct
bpf_prog_load_optswithcleanup_info,cleanup_info_cntandcleanup_info_rec_size, pass them through toBPF_PROG_LOAD, and grow the attr sizebpf_prog_load()computes to cover the new fields.Signed-off-by Yonghong Song <[email protected]>This isn't a bug, but would it help a future reader if the changelog noted that
cleanup_info_cntis the last field of theBPF_PROG_LOADattr struct, sooffsetofend()on it covers all three new fields, rather than restating the three edits?
Okay, will explicit mention cleanup_info_cnt is the last field.
Quoted 6 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerPATCH v3 15/20 libbpf: Collect .bpf_cleanup records and pass them to the kernel
In reply to Yonghong SongParse the compiler-emitted .bpf_cleanup section and hand the resulting table to BPF_PROG_LOAD.
Each record is three 4-byte fields, and each field is a byte offset into some code section named by a matching .rel.bpf_cleanup relocation. bpf_object__init_cleanup_info() resolves both halves once at open time and keeps (section, instruction index) pairs; it rejects a record whose field has no relocation, whose relocation is not one of the two 32-bit types that spell a data reference to a code section -- R_BPF_64_NODYLD32 from LLVM, R_BPF_64_ABS32 from GNU as -- or whose offset is not instruction aligned.
The records are sorted by begin_off once the offsets are final: the kernel wants the table sorted with disjoint ranges so that it can find the record covering a call site with a binary search, and records arrive in .bpf_cleanup order, which says nothing about where the subprograms they describe were appended. Overlapping ranges are reported here, where the program name and both regions are still at hand.
bpf_object_load_prog() then passes the per-program table through the bpf_prog_load() options added in the previous patch, with the record size carried on the program the way func_info and line_info carry theirs rather than taken from a sizeof() at the call site. bpf_program__clone() carries it too. That is a second load path -- the one veristat uses -- and without the table the kernel sees landing pads nothing reaches and refuses the program with "unreachable insn".
--- tools/lib/bpf/libbpf.c | 299 ++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf_internal.h | 3 + 2 files changed, 302 insertions(+)
tools/lib/bpf/libbpf.c +299 -0
@@ -514,6 +514,11 @@ struct bpf_program {void *line_info;__u32 line_info_rec_size;__u32 line_info_cnt; + +struct bpf_cleanup_info *cleanup_info; +__u32 cleanup_info_rec_size; +__u32 cleanup_info_cnt; +__u32 prog_flags;__u8 hash[SHA256_DIGEST_LENGTH ];
@@ -549,6 +554,7 @@ struct bpf_struct_ops {#define STRUCT_OPS_SEC ".struct_ops"#define STRUCT_OPS_LINK_SEC ".struct_ops.link"#define ARENA_SEC ".addr_space.1" +#define CLEANUP_SEC ".bpf_cleanup"enum libbpf_map_type {LIBBPF_MAP_UNSPEC ,
@@ -677,6 +683,25 @@ struct elf_sec_desc {Elf_Data *data; }; +#define CLEANUP_REC_FIELDS (sizeof(struct bpf_cleanup_info) / sizeof(__u32)) + +/* Index of each field of struct bpf_cleanup_info, read as an array of __u32. */ +enum { +CLEANUP_REC_BEGIN , +CLEANUP_REC_END , +CLEANUP_REC_PAD , +}; + +/* One (begin, end, landing_pad) triple from .bpf_cleanup, with each field +* resolved from its relocation to an ELF section plus a section-relative +* instruction index. The mapping to final program instruction indices can only +* happen after subprogram placement, which differs per main program. +*/ +struct cleanup_raw_rec { +int sec_idx[CLEANUP_REC_FIELDS ]; +size_t insn_idx[CLEANUP_REC_FIELDS ]; +}; +struct elf_state {int fd;const void *obj_buf;
@@ -696,6 +721,8 @@ struct elf_state {bool has_st_ops;int arena_data_shndx;int jumptables_data_shndx; +Elf_Data *cleanup_data; +int cleanup_shndx; };struct usdt_manager ;
@@ -771,6 +798,9 @@ struct bpf_object {void *jumptables_data;size_t jumptables_data_sz; +struct cleanup_raw_rec *cleanup_recs; +size_t cleanup_rec_cnt; +struct {struct bpf_program *prog;unsigned int sym_off;
@@ -817,7 +847,10 @@ static void bpf_program__exit(struct bpf_program *prog)zfree (&prog->sec_name);zfree (&prog->insns);zfree (&prog->reloc_desc); +zfree (&prog->cleanup_info); + prog->cleanup_info_rec_size =0 ; + prog->cleanup_info_cnt =0 ; prog->nr_reloc =0 ; prog->insns_cnt =0 ; prog->sec_idx =-1 ;
@@ -1559,6 +1592,7 @@ static struct bpf_object *bpf_object__new(const char *path, obj->efile.obj_buf = obj_buf; obj->efile.obj_buf_sz = obj_buf_sz; obj->efile.btf_maps_shndx =-1 ; + obj->efile.cleanup_shndx =-1 ; obj->kconfig_map_idx =-1 ; obj->arena_map_idx =-1 ;
@@ -4045,6 +4079,9 @@ static int bpf_object__elf_collect(struct bpf_object *obj) sec_desc->shdr = sh; sec_desc->data = data; obj->efile.has_st_ops = true; + }else if (strcmp (name,CLEANUP_SEC ) ==0 ) { + obj->efile.cleanup_data = data; + obj->efile.cleanup_shndx = idx; }else if (strcmp (name,ARENA_SEC ) ==0 ) { obj->efile.arena_data = data; obj->efile.arena_data_shndx = idx;
@@ -4072,6 +4109,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj) strcmp(name, ".rel" STRUCT_OPS_LINK_SEC) && strcmp(name, ".rel?" STRUCT_OPS_SEC) && strcmp(name, ".rel?" STRUCT_OPS_LINK_SEC) && + strcmp(name, ".rel" CLEANUP_SEC) && strcmp(name, ".rel" MAPS_ELF_SEC)) { pr_info("elf: skipping relo section(%d) %s for section(%d) %s\n", idx, name, targ_sec_idx,
@@ -4852,6 +4890,229 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,return NULL ; } +static int bpf_object__init_cleanup_info (struct bpf_object *obj) +{ +Elf_Data *data = obj->efile.cleanup_data; +Elf_Data *relo =NULL ; +size_t i, nrels, nslots, nrecs; +struct cleanup_raw_rec *recs; +int *slot_sec, ret =0 ; +size_t *slot_val; +const __u32 *vals; +bool native; + +if (!data || obj->efile.cleanup_shndx <0 ) +return 0 ; + + native =is_native_endianness (obj); + +for (i =0 ; i < obj->efile.sec_cnt; i++) { +struct elf_sec_desc *sd = &obj->efile.secs[i]; + +if (sd->sec_type ==SEC_RELO && sd->shdr && + sd->shdr->sh_info == (Elf64_Word )obj->efile.cleanup_shndx) { + relo = sd->data; +break ; + } + } +if (!relo) { +pr_warn ("%s present without relocations\n" ,CLEANUP_SEC ); +return -LIBBPF_ERRNO__FORMAT ; + } +if (data->d_size %sizeof (struct bpf_cleanup_info )) { +pr_warn ("%s size %zu is not a multiple of the record size %zu\n" , +CLEANUP_SEC , data->d_size,sizeof (struct bpf_cleanup_info )); +return -LIBBPF_ERRNO__FORMAT ; + } + + vals = data->d_buf; + nslots = data->d_size /sizeof (__u32); + nrecs = data->d_size /sizeof (struct bpf_cleanup_info ); + + slot_sec =calloc (nslots,sizeof (*slot_sec)); + slot_val =calloc (nslots,sizeof (*slot_val)); + recs =calloc (nrecs ?:1 ,sizeof (*recs)); +if (!slot_sec || !slot_val || !recs) { + ret = -ENOMEM ; + goto out; + } +for (i =0 ; i < nslots; i++) + slot_sec[i] =-1 ; + +/* One relocation per 4-byte field, naming the section it points into. */ + nrels = relo->d_size /sizeof (Elf64_Rel); +for (i =0 ; i < nrels; i++) { +Elf64_Rel *rel =elf_rel_by_idx (relo, i); +Elf64_Sym *sym =elf_sym_by_idx (obj,ELF64_R_SYM (rel->r_info)); +size_t type =ELF64_R_TYPE (rel->r_info); +size_t slot = rel->r_offset /sizeof (__u32); + +if (type !=R_BPF_64_NODYLD32 && type !=R_BPF_64_ABS32 ) { +pr_warn ("%s: relocation %zu has unexpected type %zu\n" , +CLEANUP_SEC , i, type); + ret = -LIBBPF_ERRNO__FORMAT ; + goto out; + } +if (!sym || slot >= nslots || rel->r_offset %sizeof (__u32)) { +pr_warn ("%s: bad relocation %zu\n" ,CLEANUP_SEC , i); + ret = -LIBBPF_ERRNO__FORMAT ; + goto out; + } + slot_sec[slot] = sym->st_shndx; +/* The addend lives in the section data, which libelf leaves in +* the object's byte order; a non-section symbol additionally +* contributes its own value. +*/ + slot_val[slot] = (native ? vals[slot] :bswap_32 (vals[slot])) + + sym->st_value; + } + +for (i =0 ; i < nslots; i++) { +struct cleanup_raw_rec *rec = &recs[i /CLEANUP_REC_FIELDS ]; +size_t field = i %CLEANUP_REC_FIELDS ; + +if (slot_sec[i] <0 ) { +pr_warn ("%s: field %zu has no relocation\n" ,CLEANUP_SEC , i); + ret = -LIBBPF_ERRNO__FORMAT ; + goto out; + } +if (slot_val[i] %BPF_INSN_SZ ) { +pr_warn ("%s: field %zu offset %zu is not instruction aligned\n" , +CLEANUP_SEC , i, slot_val[i]); + ret = -LIBBPF_ERRNO__FORMAT ; + goto out; + } + rec->sec_idx[field] = slot_sec[i]; + rec->insn_idx[field] = slot_val[i] /BPF_INSN_SZ ; + } + + obj->cleanup_recs = recs; + obj->cleanup_rec_cnt = nrecs; + recs =NULL ; +out: +free (recs); +free (slot_val); +free (slot_sec); +return ret; +} + +static int cmp_cleanup_info (const void *a,const void *b) +{ +const struct bpf_cleanup_info *x = a, *y = b; + +if (x->begin_off == y->begin_off) +return 0 ; +return x->begin_off < y->begin_off ?-1 :1 ; +} + +static int bpf_prog_collect_cleanup_info (struct bpf_object *obj, +struct bpf_program *prog) +{ +size_t i; +int j; + +for (i =0 ; i < obj->cleanup_rec_cnt; i++) { +struct cleanup_raw_rec *raw = &obj->cleanup_recs[i]; +struct bpf_program *owner =NULL ; +struct bpf_cleanup_info ci = {}; +__u32 fields[CLEANUP_REC_FIELDS ]; +void *tmp; + +if (raw->sec_idx[CLEANUP_REC_BEGIN ] == raw->sec_idx[CLEANUP_REC_END ] && + raw->insn_idx[CLEANUP_REC_BEGIN ] >= raw->insn_idx[CLEANUP_REC_END ]) { +pr_warn ("%s: record %zu is an empty range [%zu,%zu)\n" , +CLEANUP_SEC , i, raw->insn_idx[CLEANUP_REC_BEGIN ], + raw->insn_idx[CLEANUP_REC_END ]); +return -LIBBPF_ERRNO__FORMAT ; + } + +for (j =0 ; j <CLEANUP_REC_FIELDS ; j++) { +size_t idx = raw->insn_idx[j], final; +struct bpf_program *p; + +/* The end of a range is exclusive, so it may name the +* instruction just past the last one of a function, +* which belongs to the next function or to nothing at +* all. Ask about the last instruction the range covers, +* the way the kernel does. +*/ +if (j ==CLEANUP_REC_END ) { +if (!idx) { +pr_warn ("%s: record %zu ends at instruction 0\n" , +CLEANUP_SEC , i); +return -LIBBPF_ERRNO__FORMAT ; + } + idx--; + } + + p =find_prog_by_sec_insn (obj, raw->sec_idx[j], idx); +if (!p) { +pr_warn ("%s: record %zu field %d is not inside a function\n" , +CLEANUP_SEC , i, j); +return -LIBBPF_ERRNO__FORMAT ; + } +if (!owner) { + owner = p; + }else if (owner != p) { +pr_warn ("%s: record %zu spans functions '%s' and '%s'\n" , +CLEANUP_SEC , i, owner->name, p->name); +return -LIBBPF_ERRNO__FORMAT ; + } + +if (owner == prog) { + final = raw->insn_idx[j] - prog->sec_insn_off; + }else if (prog_is_subprog (obj, owner) && owner->sub_insn_off) { +/* sub_insn_off is where this subprogram was +* appended to the main program being relocated; +* zero means it is not part of it. +*/ + final = owner->sub_insn_off + + raw->insn_idx[j] - owner->sec_insn_off; + }else { + owner =NULL ; +break ; + } + fields[j] = final; + } +if (!owner) +continue ; + + ci.begin_off = fields[CLEANUP_REC_BEGIN ]; + ci.end_off = fields[CLEANUP_REC_END ]; + ci.landing_pad_off = fields[CLEANUP_REC_PAD ]; + + tmp =libbpf_reallocarray (prog->cleanup_info, prog->cleanup_info_cnt +1 , +sizeof (*prog->cleanup_info)); +if (!tmp) +return -ENOMEM ; + prog->cleanup_info = tmp; + prog->cleanup_info_rec_size =sizeof (struct bpf_cleanup_info ); + prog->cleanup_info[prog->cleanup_info_cnt++] = ci; + +pr_debug ("prog '%s': cleanup region [%u,%u) -> landing pad %u\n" , + prog->name, ci.begin_off, ci.end_off, ci.landing_pad_off); + } + +if (!prog->cleanup_info_cnt) +return 0 ; + +qsort (prog->cleanup_info, prog->cleanup_info_cnt, +sizeof (*prog->cleanup_info), cmp_cleanup_info); +for (i =1 ; i < prog->cleanup_info_cnt; i++) { +struct bpf_cleanup_info *prev = &prog->cleanup_info[i -1 ]; +struct bpf_cleanup_info *cur = &prog->cleanup_info[i]; + +if (cur->begin_off < prev->end_off) { +pr_warn ("prog '%s': overlapping cleanup regions [%u,%u) and [%u,%u)\n" , + prog->name, prev->begin_off, prev->end_off, + cur->begin_off, cur->end_off); +return -LIBBPF_ERRNO__FORMAT ; + } + } + +return 0 ; +} +static int bpf_object__collect_prog_relos (struct bpf_object *obj,Elf64_Shdr *shdr,Elf_Data *data) {
@@ -7561,6 +7822,13 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_patreturn err; } } + + err =bpf_prog_collect_cleanup_info (obj, prog); +if (err) { +pr_warn ("prog '%s': failed to collect cleanup info: %s\n" , + prog->name,errstr (err)); +return err; + } }for (i =0 ; i < obj->nr_programs; i++) { prog = &obj->programs[i];
@@ -7751,6 +8019,9 @@ static int bpf_object__collect_relos(struct bpf_object *obj)return -LIBBPF_ERRNO__INTERNAL ; } +if (idx == obj->efile.cleanup_shndx) +continue ; +if (obj->efile.secs[idx].sec_type ==SEC_ST_OPS ) err =bpf_object__collect_st_ops_relos (obj, shdr, data);else if (idx == obj->efile.btf_maps_shndx)
@@ -8023,6 +8294,11 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog load_attr.line_info_rec_size = prog->line_info_rec_size; load_attr.line_info_cnt = prog->line_info_cnt; } +if (prog->cleanup_info_cnt) { + load_attr.cleanup_info = prog->cleanup_info; + load_attr.cleanup_info_cnt = prog->cleanup_info_cnt; + load_attr.cleanup_info_rec_size = prog->cleanup_info_rec_size; + } load_attr.log_level = log_level; load_attr.prog_flags = prog->prog_flags; load_attr.fd_array = obj->fd_array;
@@ -8579,6 +8855,7 @@ static struct bpf_object *bpf_object_open(const char *path, const void *obj_buf, err = err ? :bpf_object__init_maps (obj, opts); err = err ? :bpf_object_init_progs (obj, opts); err = err ? :bpf_object__collect_relos (obj); + err = err ? :bpf_object__init_cleanup_info (obj);if (err) goto out;
@@ -9692,6 +9969,9 @@ void bpf_object__close(struct bpf_object *obj)zfree (&obj->jumptables_data); obj->jumptables_data_sz =0 ; +zfree (&obj->cleanup_recs); + obj->cleanup_rec_cnt =0 ; +for (i =0 ; i < obj->jumptable_map_cnt; i++)close (obj->jumptable_maps[i].fd);zfree (&obj->jumptable_maps);
@@ -10084,6 +10364,25 @@ int bpf_program__clone(struct bpf_program *prog, const struct bpf_prog_load_opts attr.line_info_rec_size = info ? info_rec_size : prog->line_info_rec_size; } +/* exception cleanup table */ + info =OPTS_GET (opts, cleanup_info,NULL ); + info_cnt =OPTS_GET (opts, cleanup_info_cnt,0 ); + info_rec_size =OPTS_GET (opts, cleanup_info_rec_size,0 ); +if (!!info != !!info_cnt || !!info != !!info_rec_size) { +pr_warn ("prog '%s': cleanup_info, cleanup_info_cnt, and cleanup_info_rec_size must all be specified or all omitted\n" , + prog->name); +return libbpf_err (-EINVAL ); + } +if (info) { + attr.cleanup_info = info; + attr.cleanup_info_cnt = info_cnt; + attr.cleanup_info_rec_size = info_rec_size; + }else if (prog->cleanup_info_cnt) { + attr.cleanup_info = prog->cleanup_info; + attr.cleanup_info_cnt = prog->cleanup_info_cnt; + attr.cleanup_info_rec_size = prog->cleanup_info_rec_size; + } +/* Logging is caller-controlled; no fallback to prog/obj log settings */ attr.log_buf =OPTS_GET (opts, log_buf,NULL ); attr.log_size =OPTS_GET (opts, log_size,0 );
tools/lib/bpf/libbpf_internal.h +3 -0
@@ -56,6 +56,9 @@#ifndef R_BPF_64_ABS32 #define R_BPF_64_ABS32 3#endif +#ifndef R_BPF_64_NODYLD32 +#define R_BPF_64_NODYLD32 4 +#endif #ifndef R_BPF_64_32 #define R_BPF_64_32 10#endif
Signature · 1 line
2.53.0-Meta
Yonghong Song
reviewerPATCH v3 16/20 libbpf: Carry the exception cleanup table through the light skeleton
In reply to Yonghong SongA light skeleton does not call bpf_prog_load(). bpf_gen__prog_load() builds its own union bpf_attr field by field, and the loader program it emits is what issues BPF_PROG_LOAD when the skeleton runs -- so a program loaded this way reached the kernel without the table the previous patch collected for it. The load then fails, because the landing pads are code nothing reaches and the verifier says so. It says "unreachable insn", which names neither the skeleton nor the table.
Carry it the way func_info and line_info are carried: the records go into the loader's blob of bytes, the count and record size into the attr, and a relocation stores the blob's address into attr.cleanup_info once that address is known. The attr grows to its new last field, cleanup_info_cnt. Records are 4-byte fields like the other info blobs, so a cross-endian build has to swap them too.
Growing the attr is what makes the relocation conditional. Every other attr in this file stops at the last field it sets, and this one now runs to the end of the union: a kernel that predates cleanup_info accepts an attr longer than its own only while the tail it does not know reads as zero. The count and the record size are already zero for a program with no table, but the relocation stores a blob address, which never is, so a program that has no landing pads only keeps loading on such a kernel if the relocation is left out.
--- tools/lib/bpf/gen_loader.c | 29 +++++++++++++++++++++++++---- tools/lib/bpf/libbpf_internal.h | 7 +++++++ 2 files changed, 32 insertions(+), 4 deletions(-)
tools/lib/bpf/gen_loader.c +25 -4
@@ -981,13 +981,15 @@ static void cleanup_relos(struct bpf_gen *gen, int insns)cleanup_core_relo (gen); } -/* Convert func, line, and core relo info blobs to target endianness */ +/* Convert func, line, core relo and cleanup info blobs to target endianness */ static void info_blob_bswap (struct bpf_gen *gen,int func_info,int line_info, -int core_relos,struct bpf_prog_load_opts *load_attr) +int core_relos,int cleanup_info, +struct bpf_prog_load_opts *load_attr) {struct bpf_func_info *fi = gen->data_start + func_info;struct bpf_line_info *li = gen->data_start + line_info;struct bpf_core_relo *cr = gen->data_start + core_relos; +struct bpf_cleanup_info *ci = gen->data_start + cleanup_info;int i;for (i =0 ; i < load_attr->func_info_cnt; i++)
@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,for (i =0 ; i < gen->core_relo_cnt; i++)bpf_core_relo_bswap (cr++); + +for (i =0 ; i < load_attr->cleanup_info_cnt; i++) +bpf_cleanup_info_bswap (ci++); } void bpf_gen__prog_load(struct bpf_gen *gen,
@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen, load_attr->line_info_rec_size;int core_relo_tot_sz = gen->core_relo_cnt *sizeof (struct bpf_core_relo ); +int cleanup_info_tot_sz = load_attr->cleanup_info_cnt * + load_attr->cleanup_info_rec_size;int prog_load_attr, license_off, insns_off, func_info, line_info, core_relos; - int attr_size = offsetofend(union bpf_attr, core_relo_rec_size); + int attr_size = offsetofend(union bpf_attr, cleanup_info_cnt); +int cleanup_info;union bpf_attr attr;memset (&attr,0 , attr_size);
@@ -1061,9 +1069,17 @@ void bpf_gen__prog_load(struct bpf_gen *gen, core_relos, gen->core_relo_cnt, sizeof(struct bpf_core_relo)); + attr.cleanup_info_rec_size =tgt_endian (load_attr->cleanup_info_rec_size); + attr.cleanup_info_cnt =tgt_endian (load_attr->cleanup_info_cnt); + cleanup_info =add_data (gen, load_attr->cleanup_info, cleanup_info_tot_sz); +pr_debug ("gen: prog_load: cleanup_info: off %d cnt %u rec size %u\n" , + cleanup_info, load_attr->cleanup_info_cnt, + load_attr->cleanup_info_rec_size); +/* convert all info blobs to target endianness */ if (gen->swapped_endian && !gen->error) -info_blob_bswap (gen, func_info, line_info, core_relos, load_attr); +info_blob_bswap (gen, func_info, line_info, core_relos, cleanup_info, + load_attr);libbpf_strlcpy (attr.prog_name, prog_name,sizeof (attr.prog_name)); prog_load_attr =add_data (gen, &attr, attr_size);
@@ -1085,6 +1101,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,/* populate union bpf_attr with a pointer to core_relos */ emit_rel_store (gen,attr_field (prog_load_attr, core_relos), core_relos); +/* populate union bpf_attr with a pointer to cleanup_info, if there is one */ +if (load_attr->cleanup_info_cnt) +emit_rel_store (gen,attr_field (prog_load_attr, cleanup_info), + cleanup_info); +/* populate union bpf_attr fd_array with a pointer to data where map_fds are saved */ emit_rel_store (gen,attr_field (prog_load_attr, fd_array), gen->fd_array);
tools/lib/bpf/libbpf_internal.h +7 -0
@@ -572,6 +572,13 @@ static inline void bpf_core_relo_bswap(struct bpf_core_relo *i) i->kind =bswap_32 (i->kind); } +static inline void bpf_cleanup_info_bswap (struct bpf_cleanup_info *i) +{ + i->begin_off =bswap_32 (i->begin_off); + i->end_off =bswap_32 (i->end_off); + i->landing_pad_off =bswap_32 (i->landing_pad_off); +} +enum btf_field_iter_kind {BTF_FIELD_ITER_IDS ,BTF_FIELD_ITER_STRS ,
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v3 16/20] libbpf: Carry the exception cleanup table through the light skeleton
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] bpf: Integer overflow in cleanup_info_tot_sz leads to heap buffer overflow --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: Integer overflow in cleanup_info_tot_sz leads to heap buffer overflow --
commit ae662736be5da05d165c478a926196fe248e707a
Author: Yonghong Song <[email protected]>
libbpf: Carry the exception cleanup table through the light skeleton
This commit updates the light skeleton generator to emit BPF_PROG_LOAD instructions that include the exception cleanup table. It stores the cleanup records in the loader's bytes blob, calculates the needed buffer size, and byte-swaps the records for cross-endian builds.
Quoted 4 lines
tools/lib/bpf/gen_loader.c +0
-0
[ ... ]
tools/lib/bpf/gen_loader.c static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info, · line 1005
Quoted 7 lines
tools/lib/bpf/gen_loader.c +3
-0@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,for (i =0 ; i < gen->core_relo_cnt; i++)bpf_core_relo_bswap (cr++); + +for (i =0 ; i < load_attr->cleanup_info_cnt; i++) +bpf_cleanup_info_bswap (ci++);
[Severity: High]
Does this loop write beyond the allocated buffer if cleanup_info_tot_sz overflowed during its calculation?
If an untrusted user provides a malicious BPF ELF object with a very large .bpf_cleanup section, load_attr->cleanup_info_cnt can be extremely large. The loop uses this un-wrapped count to byte-swap entries, writing past the end of the buffer if the allocation size was wrapped.
tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1020
Quoted 9 lines
tools/lib/bpf/gen_loader.c +2
-0} void bpf_gen__prog_load(struct bpf_gen *gen,@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen, load_attr->line_info_rec_size;int core_relo_tot_sz = gen->core_relo_cnt *sizeof (struct bpf_core_relo ); +int cleanup_info_tot_sz = load_attr->cleanup_info_cnt * + load_attr->cleanup_info_rec_size;
[Severity: High]
Can this multiplication overflow a 32-bit integer?
If load_attr->cleanup_info_cnt is a huge value (e.g., 0x15555556), multiplying it by cleanup_info_rec_size (12) will wrap around a 32-bit integer.
This would result in a small wrapped value being assigned to cleanup_info_tot_sz (e.g., 0x15555556 * 12 = 4294967304, wrapping to 8). The resulting undersized allocation passed to add_data() would then lead to the out-of-bounds write in info_blob_bswap().
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
Yonghong Song
reviewerRe: [PATCH bpf-next v3 16/20] libbpf: Carry the exception cleanup table through the light skeleton
Quoted 48 lines · On 9/19/26 11:01 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: Integer overflow incleanup_info_tot_szleads to heap buffer overflow --commit
ae662736be5da05d165c478a926196fe248e707a
Author: Yonghong Song <[email protected]>libbpf: Carry the exception cleanup table through the light skeleton
This commit updates the light skeleton generator to emit
BPF_PROG_LOADinstructions that include the exception cleanup table. It stores the cleanup records in the loader's bytes blob, calculates the needed buffer size, and byte-swaps the records for cross-endian builds.Quoted 4 lines
tools/lib/bpf/gen_loader.c +0
-0[ ... ]
tools/lib/bpf/gen_loader.c static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info, · line 1005
Quoted 7 lines
tools/lib/bpf/gen_loader.c +3
-0@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,for (i =0 ; i < gen->core_relo_cnt; i++)bpf_core_relo_bswap (cr++); + +for (i =0 ; i < load_attr->cleanup_info_cnt; i++) +bpf_cleanup_info_bswap (ci++);[Severity: High]
Does this loop write beyond the allocated buffer ifcleanup_info_tot_szoverflowed during its calculation?If an untrusted user provides a malicious BPF ELF object with a very large .bpf_cleanup section,
load_attr->cleanup_info_cntcan be extremely large. The loop uses this un-wrapped count to byte-swap entries, writing past the end of the buffer if the allocation size was wrapped.tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1020
Quoted 9 lines
tools/lib/bpf/gen_loader.c +2
-0} void bpf_gen__prog_load(struct bpf_gen *gen,@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen, load_attr->line_info_rec_size;int core_relo_tot_sz = gen->core_relo_cnt *sizeof (struct bpf_core_relo ); +int cleanup_info_tot_sz = load_attr->cleanup_info_cnt * + load_attr->cleanup_info_rec_size;[Severity: High]
Can this multiplication overflow a 32-bit integer?
Quoted 8 lines
If
load_attr->cleanup_info_cntis a huge value (e.g.,0x15555556), multiplying it bycleanup_info_rec_size(12) will wrap around a 32-bit integer.This would result in a small wrapped value being assigned to
cleanup_info_tot_sz(e.g.,0x15555556* 12 = 4294967304, wrapping to 8). The resulting undersized allocation passed toadd_data()would then lead to the out-of-bounds write ininfo_blob_bswap().
Overflow is indeed possible. Will fix it, e.g. with cleanup_info_tot_sz as __u64 and later check against INT32_MAX.
Yonghong Song
reviewerPATCH v3 17/20 libbpf: Let the static linker carry .bpf_cleanup relocations
In reply to Yonghong SongAn object that carries a compiler-emitted exception cleanup table cannot be linked today. The table's fields are byte offsets into a code section, materialised by a 32-bit relocation against that section's symbol with the offset itself as the implicit addend, and the linker rejects both halves of
that: the relocation type is not in the list it accepts, and a relocation against an STT_SECTION symbol from a non-executable section is an outright error.
Both spellings of that relocation have to be taken. LLVM emits R_BPF_64_NODYLD32 for a .long against a section symbol; GNU as emits R_BPF_64_ABS32, which is what bpf_reloc_type_lookup() maps BFD_RELOC_32 to. They describe the same value, and the selftests are built with both compilers.
Taking them is keyed on the relocation type rather than on the section name, so any non-executable section could reach the new arm, and what it replaces is an unconditional refusal. Check what the refusal used to make
unnecessary: a relocated section may be SHT_NOBITS, which extend_sec() leaves with no raw_data, and r_offset is checked for alignment only where the section holds instructions. Refuse those rather than write through them.
--- tools/lib/bpf/linker.c | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-)
tools/lib/bpf/linker.c +35 -1
@@ -1036,7 +1036,8 @@ static int linker_sanity_check_elf_relos(struct src_obj *obj, struct src_sec *sesize_t sym_type =ELF64_R_TYPE (relo->r_info);if (sym_type !=R_BPF_64_64 && sym_type !=R_BPF_64_32 && - sym_type !=R_BPF_64_ABS64 && sym_type !=R_BPF_64_ABS32 ) { + sym_type !=R_BPF_64_ABS64 && sym_type !=R_BPF_64_ABS32 && + sym_type !=R_BPF_64_NODYLD32 ) {pr_warn ("ELF relo #%d in section #%zu has unexpected type %zu in %s\n" , i, sec->sec_idx, sym_type, obj->filename);return -EINVAL ;
@@ -2274,6 +2275,39 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; + }else if (sym_type ==R_BPF_64_NODYLD32 || + sym_type ==R_BPF_64_ABS32 ) { +__u32 *val; + +/* Two spellings of the one thing: LLVM +* emits NODYLD32 for a .long against a +* section symbol, GNU as emits ABS32 +* (bpf_reloc_type_lookup() maps +* BFD_RELOC_32 to it), and the value +* they describe is the same. +*/ + +/* An SHT_NOBITS section has no +* raw_data, and r_offset is sanity +* checked only for sections holding +* instructions. Check both here +* before writing. +*/ +if (!dst_linked_sec->raw_data || + dst_rel->r_offset %sizeof (*val) || + dst_rel->r_offset +sizeof (*val) > + (size_t )dst_linked_sec->sec_sz) { +pr_warn ("ELF relo #%d in section #%zu points outside the data of section '%s' in %s\n" , + j, src_sec->sec_idx, + dst_linked_sec->sec_name, + obj->filename); +return -EINVAL ; + } + val = dst_linked_sec->raw_data + dst_rel->r_offset; +if (linker->swapped_endian) + *val =bswap_32 (bswap_32 (*val) + sec->dst_off); +else + *val += sec->dst_off; }else {pr_warn ("relocation against STT_SECTION in non-exec section is not supported!\n" );return -EINVAL ;
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v3 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] libbpf: integer overflow in relocation bounds check allows out-of-bounds heap access - [Low]…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] libbpf: integer overflow in relocation bounds check allows out-of-bounds heap access
- [Low] libbpf: incompatible pointer cast to __u32 * violates strict aliasing
- [Low] bpf: newly added multi-line comments violate the BPF kernel comment style --
commit c31e91765bc81fed1c51952c88d0792cd3b4c2bd
Author: Yonghong Song <[email protected]>
libbpf: Let the static linker carry .bpf_cleanup relocations
This commit updates the static linker to accept and properly handle R_BPF_64_NODYLD32 and R_BPF_64_ABS32 relocations against STT_SECTION symbols in non-executable sections. This enables linking of objects that contain compiler-emitted exception cleanup tables.
Quoted 4 lines
tools/lib/bpf/linker.c +0
-0
[ ... ]
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2283
Quoted 10 lines
tools/lib/bpf/linker.c +6
-0@@ -2274,6 +2275,39 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; + }else if (sym_type ==R_BPF_64_NODYLD32 || + sym_type ==R_BPF_64_ABS32 ) { +__u32 *val; + +/* Two spellings of the one thing: LLVM +* emits NODYLD32 for a .long against a
[Severity: Low]
Does this multi-line comment format violate the BPF subsystem guidelines? The BPF guidelines specify that multi-line comments must have the opening /* on its own line, with the text starting on the next line.
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2291
Quoted 8 lines
tools/lib/bpf/linker.c +8
-0+* section symbol, GNU as emits ABS32 +* (bpf_reloc_type_lookup() maps +* BFD_RELOC_32 to it), and the value +* they describe is the same. +*/ + +/* An SHT_NOBITS section has no +* raw_data, and r_offset is sanity
[Severity: Low]
This isn't a functional issue, but this comment also places text on the same line as the opening /*.
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2299
Quoted 8 lines
tools/lib/bpf/linker.c +8
-0+* checked only for sections holding +* instructions. Check both here +* before writing. +*/ +if (!dst_linked_sec->raw_data || + dst_rel->r_offset %sizeof (*val) || + dst_rel->r_offset +sizeof (*val) > + (size_t )dst_linked_sec->sec_sz) {
[Severity: High]
Can this bounds check bypass validation due to an integer overflow?
If a malicious BPF ELF object provides an oversized r_offset (for example, 0xFFFFFFFFFFFFFFFC), adding sizeof(*val) could wrap around to 0. This bypasses the check against dst_linked_sec->sec_sz and allows out-of-bounds heap memory access.
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2310
Quoted 11 lines
tools/lib/bpf/linker.c +11
-0+pr_warn ("ELF relo #%d in section #%zu points outside the data of section '%s' in %s\n" , + j, src_sec->sec_idx, + dst_linked_sec->sec_name, + obj->filename); +return -EINVAL ; + } + val = dst_linked_sec->raw_data + dst_rel->r_offset; +if (linker->swapped_endian) + *val =bswap_32 (bswap_32 (*val) + sec->dst_off); +else + *val += sec->dst_off;
[Severity: Low]
Does casting the raw byte buffer to a __u32 pointer violate strict aliasing rules?
The tools/ build assumes strict aliasing, and accessing generic byte buffers this way can lead to compiler misoptimization. Would it be safer to use memcpy or unaligned access macros here?
Quoted 3 lines
tools/lib/bpf/linker.c +0
-0}else {pr_warn ("relocation against STT_SECTION in non-exec section is not supported!\n" );return -EINVAL ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
Yonghong Song
reviewerRe: [PATCH bpf-next v3 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations
Quoted 60 lines · On 9/19/26 11:00 PM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] libbpf: integer overflow in relocation bounds check allows out-of-bounds heap access
- [Low] libbpf: incompatible pointer cast to__u32* violates strict aliasing
- [Low] bpf: newly added multi-line comments violate the BPF kernel comment style --commit
c31e91765bc81fed1c51952c88d0792cd3b4c2bd
Author: Yonghong Song <[email protected]>libbpf: Let the static linker carry .bpf_cleanup relocations
This commit updates the static linker to accept and properly handle
R_BPF_64_NODYLD32andR_BPF_64_ABS32relocations againstSTT_SECTIONsymbols in non-executable sections. This enables linking of objects that contain compiler-emitted exception cleanup tables.Quoted 4 lines
tools/lib/bpf/linker.c +0
-0[ ... ]
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2283
Quoted 10 lines
tools/lib/bpf/linker.c +6
-0@@ -2274,6 +2275,39 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; + }else if (sym_type ==R_BPF_64_NODYLD32 || + sym_type ==R_BPF_64_ABS32 ) { +__u32 *val; + +/* Two spellings of the one thing: LLVM +* emits NODYLD32 for a .long against a [Severity: Low]
Does this multi-line comment format violate the BPF subsystem guidelines? The BPF guidelines specify that multi-line comments must have the opening /* on its own line, with the text starting on the next line.tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2291
Quoted 8 lines
tools/lib/bpf/linker.c +8
-0+* section symbol, GNU as emits ABS32 +* (bpf_reloc_type_lookup() maps +* BFD_RELOC_32 to it), and the value +* they describe is the same. +*/ + +/* An SHT_NOBITS section has no +* raw_data, and r_offset is sanity [Severity: Low]
This isn't a functional issue, but this comment also places text on the same line as the opening /*.tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2299
Quoted 8 lines
tools/lib/bpf/linker.c +8
-0+* checked only for sections holding +* instructions. Check both here +* before writing. +*/ +if (!dst_linked_sec->raw_data || + dst_rel->r_offset %sizeof (*val) || + dst_rel->r_offset +sizeof (*val) > + (size_t )dst_linked_sec->sec_sz) {[Severity: High]
Can this bounds check bypass validation due to an integer overflow?
Quoted 4 lines
If a malicious BPF ELF object provides an oversized
r_offset(for example,0xFFFFFFFFFFFFFFFC), addingsizeof(*val)could wrap around to 0. This bypasses the check againstdst_linked_sec->sec_szand allows out-of-bounds heap memory access.
Yes, overflow is possible. Will fix.
Quoted 16 lines
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2310
Quoted 12 lines
tools/lib/bpf/linker.c +11
-0+pr_warn ("ELF relo #%d in section #%zu points outside the data of section '%s' in %s\n" , + j, src_sec->sec_idx, + dst_linked_sec->sec_name, + obj->filename); +return -EINVAL ; + } + val = dst_linked_sec->raw_data + dst_rel->r_offset; +if (linker->swapped_endian) + *val =bswap_32 (bswap_32 (*val) + sec->dst_off); +else + *val += sec->dst_off;[Severity: Low]
Does casting the raw byte buffer to a__u32pointer violate strict aliasing rules?
Quoted 3 lines
The
tools/build assumes strict aliasing, and accessing generic byte buffers this way can lead to compiler misoptimization. Would it be safer to use memcpy or unaligned access macros here?
I think it should be okay. In the above we have
struct bpf_insn *insn;
...
insn = dst_linked_sec->raw_data + dst_rel->r_offset;
Quoted 4 lines
Quoted 4 lines
tools/lib/bpf/linker.c +0
-0}else {pr_warn ("relocation against STT_SECTION in non-exec section is not supported!\n" );return -EINVAL ;
[email protected]
AI reviewRe: [PATCH bpf-next v3 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations
In reply to Yonghong SongCan the addition here wrap around? Since dst_rel->r_offset is Elf64_Addr (uint64_t) and sizeof(*val) is size_t, the expression
Quoted 30 lines
tools/lib/bpf/linker.c +14
-1@@ -1036,7 +1036,8 @@ static int linker_sanity_check_elf_relos(struct src_obj *obj, struct src_sec *sesize_t sym_type =ELF64_R_TYPE (relo->r_info);if (sym_type !=R_BPF_64_64 && sym_type !=R_BPF_64_32 && - sym_type !=R_BPF_64_ABS64 && sym_type !=R_BPF_64_ABS32 ) { + sym_type !=R_BPF_64_ABS64 && sym_type !=R_BPF_64_ABS32 && + sym_type !=R_BPF_64_NODYLD32 ) {pr_warn ("ELF relo #%d in section #%zu has unexpected type %zu in %s\n" , i, sec->sec_idx, sym_type, obj->filename);return -EINVAL ;@@ -2274,6 +2275,39 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; + }else if (sym_type ==R_BPF_64_NODYLD32 || + sym_type ==R_BPF_64_ABS32 ) { +__u32 *val; + +/* Two spellings of the one thing: LLVM +* emits NODYLD32 for a .long against a +* section symbol, GNU as emits ABS32 +* (bpf_reloc_type_lookup() maps +* BFD_RELOC_32 to it), and the value +* they describe is the same. +*/ +
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2299
Quoted 10 lines
tools/lib/bpf/linker.c +10
-0+/* An SHT_NOBITS section has no +* raw_data, and r_offset is sanity +* checked only for sections holding +* instructions. Check both here +* before writing. +*/ +if (!dst_linked_sec->raw_data || + dst_rel->r_offset %sizeof (*val) || + dst_rel->r_offset +sizeof (*val) > + (size_t )dst_linked_sec->sec_sz) {
^^^^
Can the addition here wrap around? Since dst_rel->r_offset is Elf64_Addr (uint64_t) and sizeof(*val) is size_t, the expression
dst_rel->r_offset + sizeof(*val) > (size_t)dst_linked_sec->sec_sz
is evaluated in unsigned 64-bit arithmetic. For r_offset == 0xFFFFFFFFFFFFFFFC (that is, 2^64 - 4):
r_offset % sizeof(*val) is 0 (since 2^64 - 4 is a multiple of 4)
r_offset + 4 wraps to 0
0 > (size_t)sec_sz is falseso all three sub-tests pass, and the code falls through to
val = dst_linked_sec->raw_data + dst_rel->r_offset; /* raw_data - 4 */
*val += sec->dst_off;which performs a 4-byte read-modify-write immediately before the buffer obtained from realloc(), corrupting the heap chunk header.
r_offset is fully controlled by the input object and is never bounds-checked. The existing r_offset validation in linker_sanity_check_elf_relos() (near the top of this diff) is an alignment-only test and applies only to SHF_EXECINSTR sections. The .bpf_cleanup section is not executable, so nothing constrains r_offset before this point.
Reached via: bpf_linker__add_file() -> bpf_linker_add_file() -> linker_append_elf_relos() -> this arm. Triggered by any object with a .rel.bpf_cleanup entry of type R_BPF_64_ABS32 or R_BPF_64_NODYLD32 against an STT_SECTION symbol and r_offset == 2^64 - 4.
The commit message notes "r_offset is checked for alignment only where the section holds instructions. Refuse those rather than write through them", which suggests the guard is meant to close this hole. Ordering the comparison so it cannot wrap fixes it:
dst_rel->r_offset > (size_t)dst_linked_sec->sec_sz - sizeof(*val)
with sec_sz >= 4 established first, or an explicit test that r_offset does not exceed sec_sz ahead of the addition.
Quoted 4 lines
tools/lib/bpf/linker.c +4
-0+pr_warn ("ELF relo #%d in section #%zu points outside the data of section '%s' in %s\n" , + j, src_sec->sec_idx, + dst_linked_sec->sec_name, + obj->filename);
tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2313
Quoted 10 lines
tools/lib/bpf/linker.c +7
-0+return -EINVAL ; + } + val = dst_linked_sec->raw_data + dst_rel->r_offset; +if (linker->swapped_endian) + *val =bswap_32 (bswap_32 (*val) + sec->dst_off); +else + *val += sec->dst_off; }else {pr_warn ("relocation against STT_SECTION in non-exec section is not supported!\n" );return -EINVAL ;
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerRe: [PATCH bpf-next v3 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations
Quoted 79 lines · On 9/19/26 11:31 PM, [email protected] wrote:
Quoted 30 lines
tools/lib/bpf/linker.c +14
-1@@ -1036,7 +1036,8 @@ static int linker_sanity_check_elf_relos(struct src_obj *obj, struct src_sec *sesize_t sym_type =ELF64_R_TYPE (relo->r_info);if (sym_type !=R_BPF_64_64 && sym_type !=R_BPF_64_32 && - sym_type !=R_BPF_64_ABS64 && sym_type !=R_BPF_64_ABS32 ) { + sym_type !=R_BPF_64_ABS64 && sym_type !=R_BPF_64_ABS32 && + sym_type !=R_BPF_64_NODYLD32 ) {pr_warn ("ELF relo #%d in section #%zu has unexpected type %zu in %s\n" , i, sec->sec_idx, sym_type, obj->filename);return -EINVAL ;@@ -2274,6 +2275,39 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob insn->imm += sec->dst_off /sizeof (struct bpf_insn ); else insn->imm += sec->dst_off; + }else if (sym_type ==R_BPF_64_NODYLD32 || + sym_type ==R_BPF_64_ABS32 ) { +__u32 *val; + +/* Two spellings of the one thing: LLVM +* emits NODYLD32 for a .long against a +* section symbol, GNU as emits ABS32 +* (bpf_reloc_type_lookup() maps +* BFD_RELOC_32 to it), and the value +* they describe is the same. +*/ +tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2299
Quoted 10 lines
tools/lib/bpf/linker.c +10
-0+/* An SHT_NOBITS section has no +* raw_data, and r_offset is sanity +* checked only for sections holding +* instructions. Check both here +* before writing. +*/ +if (!dst_linked_sec->raw_data || + dst_rel->r_offset %sizeof (*val) || + dst_rel->r_offset +sizeof (*val) > + (size_t )dst_linked_sec->sec_sz) {^^^^Can the addition here wrap around? Since
dst_rel->r_offsetisElf64_Addr(uint64_t) andsizeof(*val)issize_t, the expressiondst_rel->r_offset + sizeof(*val) > (size_t)dst_linked_sec->sec_szis evaluated in unsigned 64-bit arithmetic. For
r_offset==0xFFFFFFFFFFFFFFFC(that is, 2^64 - 4):r_offset % sizeof(*val) is 0 (since 2^64 - 4 is a multiple of 4) r_offset + 4 wraps to 0 0 > (size_t)sec_sz is falseso all three sub-tests pass, and the code falls through to
val = dst_linked_sec->raw_data + dst_rel->r_offset; /* raw_data - 4 */ *val += sec->dst_off;which performs a 4-byte read-modify-write immediately before the buffer obtained from
realloc(), corrupting the heap chunk header.
r_offsetis fully controlled by the input object and is never bounds-checked. The existingr_offsetvalidation inlinker_sanity_check_elf_relos()(near the top of this diff) is an alignment-only test and applies only toSHF_EXECINSTRsections. The .bpf_cleanup section is not executable, so nothing constrainsr_offsetbefore this point.Reached via:
bpf_linker__add_file()->bpf_linker_add_file()->linker_append_elf_relos()-> this arm. Triggered by any object with a .rel.bpf_cleanup entry of typeR_BPF_64_ABS32orR_BPF_64_NODYLD32against anSTT_SECTIONsymbol andr_offset== 2^64 - 4.The commit message notes "
r_offsetis checked for alignment only where the section holds instructions. Refuse those rather than write through them", which suggests the guard is meant to close this hole. Ordering the comparison so it cannot wrap fixes it:
Quoted 4 lines
dst_rel->r_offset > (size_t)dst_linked_sec->sec_sz - sizeof(*val)with
sec_sz>= 4 established first, or an explicit test thatr_offsetdoes not exceedsec_szahead of the addition.
Yes, wrap arround is possible. Will fix.
Quoted 21 lines
Quoted 5 lines
tools/lib/bpf/linker.c +4
-0+pr_warn ("ELF relo #%d in section #%zu points outside the data of section '%s' in %s\n" , + j, src_sec->sec_idx, + dst_linked_sec->sec_name, + obj->filename);tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2313
Quoted 10 lines
tools/lib/bpf/linker.c +7
-0+return -EINVAL ; + } + val = dst_linked_sec->raw_data + dst_rel->r_offset; +if (linker->swapped_endian) + *val =bswap_32 (bswap_32 (*val) + sec->dst_off); +else + *val += sec->dst_off; }else {pr_warn ("relocation against STT_SECTION in non-exec section is not supported!\n" );return -EINVAL ;---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerPATCH v3 18/20 selftests/bpf: Add an end-to-end .bpf_cleanup exception test
In reply to Yonghong SongC has no unwinding, so nothing here comes out of the frontend: the frames that own a resource are written as __naked inline assembly, which spells out by hand exactly what a frontend emits -- a call site bracketed by two labels, a landing pad unreachable in the compiler's CFG, and a .bpf_cleanup record tying them together. The assembler turns ".long <text label>" into the same R_BPF_64_NODYLD32 relocation the BPF AsmPrinter emits, so libbpf and the kernel see an object indistinguishable from a compiler-generated one.
Call chain: entry -> foo1 -> foo1v -> foo2 -> foo3. foo3 holds a non-preemptible section and throws inside it; foo2 holds an RCU read lock and has two call sites sharing one pad, one of them its own throw; foo1v is a void frame whose pad ends in a jump to a resume block placed after an unrelated block that ends in a plain exit; foo1 owns nothing and gets no record; entry is the boundary.
There are also some shapes the kernel refuses -- the ones with no correct answer, and the ones a pad running on the bpf_throw() walker's stack cannot express -- plus the return-value rejection that delivering an exception at the boundary of a program type which constrains its return value produces. The test skips rather than fails where the JIT cannot dispatch a landing pad at all.
--- .../selftests/bpf/exceptions_cleanup.h | 29 + .../bpf/prog_tests/exceptions_cleanup.c | 80 +++ .../selftests/bpf/progs/exceptions_cleanup.c | 152 ++++ .../bpf/progs/exceptions_cleanup_fail.c | 662 ++++++++++++++++++ 4 files changed, 923 insertions(+) create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
tools/testing/selftests/bpf/exceptions_cleanup.h new file +29 -0
@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#ifndef __EXCEPTIONS_CLEANUP_H__ +#define __EXCEPTIONS_CLEANUP_H__ + +#define THROW_COOKIE 0x100 + +/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */ +#define RAN_FOO3_PREEMPT 0x1 +#define RAN_FOO2_RCU 0x2 +#define RAN_FOO1V_PREEMPT 0x4 +#define RAN_FOO2_DROP 0x8 +#define RAN_BUMP 0x10 + +#define CLEANUP_REC (begin, end, landing_pad) \ + ".pushsection .bpf_cleanup,\"a\",@progbits;" \ + ".long " begin ";" \ + ".long " end ";" \ + ".long " landing_pad ";" \ + ".popsection;" + +/* Set a bit in @pads_ran. */ +#define PAD_RAN (bit) \ + "r1 = %[pads_ran] ll;" \ + "r2 = *(u64 *)(r1 + 0);" \ + "r2 |= " bit ";" \ + "*(u64 *)(r1 + 0) = r2;" + +#endif /* __EXCEPTIONS_CLEANUP_H__ */
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +80 -0
80 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_cleanup.c new file +152 -0
152 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +662 -0
662 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v3 18/20] selftests/bpf: Add an end-to-end .bpf_cleanup exception test
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] bpf: multi-line comments violate mandatory subsystem style --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: multi-line comments violate mandatory subsystem style
--
commit f500599d1234044cd45952e76c1ac50f7c35ab78
Author: Yonghong Song <[email protected]>
selftests/bpf: Add an end-to-end .bpf_cleanup exception test
This commit introduces end-to-end tests for the .bpf_cleanup exception handling mechanism. It verifies both successful cleanup paths and various scenarios where the kernel correctly rejects invalid landing pads or unwind behaviors.
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +0
-0
[ ... ]
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c void test_exceptions_cleanup(void) · line 65
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +8
-0@@ -58,11 +58,11 @@ void test_exceptions_cleanup(void) + +/* foo3 throws; every pad runs and the cookie is delivered at entry. */ +if (test__start_subtest ("throw_from_foo3" )) +run (skel,101 ,THROW_COOKIE ,PADS_FOO3_THREW ); + +/* foo3 returns 2 ^ 1 == 3, so foo2 throws from its own second region; +* foo3's frame is long gone, so its pad must not run. +*/
[Severity: Low]
This isn't a bug, but does this violate the BPF subsystem's multi-line comment style in test_exceptions_cleanup()? The guidelines require the opening /* to be on its own line.
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +4
-0+if (test__start_subtest ("throw_from_foo2" )) +run (skel,2 ,THROW_COOKIE ,PADS_FOO2_THREW ); + +exceptions_cleanup__destroy (skel);
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +0
-0
[ ... ]
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c static __used __noinline void __kfunc_btf_anchor(void) · line 22
Quoted 9 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +8
-0@@ -15,10 +15,10 @@ static __used __noinline void __kfunc_btf_anchor(void) +bpf_preempt_enable (); +bpf_unwind_resume (); +} + +/* 1. A subprogram that may unwind, also used as a helper callback: +* bpf_loop()'s own kernel frame would end the walk before it found a +* boundary. +*/
[Severity: Low]
This isn't a bug, but does this multi-line comment format violate the BPF subsystem style just above throwing_cb()? Similar to the above, the opening /* should be on its own line. It appears several other comments in this file follow this same pattern.
Quoted 4 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +4
-0+static int throwing_cb (__u32 idx,void *ctx) +{ +bpf_throw (0xbad ); +return 0 ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
Yonghong Song
reviewerPATCH v3 19/20 selftests/bpf: Cover the exception cleanup shapes the chain does not reach
In reply to Yonghong SongThe end-to-end test walks one call chain with a pad in most of its frames. This adds the shapes that chain does not reach: what a cleanup table leaves dead for the sweep, a callee called from both a covered and an uncovered site, a pad that reads its frame's callee-saved registers, a tail-call-reachable callee, a pad in the main program's own frame, a tail call that is taken, an extension standing in for a covered call, a throwing subprogram named by a BPF_PSEUDO_FUNC, a record covering bpf_throw() itself, a pad that calls a subprogram an extension can replace, a covered bpf_throw() the sweep leaves last, a region ending on a 16-byte instruction, a pad that reloads from and writes to its own frame, a pad two frames up, a record covering only a nounwind call, a pad terminated by _Unwind_Resume rather than bpf_unwind_resume, a pad that calls a subprogram which tail calls, the tail-call target carrying a table of its own, an extension carrying a table of its own, a pad whose first instruction is a nop, a pad that indexes its frame by a register the frame set before the throwing call, and the same over a throwing global subprogram.
--- .../selftests/bpf/exceptions_cleanup.h | 23 + .../bpf/prog_tests/exceptions_cleanup.c | 346 ++++++ .../bpf/progs/exceptions_cleanup_ext_table.c | 48 + .../bpf/progs/exceptions_cleanup_freplace.c | 17 + .../progs/exceptions_cleanup_pad_freplace.c | 17 + .../bpf/progs/exceptions_cleanup_shapes.c | 984 ++++++++++++++++++ 6 files changed, 1435 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
tools/testing/selftests/bpf/exceptions_cleanup.h +23 -0
@@ -4,6 +4,7 @@#define __EXCEPTIONS_CLEANUP_H__#define THROW_COOKIE 0x100 +#define INNER_COOKIE 0x200/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */ #define RAN_FOO3_PREEMPT 0x1
@@ -12,6 +13,28 @@#define RAN_FOO2_DROP 0x8#define RAN_BUMP 0x10 +/* progs/exceptions_cleanup_shapes.c: one bit per shape, numbered its own way. */ +#define RAN_SWEEP 0x1 +#define RAN_SHARED 0x2 +#define RAN_REGS 0x4 +#define RAN_TAIL_CALL 0x8 +#define RAN_MAIN_PAD 0x10 +#define RAN_TC_TAKEN 0x20 +#define RAN_FREPLACE 0x40 +#define RAN_ADDR_TAKEN 0x80 +#define RAN_NO_SUBPROG 0x100 +#define RAN_PAD_CALLS 0x200 +#define RAN_PAD_FIRST 0x400 +#define RAN_WIDE_REC 0x800 +#define RAN_PAD_STACK 0x1000 +#define RAN_DEEP_PAD 0x2000 +#define RAN_NOUNWIND_REC 0x4000 +#define RAN_RESUME_ALIAS 0x8000 +#define RAN_PAD_TAIL_CALL 0x10000 +#define RAN_NOP_PAD 0x20000 +#define RAN_VAR_STACK 0x40000 +#define RAN_GLOBAL_PAD 0x80000 +#define CLEANUP_REC (begin, end, landing_pad) \ ".pushsection .bpf_cleanup,\"a\",@progbits;" \ ".long " begin ";" \
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +346 -0
363 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c new file +48 -0
48 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "exceptions_cleanup.h" + +/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */ +int btf_int_anchor; + +SEC("freplace/fr_callee") +__u64 new_fr_callee (__u64 x) +{ +bpf_throw (THROW_COOKIE ); +return 0 ; +} + +char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "exceptions_cleanup.h" + +/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */ +int btf_int_anchor; + +SEC("freplace/pad_callee") +__u64 new_pad_callee (__u64 x) +{ +bpf_throw (INNER_COOKIE ); +return 0 ; +} + +char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +984 -0
984 lines not on this page · raw message
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v3 19/20] selftests/bpf: Cover the exception cleanup shapes the chain does not reach
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] bpf: multi-line comments violate the modern BPF subsystem style --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: multi-line comments violate the modern BPF subsystem style
--
commit 31d63d33bfdfb4e044e29aa2e2717a6f909808c9
Author: Yonghong Song <[email protected]>
selftests/bpf: Cover the exception cleanup shapes the chain does not reach
This commit adds tests to cover exception cleanup scenarios that the call chain doesn't reach. It introduces new shapes like dead continuations, shared callees, and throwing subprograms in BPF selftests.
Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +0
-0
[ ... ]
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 224
Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +8
-0+/* A pad in the main program's own frame, not in a subprogram. */ +if (test__start_subtest ("main_program_pad" )) +run_shape (skel, skel->progs.entry_main_pad,101 ,THROW_COOKIE , +RAN_MAIN_PAD ); + +/* The same call site either way: the subprogram's throw unwinds into +* this frame and runs its pad, an extension's stops at its own boundary. +*/
[Severity: Low]
This isn't a bug, but does this comment in test_shapes() violate the modern BPF subsystem multi-line comment style? The opening /* is preferred to be on its own line.
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +5
-0+if (test__start_subtest ("freplace_subprog_throws" )) +run_shape (skel, skel->progs.entry_freplace,7 ,THROW_COOKIE , +RAN_FREPLACE ); +if (test__start_subtest ("freplace_extension_throws" )) +test_freplace (skel);
[ ... ]
Quoted 3 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +0
-0
[ ... ]
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 33
Quoted 8 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +8
-0+__u64 magic =0x5eed ; +__u64 pads_ran =0 ; +__u64 pad_runs =0 ; + +/* 1. Everything a cleanup table leaves dead: a throw's continuation, the tail +* after a resume with an ld_imm64 and a branch in it, and the block only that +* continuation reaches. +*/
[Severity: Low]
This isn't a bug, but does this multi-line comment above sweep_frame() follow the preferred BPF subsystem format? The opening /* should typically reside on its own line.
Quoted 2 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +2
-0+static __used __naked __noinline __u64 sweep_frame(void) +{
[ ... ]
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 83
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +10
-0+SEC("syscall") +int entry_sweep (void *ctx) +{ +return sweep_frame (); +} + +/* 2. A callee called from both a covered and an uncovered site: the pad +* belongs to the call site, not the callee. Either site can throw, and the RCU +* lock between them is what an unwind out of the uncovered one leaks. +*/
[Severity: Low]
This isn't a bug, but similarly, does this comment above shared_callee() need the opening /* placed on its own line to match the modern BPF comment style?
Quoted 3 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +3
-0+static __used __noinline __u64 shared_callee(__u64 x) +{ +if (x >100 )
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19
[email protected]
AI reviewRe: [PATCH bpf-next v3 19/20] selftests/bpf: Cover the exception cleanup shapes the chain does not reach
In reply to Yonghong SongDoes the shape 2 header comment accurately describe what the code does?
Quoted 5 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +0
-0
[ ... ]
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +10
-0+/* 2. A callee called from both a covered and an uncovered site: the pad +* belongs to the call site, not the callee. Either site can throw, and the RCU +* lock between them is what an unwind out of the uncovered one leaks. +*/ +static __used __noinline __u64 shared_callee(__u64 x) +{ +if (x >100 ) +bpf_throw (THROW_COOKIE ); +return x +1 ; +}
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 99
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +10
-0+ +static __used __naked __noinline __u64 shared_frame(void) +{ + asmvolatile ( +"r1 = %[input] ll;" +"r6 = *(u64 *)(r1 + 0);" +"r1 = %[outer_input] ll;" +"r1 = *(u64 *)(r1 + 0);" +"call shared_callee;" /* uncovered: no pad for its unwind */ + "call bpf_rcu_read_lock;"
Does the shape 2 header comment accurately describe what the code does?
The comment states "the RCU lock between them is what an unwind out of the uncovered one leaks", but the uncovered call to shared_callee() happens before bpf_rcu_read_lock() is invoked. When the uncovered site throws (which prog_tests/exceptions_cleanup.c tests in the shared_callee_uncovered_throw subtest with outer_input = 101, input = 1), active_rcu_locks is 0 and there is nothing to leak.
The verifier would also reject a throw with the lock held via check_resource_leak() in kernel/bpf/verifier.c, which checks at unwind_finish() and refuses with "bpf_throw cannot be used inside bpf_rcu_read_lock-ed region" if active_rcu_locks is set.
The lock sits between the two call sites so that only the covered one unwinds with it held, making the pad's bpf_rcu_read_unlock() observable.
Should the comment describe that the lock is taken between the two sites rather than before the uncovered one?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerRe: [PATCH bpf-next v3 19/20] selftests/bpf: Cover the exception cleanup shapes the chain does not reach
Quoted 41 lines · On 9/19/26 11:46 PM, [email protected] wrote:
Quoted 5 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +0
-0[ ... ]
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +10
-0+/* 2. A callee called from both a covered and an uncovered site: the pad +* belongs to the call site, not the callee. Either site can throw, and the RCU +* lock between them is what an unwind out of the uncovered one leaks. +*/ +static __used __noinline __u64 shared_callee(__u64 x) +{ +if (x >100 ) +bpf_throw (THROW_COOKIE ); +return x +1 ; +}tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 99
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +10
-0+ +static __used __naked __noinline __u64 shared_frame(void) +{ + asmvolatile ( +"r1 = %[input] ll;" +"r6 = *(u64 *)(r1 + 0);" +"r1 = %[outer_input] ll;" +"r1 = *(u64 *)(r1 + 0);" +"call shared_callee;" /* uncovered: no pad for its unwind */ + "call bpf_rcu_read_lock;"Does the shape 2 header comment accurately describe what the code does?
The comment states "the RCU lock between them is what an unwind out of the uncovered one leaks", but the uncovered call to
shared_callee()happens beforebpf_rcu_read_lock()is invoked. When the uncovered site throws (whichprog_tests/exceptions_cleanup.ctests in theshared_callee_uncovered_throwsubtest withouter_input= 101, input = 1),active_rcu_locksis 0 and there is nothing to leak.The verifier would also reject a throw with the lock held via
check_resource_leak()inkernel/bpf/verifier.c, which checks atunwind_finish()and refuses with "bpf_throwcannot be used inside bpf_rcu_read_lock-ed region" ifactive_rcu_locksis set.
Quoted 5 lines
The lock sits between the two call sites so that only the covered one unwinds with it held, making the pad's
bpf_rcu_read_unlock()observable.Should the comment describe that the lock is taken between the two sites rather than before the uncovered one?
Yes, will change comment like
... the RCU lock is taken between them, so only the covered call unwinds holding it.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerPATCH v3 20/20 selftests/bpf: Load an exception cleanup program from a light skeleton
In reply to Yonghong SongFor a light skeleton, libbpf hands the records to bpf_gen__prog_load(), which writes them into a blob and emits a loader program that issues BPF_PROG_LOAD from inside the kernel. Nothing about that is shared with the ordinary path: the attr is built field by field, and the kfunc names are resolved by the loader program when it runs.
exceptions_cleanup_light.c is the smallest program that can tell whether a table survives that trip: one record covering the bpf_throw() call itself, one pad that sets a bit and resumes. If the table arrives, the pad runs and the cookie comes back; if it does not, the program does not load at all.
--- tools/testing/selftests/bpf/Makefile | 2 +- .../selftests/bpf/exceptions_cleanup.h | 3 ++ .../bpf/prog_tests/exceptions_cleanup.c | 28 +++++++++++++ .../bpf/progs/exceptions_cleanup_light.c | 39 +++++++++++++++++++ 4 files changed, 71 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c
tools/testing/selftests/bpf/Makefile +1 -1
@@ -523,7 +523,7 @@ LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ LSKELS := fexit_sleep.c trace_printk.c trace_vprintk.c map_ptr_kern.c \ core_kern.c core_kern_overflow.c test_ringbuf.c \ test_ringbuf_n.c test_ringbuf_map_key.c test_ringbuf_write.c \ - test_ringbuf_overwrite.c + test_ringbuf_overwrite.c exceptions_cleanup_light.c LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
tools/testing/selftests/bpf/exceptions_cleanup.h +3 -0
@@ -35,6 +35,9 @@#define RAN_VAR_STACK 0x40000#define RAN_GLOBAL_PAD 0x80000 +/* progs/exceptions_cleanup_light.c: the one pad it has. */ +#define RAN_LIGHT 0x1 +#define CLEANUP_REC (begin, end, landing_pad) \ ".pushsection .bpf_cleanup,\"a\",@progbits;" \ ".long " begin ";" \
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +28 -0
@@ -8,6 +8,7 @@#include "exceptions_cleanup_freplace.skel.h" #include "exceptions_cleanup_pad_freplace.skel.h" #include "exceptions_cleanup_ext_table.skel.h" +#include "exceptions_cleanup_light.lskel.h" /* foo3 threw: every frame that has a pad ran it. */ #define PADS_FOO3_THREW \
@@ -170,6 +171,30 @@ static void test_ext_table(struct exceptions_cleanup_shapes *skel)exceptions_cleanup_ext_table__destroy (fr); } +static void test_light_skeleton (void ) +{ +struct exceptions_cleanup_light_lskel *skel; +__u64 ctx =0 ; +int err; + +LIBBPF_OPTS (bpf_test_run_opts, topts, + .ctx_in = &ctx, + .ctx_size_in = sizeof(ctx), + ); + + skel =exceptions_cleanup_light_lskel__open_and_load (); +if (!ASSERT_OK_PTR (skel,"light open_and_load" )) +return ; + + err =bpf_prog_test_run_opts (skel->progs.entry_light.prog_fd, &topts); +if (!ASSERT_OK (err,"run" )) + goto out; +ASSERT_EQ (topts.retval,THROW_COOKIE ,"retval" ); +ASSERT_EQ (skel->bss->pads_ran,RAN_LIGHT ,"pads_ran" ); +out: +exceptions_cleanup_light_lskel__destroy (skel); +} +static void test_shapes (void ) {struct exceptions_cleanup_shapes *skel;
@@ -420,6 +445,9 @@ void test_exceptions_cleanup(void)exceptions_cleanup__destroy (skel); +if (test__start_subtest ("light_skeleton" )) +test_light_skeleton (); +test_shapes ();RUN_TESTS (exceptions_cleanup_fail);
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +39 -0
@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" +#include "exceptions_cleanup.h" + +static __used __noinline void __kfunc_btf_anchor(void) +{ +bpf_throw (0 ); +bpf_preempt_disable (); +bpf_preempt_enable (); +bpf_unwind_resume (); +} + +__u64 pads_ran =0 ; + +SEC("syscall") +__naked int entry_light (void ) +{ + asmvolatile ( +"call bpf_preempt_disable;" +"r1 = %[cookie];" +"1:" "call bpf_throw;" /* cleanup region */ +"2:" +"exit;" +"3:" /* landing pad */ +"call bpf_preempt_enable;" + PAD_RAN("%[ran]") +"call bpf_unwind_resume;" +"exit;" +CLEANUP_REC ("1b" ,"2b" ,"3b" ) + : + : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_LIGHT), +__imm_addr (pads_ran) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next v3 20/20] selftests/bpf: Load an exception cleanup program from a light skeleton
Are the kfunc declarations in scope here? The file includes only <vmlinux.h>, <bpf/bpf_helpers.h>, "bpf_misc.h", and "exceptions_cleanup.h".
Quoted 10 lines
tools/testing/selftests/bpf/Makefile +1
-1@@ -523,7 +523,7 @@ LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ LSKELS := fexit_sleep.c trace_printk.c trace_vprintk.c map_ptr_kern.c \ core_kern.c core_kern_overflow.c test_ringbuf.c \ test_ringbuf_n.c test_ringbuf_map_key.c test_ringbuf_write.c \ - test_ringbuf_overwrite.c + test_ringbuf_overwrite.c exceptions_cleanup_light.c
[ ... ]
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +4
-0@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h>
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c line 14
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +10
-0+#include "bpf_misc.h" +#include "exceptions_cleanup.h" + +static __used __noinline void __kfunc_btf_anchor(void) +{ +bpf_throw (0 ); +bpf_preempt_disable (); +bpf_preempt_enable (); +bpf_unwind_resume (); +}
Are the kfunc declarations in scope here? The file includes only <vmlinux.h>, <bpf/bpf_helpers.h>, "bpf_misc.h", and "exceptions_cleanup.h".
bpf_throw(), bpf_preempt_disable(), and bpf_preempt_enable() are normally declared with __ksym in tools/testing/selftests/bpf/ bpf_experimental.h, which is not included. bpf_unwind_resume() has no declaration in any userspace header in the tree.
Without __ksym == __attribute__((section(".ksyms"))), these become implicit function declarations, and libbpf's bpf_object__collect_externs() will reject any extern not in a ".ksyms" DATASEC when it tries to relocate the calls in the inline asm below.
The __kfunc_btf_anchor() function exists specifically to produce extern BTF FUNCs that libbpf can relocate for the 'call bpf_throw' and 'call bpf_unwind_resume' sites in the inline asm. If the declarations are implicit, either the compile fails or bpftool rejects the object when generating the light skeleton.
Should this file include "bpf_experimental.h" and add a declaration for bpf_unwind_resume()? The identical pattern appears in progs/exceptions_cleanup.c, progs/exceptions_cleanup_shapes.c, and progs/exceptions_cleanup_ext_table.c from earlier commits in this series, so the fix may need to be series-wide.
Quoted 15 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +15
-0+ +__u64 pads_ran =0 ; + +SEC("syscall") +__naked int entry_light (void ) +{ + asmvolatile ( +"call bpf_preempt_disable;" +"r1 = %[cookie];" +"1:" "call bpf_throw;" /* cleanup region */ +"2:" +"exit;" +"3:" /* landing pad */ +"call bpf_preempt_enable;" + PAD_RAN("%[ran]")
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c line 39
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +10
-0+"call bpf_unwind_resume;" +"exit;" +CLEANUP_REC ("1b" ,"2b" ,"3b" ) + : + : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_LIGHT), +__imm_addr (pads_ran) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL";
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538
Yonghong Song
reviewerRe: [PATCH bpf-next v3 20/20] selftests/bpf: Load an exception cleanup program from a light skeleton
Quoted 36 lines · On 9/19/26 11:31 PM, [email protected] wrote:
Quoted 10 lines
tools/testing/selftests/bpf/Makefile +1
-1@@ -523,7 +523,7 @@ LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ LSKELS := fexit_sleep.c trace_printk.c trace_vprintk.c map_ptr_kern.c \ core_kern.c core_kern_overflow.c test_ringbuf.c \ test_ringbuf_n.c test_ringbuf_map_key.c test_ringbuf_write.c \ - test_ringbuf_overwrite.c + test_ringbuf_overwrite.c exceptions_cleanup_light.c[ ... ]
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +4
-0@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c line 14
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +10
-0+#include "bpf_misc.h" +#include "exceptions_cleanup.h" + +static __used __noinline void __kfunc_btf_anchor(void) +{ +bpf_throw (0 ); +bpf_preempt_disable (); +bpf_preempt_enable (); +bpf_unwind_resume (); +}Are the kfunc declarations in scope here? The file includes only <
vmlinux.h>, <bpf/bpf_helpers.h>, "bpf_misc.h", and "exceptions_cleanup.h".
Quoted 4 lines
bpf_throw(),bpf_preempt_disable(), andbpf_preempt_enable()are normally declared with__ksymintools/testing/selftests/bpf/bpf_experimental.h, which is not included.bpf_unwind_resume()has no declaration in any userspace header in the tree.
I prefer to use vmlinux.h.
Quoted 49 lines
Without
__ksym== __attribute__((section(".ksyms"))), these become implicit function declarations, and libbpf'sbpf_object__collect_externs()will reject any extern not in a ".ksyms" DATASEC when it tries to relocate the calls in the inline asm below.The
__kfunc_btf_anchor()function exists specifically to produce extern BTF FUNCs that libbpf can relocate for the 'callbpf_throw' and 'callbpf_unwind_resume' sites in the inline asm. If the declarations are implicit, either the compile fails or bpftool rejects the object when generating the light skeleton.Should this file include "
bpf_experimental.h" and add a declaration forbpf_unwind_resume()? The identical pattern appears inprogs/exceptions_cleanup.c,progs/exceptions_cleanup_shapes.c, andprogs/exceptions_cleanup_ext_table.cfrom earlier commits in this series, so the fix may need to be series-wide.Quoted 15 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +15
-0+ +__u64 pads_ran =0 ; + +SEC("syscall") +__naked int entry_light (void ) +{ + asmvolatile ( +"call bpf_preempt_disable;" +"r1 = %[cookie];" +"1:" "call bpf_throw;" /* cleanup region */ +"2:" +"exit;" +"3:" /* landing pad */ +"call bpf_preempt_enable;" + PAD_RAN("%[ran]")tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c line 39
Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +10
-0+"call bpf_unwind_resume;" +"exit;" +CLEANUP_REC ("1b" ,"2b" ,"3b" ) + : + : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_LIGHT), +__imm_addr (pads_ran) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL";---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/35492765538