Skip to contentSkip to conversation

bpf: Run exception cleanup landing pads when bpf_throw() unwinds

PATCH Superseded Yonghong Song 5 participants 49 messages Sep 23 – Sep 24

Subsystems (9)
x86 Kselftest BPF arm64 +5 more subsystems

v5 of 12 Superseded Latest is v12 →

lore
Share & tools
lore

Yonghong Song

author reviewer
#0 lore raw

v5 cover bpf: Run exception cleanup landing pads when bpf_throw() unwinds

bpf_throw() walks the BPF call stack to the exception boundary and discards every frame in between. A frame that owns something -- an RCU read lock, a preemption-disabled section, a referenced kptr -- never gets to give it back, so the verifier refuses to let such a frame throw at all. That is the whole reason a Rust program cannot use bpf_throw() as its panic path right now: Rust's Drop glue *is* that give-back, and there is nowhere to run it.

LLVM 23 added the compiler half ([1]). A Rust function that owns a value across a call that can unwind

    fn foo() {
        let _guard = RcuReadGuard::new();  /* bpf_rcu_read_lock()   */
        may_throw();                       /* extern "C-unwind"     */
    }                                      /* Drop: rcu_read_unlock */

lowers to an invoke with a cleanup landing pad holding the Drop call, and the BPF backend writes one record per invoke region into a .bpf_cleanup
section: a flat table of 12-byte (begin, end, landing_pad) triples, each field a byte offset into the code section. The rule is "if a frame unwinds with its return address in [begin, end), run landing_pad before discarding it". A pad ends with a call to _Unwind_Resume(), which the kernel provides as the bpf_unwind_resume() kfunc.

This series is the kernel half: take that table at BPF_PROG_LOAD, teach the verifier that a covered call can also go to its landing pad, and have bpf_throw() run the pads as it walks.

C has no unwinding, so the selftests spell out by hand what a frontend emits -- a call site bracketed by two labels, a landing pad, and a record tying them together. The frame above, written that way:

"call bpf_rcu_read_lock;"
"1:" "call foo3;" /* cleanup region */
"2:"
... normal path, ends in bpf_rcu_read_unlock ...
"6:" /* landing pad */
"call bpf_rcu_read_unlock;"
"call bpf_unwind_resume;" CLEANUP_REC("1b", "2b", "6b")

Right now that program does not load: the throw inside foo3 is reported as "bpf_throw cannot be used inside bpf_rcu_read_lock-ed region", because as far as the verifier is concerned nothing will ever unlock. With the series the verifier walks the unwind the same way the run time will -- into the pad, which unlocks, then on to the next frame -- and the program both loads and releases the lock when it throws.

Design
======

A pad is run, not lowered. bpf_throw() already walks the frames with arch_bpf_stack_walk(); it now looks each frame's return address up in that (sub)program's table and calls the pad as a subroutine of the walker, with the unwinding frame's frame pointer and its callee-saved registers restored from the spill its callee's prologue left. The pad therefore sees its own frame but runs on the walker's stack, far below it, so nothing it calls can disturb the frame it is cleaning up after. The JIT turns its bpf_unwind_resume() into the way back to the walker.

The verifier walks the same thing, step for step, so the resource rules are unchanged: whatever a pad releases is released in the verifier state too, and check_resource_leak() simply moves from "a throw was seen" to the end of the walk.

1 change some fields from bool type to bitfield to save space for
struct bpf_insn_aux_data
2-3 uapi: cleanup_info in BPF_PROG_LOAD, struct bpf_cleanup_info,
and the bpf_unwind_resume() kfunc
4-10 verifier: mark the covered call sites, give them an edge to the
pad, walk the unwind, and refuse the shapes that cannot be dispatched

  11     bpf_throw(): dispatch pads while walking
  12-13  x86-64 and arm64 JITs
  14-18  libbpf: collect .bpf_cleanup, pass it to the kernel, resolve

_Unwind_Resume, carry it through the light skeleton and the static linker
19-21 selftests

Limitations
===========

- Cleanup pads only. A catch pad -- one that ends in a plain exit rather than a resume, which is what Rust's catch_unwind would need -- is
refused: the walker calls a pad as a subroutine and cannot hand a frame back its own execution. LLVM refuses type-specific catches and filters on its side as well.
- A JIT that can dispatch pads is required: x86-64 (with CONFIG_UNWINDER_ORC, which bpf_throw() already needs there) and arm64. Anywhere else the load fails with -EOPNOTSUPP rather than silently doing nothing.
- No offloaded programs, no private stack, and no combining a table with an exception callback.
- In a pad body: no tail call, no BPF_LD_[ABS|IND], no indirect jump, and no on-stack call arguments -- each of them reads or writes a stack the pad does not own.
- The Rust toolchain does not properly support BPF exception handling yet. The tables the selftests use are hand-written inline asm, which the assembler turns into the same relocations the BPF AsmPrinter emits, so libbpf and the kernel see an object indistinguishable from a compiler-generated one.

[1] https://github.com/llvm/llvm-project/pull/192164
llvm commit 9d51c891b719 ("[BPF] Add exception handling support with .bpf_cleanup section")

Changelog
=========
v4 -> v5:
- v4: https://lore.kernel.org/bpf/[email protected]/
- Rebase on bpf-next.
- Clear r0 where a throw enters a landing pad: no instruction defines it, so a precision request for it outlived the state and oopsed the verifier.
- Defer entering the frames an unwind edge crossed until the backtrack reads an instruction from them; entering at the landing pad could leave bt->frame past the parent state's frames and oops the verifier.
- Stamp the popped frame count inside bpf_push_jmp_history(), so a pad's entry carries it even when the prune path is what creates the entry.
- Replace the hand-rolled CFG traversal and its separate pass with per-instruction checks in do_check(), keyed on the verifier's unwinding state; kernel/bpf/exception.c halves.
- Add a first patch packing bpf_insn_aux_data's flags into one bit field word, 144 bytes to 128, so the flags this series adds cost bits.
- Drop the per-subprogram arrays the JITs consulted for throw sites, resume sites and pad bodies, and read insn_aux_data, which a JIT already has.
- Drop the pad-entry r0 header: r0 at a pad is an unknown scalar, and a dispatcher writes a defined value there only to keep a kernel one out of BPF.
- Take the bool arguments back out of verifier_remove_insns() and the site collector, and share pop_frame() with prepare_func_exit().
- Rename the recorded call sites to throw_call and resume_call, give the exported functions a bpf_exc_ prefix, and drop cleanup_ from the statics.
v3 -> v4:
- v3: https://lore.kernel.org/bpf/[email protected]/
- Rebase on bpf-next due to conflict.
- Reserve the throw-site spill area only in a (sub)program that calls bpf_throw(): 40 bytes of stack per frame on x86-64, 80 on arm64.
- Bound the record count by the number of instructions in the program, and name both in the message.
- Refuse a .bpf_cleanup section in libbpf whose record count cannot be handed to the kernel as a count times a record size in an int.
- Fix the static linker's new bounds check, which could itself wrap, and refuse a section too small to hold one field.
- WARN once if the body of bpf_unwind_resume() is ever reached, the way bpf_throw() does where its exception callback should never return.
- Rename nr_pad_body to pad_body_bits, use BTF_ID_LIST_SINGLE, move cleanup_pad out of insn_aux_data's bools, drop an arm64 include.
v2 -> v3:
- v2: https://lore.kernel.org/bpf/[email protected]/
- Keep a landing pad's record when opt_remove_nops() deletes a pad that is a nop, instead of dropping it after the verifier has already checked the call site against it.
- Refuse a BPF_LD_[ABS|IND] in a pad body.
- Teach mark_chain_precision() about the throw-to-pad edge, which crosses frames with no instruction to account for them.
- Refuse a bpf_unwind_resume() in any frame but the one whose landing pad the walker entered.
- Check raw_data, alignment and bounds before the static linker writes through a relocation in a non-executable section, which may be SHT_NOBITS.
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Consolidate all usages of kern_extern_name() in a single patch in libbpf.
- Avoid compiler warning and add proper cleanup_info_cnt guard in libbpf when collecting .bpf_cleanup records.
- Add cleanup_info_cnt condition for emit_rel_store() with cleanup_info.

Yonghong Song (21):
bpf: Pack bpf_insn_aux_data flags into bit fields
bpf: Accept the compiler's exception cleanup table at program load
bpf: Add the bpf_unwind_resume() kfunc
bpf: Add lookups for exception cleanup resumes and landing pads
bpf: Prepare for an exception cleanup table before the CFG walk
bpf: Make exception landing pads reachable in the CFG
bpf: Explore the landing pads no call site reaches
bpf: Walk the exception unwind in the verifier
bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch
bpf: Refuse a private stack for a program with an exception cleanup
table
bpf: Dispatch exception cleanup pads from bpf_throw()
bpf, x86: Dispatch exception cleanup pads at run time
bpf, arm64: Dispatch exception cleanup pads at run time
libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
libbpf: Add cleanup_info to bpf_prog_load_opts
libbpf: Collect .bpf_cleanup records and pass them to the kernel
libbpf: Carry the exception cleanup table through the light skeleton
libbpf: Let the static linker carry .bpf_cleanup relocations selftests/bpf: Add an end-to-end .bpf_cleanup exception test selftests/bpf: Cover the exception cleanup shapes the chain does not
reach
selftests/bpf: Load an exception cleanup program from a light skeleton

 arch/arm64/net/Makefile                       |    2 +-
 arch/arm64/net/bpf_cleanup_pad.S              |   94 ++
 arch/arm64/net/bpf_jit_comp.c                 |   92 +-
 arch/x86/net/Makefile                         |    3 +
 arch/x86/net/bpf_cleanup_pad.S                |   79 ++
 arch/x86/net/bpf_jit_comp.c                   |   81 +-
 include/linux/bpf.h                           |   88 ++
 include/linux/bpf_verifier.h                  |   64 +-
 include/linux/filter.h                        |    2 +
 include/uapi/linux/bpf.h                      |    9 +
 kernel/bpf/Makefile                           |    2 +-
 kernel/bpf/backtrack.c                        |   41 +-
 kernel/bpf/cfg.c                              |   71 +-
 kernel/bpf/check_btf.c                        |  134 +++
 kernel/bpf/core.c                             |   35 +-
 kernel/bpf/exception.c                        |  328 ++++++
 kernel/bpf/exception.h                        |   23 +
 kernel/bpf/fixups.c                           |   98 +-
 kernel/bpf/helpers.c                          |   48 +
 kernel/bpf/liveness.c                         |   19 +
 kernel/bpf/states.c                           |    6 +
 kernel/bpf/syscall.c                          |    2 +-
 kernel/bpf/verifier.c                         |  161 ++-
 tools/include/uapi/linux/bpf.h                |    9 +
 tools/lib/bpf/bpf.c                           |    6 +-
 tools/lib/bpf/bpf.h                           |    7 +-
 tools/lib/bpf/gen_loader.c                    |   29 +-
 tools/lib/bpf/libbpf.c                        |  332 +++++-
 tools/lib/bpf/libbpf_internal.h               |   10 +
 tools/lib/bpf/linker.c                        |   38 +-
 tools/testing/selftests/bpf/Makefile.skel     |    2 +-
 .../selftests/bpf/exceptions_cleanup.h        |   56 +
 .../bpf/prog_tests/exceptions_cleanup.c       |  524 ++++++++
 .../selftests/bpf/progs/exceptions_cleanup.c  |  164 +++
 .../bpf/progs/exceptions_cleanup_ext_table.c  |   48 +
 .../bpf/progs/exceptions_cleanup_fail.c       |  678 +++++++++++
 .../bpf/progs/exceptions_cleanup_freplace.c   |   17 +
 .../bpf/progs/exceptions_cleanup_light.c      |   39 +
 .../progs/exceptions_cleanup_pad_freplace.c   |   17 +
 .../bpf/progs/exceptions_cleanup_shapes.c     | 1049 +++++++++++++++++
 40 files changed, 4410 insertions(+), 97 deletions(-)
 create mode 100644 arch/arm64/net/bpf_cleanup_pad.S
 create mode 100644 arch/x86/net/bpf_cleanup_pad.S
 create mode 100644 kernel/bpf/exception.c
 create mode 100644 kernel/bpf/exception.h
 create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
Signature · 1 line
2.53.0-Meta
author reviewer
#1 lore raw

PATCH v5 1/21 bpf: Pack bpf_insn_aux_data flags into bit fields

struct bpf_insn_aux_data spreads its flags over three places: seven bools a byte each, a u8 carrying three bit fields, and a u32 group with 25 of its 32 bits spare. Put them all in one u64 word, alu_state with them, and move orig_idx below it. The structure goes from 144 bytes to 128 with no holes, and 34 bits are left for later flags.

No functional change: a one-bit unsigned field holds 0 and 1 the way the bool did, and nothing takes the address of any of them.

---
 include/linux/bpf_verifier.h | 41 ++++++++++++++++++------------------
 1 file changed, 21 insertions(+), 20 deletions(-)
include/linux/bpf_verifier.h +21 -20
index 92f528c45605..15bc61de61ef 100644
@@ -675,40 +675,41 @@ struct bpf_insn_aux_data {
 	u64 map_key_state; /* constant (32 bit) key tracking for maps */
 	int ctx_field_size; /* the ctx field size for load insn, maybe 0 */
 	u32 seen; /* this insn was processed by the verifier at env->pass_cnt */
-	bool nospec; /* do not execute this instruction speculatively */
-	bool nospec_result; /* result is unsafe under speculation, nospec must follow */
-	bool zext_dst; /* this insn zero extends dst reg */
-	bool needs_zext; /* alu op needs to clear upper bits */
-	bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
-	bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
-	bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
-	u8 alu_state; /* used in combination with alu_limit */
+	u64 nospec:1; /* do not execute this instruction speculatively */
+	u64 nospec_result:1; /* result is unsafe under speculation, nospec must follow */
+	u64 zext_dst:1; /* this insn zero extends dst reg */
+	u64 needs_zext:1; /* alu op needs to clear upper bits */
+	u64 non_sleepable:1; /* helper/kfunc may be called from non-sleepable context */
+	u64 is_iter_next:1; /* bpf_iter_<type>_next() kfunc call */
+	u64 call_with_percpu_alloc_ptr:1; /* {this,per}_cpu_ptr() with prog percpu alloc */
+	u64 alu_state:8; /* used in combination with alu_limit */
 	/* true if STX or LDX instruction is a part of a spill/fill
 	 * pattern for a bpf_fastcall call.
 	 */
-	u8 fastcall_pattern:1;
+	u64 fastcall_pattern:1;
 	/* for CALL instructions, a number of spill/fill pairs in the
 	 * bpf_fastcall pattern.
 	 */
-	u8 fastcall_spills_num:3;
-	u8 arg_prog:4;
+	u64 fastcall_spills_num:3;
+	u64 arg_prog:4;
 
-	/* below fields are initialized once */
-	unsigned int orig_idx; /* original instruction index */
-	u32 jmp_point:1;
-	u32 prune_point:1;
+	/* below flags are initialized once */
+	u64 jmp_point:1;
+	u64 prune_point:1;
 	/* ensure we check state equivalence and save state checkpoint and
 	 * this instruction, regardless of any heuristics
 	 */
-	u32 force_checkpoint:1;
+	u64 force_checkpoint:1;
 	/* true if instruction is a call to a helper function that
 	 * accepts callback function as a parameter.
 	 */
-	u32 calls_callback:1;
-	u32 indirect_target:1; /* if it is an indirect jump target */
-	u32 non_stack_access:1; /* instruction can access non-stack memory */
+	u64 calls_callback:1;
+	u64 indirect_target:1; /* if it is an indirect jump target */
+	u64 non_stack_access:1; /* instruction can access non-stack memory */
 	/* true if some jump or call instruction targets this instruction */
-	u32 jump_target:1;
+	u64 jump_target:1;
+
+	unsigned int orig_idx; /* original instruction index, initialized once */
 	/*
 	 * CFG strongly connected component this instruction belongs to,
 	 * zero if it is a singleton SCC.
Signature · 1 line
2.53.0-Meta
#2 lore raw

Re: [PATCH bpf-next v5 01/21] bpf: Pack bpf_insn_aux_data flags into bit fields

Is the old size correct? The pre-patch struct appears to be 136 bytes on x86-64, not 144, which would make the savings 8 bytes rather than 16.
Quoted 7 lines

bpf: Pack bpf_insn_aux_data flags into bit fields

struct bpf_insn_aux_data spreads its flags over three places: seven bools a byte each, a u8 carrying three bit fields, and a u32 group with 25 of its 32 bits spare. Put them all in one u64 word, alu_state with them, and move orig_idx below it. The structure goes from 144 bytes to 128 with no holes, and 34 bits are left for later flags.

Is the old size correct? The pre-patch struct appears to be 136 bytes on x86-64, not 144, which would make the savings 8 bytes rather than 16.

Looking at the layout starting at offset 60 (where the naturally packed head ends with u32 seen):

Old flag area:

  60..66  7 bools (nospec through call_with_percpu_alloc_ptr)
  67      u8 alu_state
  68      u8 with three bit fields
  69..71  3 padding bytes
  72..75  unsigned int orig_idx
  76..79  u32 with bit fields

80..131 u32 scc + 4x u16 + u32 const_reg_vals[10]

That gives sizeof 136 (aligned to 8). The new layout removes the 3-byte pad at offset 69 by packing all flags into one u64 at 60..67, followed directly by unsigned int orig_idx at 68..71, giving sizeof 128.

Could the commit message be updated to reflect the actual delta?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35821138073

maintainer Acked-by on v5 1/21
#3 lore raw

Re: [PATCH bpf-next v5 01/21] bpf: Pack bpf_insn_aux_data flags into bit fields

In reply to Yonghong Song
Quoted 4 lines · Yonghong Song

struct bpf_insn_aux_data spreads its flags over three places: seven bools a byte each, a u8 carrying three bit fields, and a u32 group with 25 of its 32 bits spare. Put them all in one u64 word, alu_state with them, and move orig_idx below it. The structure goes from 144 bytes to 128 with no holes,

The bot is right, it's 136 on master and 144 when v4 of this series is applied.

Quoted 8 lines

and 34 bits are left for later flags.

No functional change: a one-bit unsigned field holds 0 and 1 the way the bool did, and nothing takes the address of any of them.

---

...

author reviewer
#4 lore raw

PATCH v5 2/21 bpf: Accept the compiler's exception cleanup table at program load

In reply to Yonghong Song

LLVM 23 added exception handling support for BPF with the .bpf_cleanup section ([1]). Rust code compiled with panic=unwind runs cleanup code (Drop glue) when bpf_throw() fires, and the LLVM BPF backend emits that section from the landing pads the frontend produced. Plain C cannot generate .bpf_cleanup unless inline asm is used. The Rust compiler does not *properly* support BPF exception handling yet, but the kernel can support the table today, and inline assembly is enough to test it.

Add the UAPI to carry the .bpf_cleanup table into the kernel. BPF_PROG_LOAD grows cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, and struct bpf_cleanup_info describes one record as a triple of instruction indices:
the half-open call-site range [begin_off, end_off) and the landing_pad_off the frame resumes at. The table arrives sorted by begin_off, with disjoint ranges and each record's three offsets inside one subprogram; check_cleanup_info() holds it to that at load time. Nothing reads it yet; the patches that follow -- the CFG walk, the unwind walk and the JITs -- are its consumers.

---
 include/linux/bpf_verifier.h   |   2 +
 include/uapi/linux/bpf.h       |   9 +++
 kernel/bpf/check_btf.c         | 134 +++++++++++++++++++++++++++++++++
 kernel/bpf/syscall.c           |   2 +-
 kernel/bpf/verifier.c          |   1 +
 tools/include/uapi/linux/bpf.h |   9 +++
 6 files changed, 156 insertions(+), 1 deletion(-)
include/linux/bpf_verifier.h +2 -0
index 15bc61de61ef..2b6b480c059e 100644
@@ -988,6 +988,8 @@ struct bpf_verifier_env {
 	struct arg_track **callsite_at_stack;
 	u32 pass_cnt; /* number of times do_check() was called */
 	u32 subprog_cnt;
+	struct bpf_cleanup_info *cleanup_info;
+	u32 cleanup_info_cnt;
 	/* number of instructions analyzed by the verifier */
 	u32 prev_insn_processed, insn_processed;
 	/* number of jmps, calls, exits analyzed so far */
include/uapi/linux/bpf.h +9 -0
index 0aaa54359aeb..646aa8758d48 100644
@@ -1669,6 +1669,9 @@ union bpf_attr {
 		 * verification.
 		 */
 		__s32		keyring_id;
+		__aligned_u64	cleanup_info;	/* exception cleanup table */
+		__u32		cleanup_info_rec_size; /* userspace bpf_cleanup_info size */
+		__u32		cleanup_info_cnt; /* number of bpf_cleanup_info records */
 	};
 
 	struct { /* anonymous struct used by BPF_OBJ_* commands */
@@ -7599,6 +7602,12 @@ struct bpf_line_info {
 	__u32	line_col;
 };
 
+struct bpf_cleanup_info {
+	__u32	begin_off;
+	__u32	end_off;
+	__u32	landing_pad_off;
+};
+
 struct bpf_spin_lock {
 	__u32	val;
 };
kernel/bpf/check_btf.c +134 -0

146 lines not on this page · raw message

kernel/bpf/syscall.c +1 -1
index 74496fd716d3..63c84a60c1b7 100644
@@ -2924,7 +2924,7 @@ int __init __used bpf_multi_func(void) { return 0; }
 BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
 
 /* last field in 'union bpf_attr' used by this command */
-#define BPF_PROG_LOAD_LAST_FIELD keyring_id
+#define BPF_PROG_LOAD_LAST_FIELD cleanup_info_cnt
 
 static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
 {
kernel/bpf/verifier.c +1 -0
index a7c9e2d8965d..c65103152320 100644
@@ -21999,6 +21999,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	kvfree(env->succ);
 	kvfree(env->gotox_tmp_buf);
 	bpf_diag_free(env);
+	kvfree(env->cleanup_info);
 	kvfree(env);
 	return ret;
 }
tools/include/uapi/linux/bpf.h +9 -0
index 0aaa54359aeb..646aa8758d48 100644
@@ -1669,6 +1669,9 @@ union bpf_attr {
 		 * verification.
 		 */
 		__s32		keyring_id;
+		__aligned_u64	cleanup_info;	/* exception cleanup table */
+		__u32		cleanup_info_rec_size; /* userspace bpf_cleanup_info size */
+		__u32		cleanup_info_cnt; /* number of bpf_cleanup_info records */
 	};
 
 	struct { /* anonymous struct used by BPF_OBJ_* commands */
@@ -7599,6 +7602,12 @@ struct bpf_line_info {
 	__u32	line_col;
 };
 
+struct bpf_cleanup_info {
+	__u32	begin_off;
+	__u32	end_off;
+	__u32	landing_pad_off;
+};
+
 struct bpf_spin_lock {
 	__u32	val;
 };
Signature · 1 line
2.53.0-Meta
author reviewer
#5 lore raw

PATCH v5 3/21 bpf: Add the bpf_unwind_resume() kfunc

In reply to Yonghong Song

A compiler-emitted cleanup landing pad ends with a call to _Unwind_Resume(), which carries the unwind on once that frame's cleanups have run. The kernel provides the same terminator as a kfunc, named bpf_unwind_resume() to keep the 'bpf_' prefix kfunc convention; a later libbpf patch resolves the compiler's name to it.

The kfunc is defined here but not registered with any program type yet. A call to it is only ever valid inside a landing pad, and the verifier cannot say that until it knows what a landing pad is, so registration waits for the patch that adds the rule.

---
 kernel/bpf/helpers.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
kernel/bpf/helpers.c +13 -0
index 301b35bd85c8..249418e60ee4 100644
@@ -3445,6 +3445,19 @@ __bpf_kfunc void bpf_throw(u64 cookie)
 	WARN(1, "A call to BPF exception callback should never return\n");
 }
 
+/*
+ * Terminator of a compiler-emitted cleanup landing pad. The compiler names
+ * this _Unwind_Resume, the base unwind ABI's entry point for carrying an
+ * unwind on once a frame's cleanups have run. To match kernel kfunc
+ * convention, the kernel calls it bpf_unwind_resume and libbpf maps the
+ * compiler's name onto it.
+ */
+__bpf_kfunc void bpf_unwind_resume(void)
+{
+	/* Never reached: a JIT emits the way out of a landing pad instead. */
+	WARN_ONCE(1, "A JIT should have replaced the exception cleanup resume\n");
+}
+
 __bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, unsigned int flags)
 {
 	struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
Signature · 1 line
2.53.0-Meta
#6 lore raw

Re: [PATCH bpf-next v5 03/21] bpf: Add the bpf_unwind_resume() kfunc

Quoted 27 lines · Yonghong Song

A compiler-emitted cleanup landing pad ends with a call to _Unwind_Resume(), which carries the unwind on once that frame's cleanups have run. The kernel provides the same terminator as a kfunc, named bpf_unwind_resume() to keep the 'bpf_' prefix kfunc convention; a later libbpf patch resolves the compiler's name to it.

The kfunc is defined here but not registered with any program type yet. A call to it is only ever valid inside a landing pad, and the verifier cannot say that until it knows what a landing pad is, so registration waits for the patch that adds the rule.

---
 kernel/bpf/helpers.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
kernel/bpf/helpers.c +3 -0
index 301b35bd85c8..249418e60ee4 100644
@@ -3445,6 +3445,19 @@ __bpf_kfunc void bpf_throw(u64 cookie)
 	WARN(1, "A call to BPF exception callback should never return\n");
 }
 
+/*
+ * Terminator of a compiler-emitted cleanup landing pad. The compiler names
+ * this _Unwind_Resume, the base unwind ABI's entry point for carrying an

kernel/bpf/helpers.c __bpf_kfunc void bpf_throw(u64 cookie) · line 3460

Quoted 10 lines
kernel/bpf/helpers.c +10 -0
+ * unwind on once a frame's cleanups have run. To match kernel kfunc
+ * convention, the kernel calls it bpf_unwind_resume and libbpf maps the
+ * compiler's name onto it.
+ */
+__bpf_kfunc void bpf_unwind_resume(void)
+{
+	/* Never reached: a JIT emits the way out of a landing pad instead. */
+	WARN_ONCE(1, "A JIT should have replaced the exception cleanup resume\n");
+}
+

Can we change this to void bpf_unwind_resume(void *ptr__ign)?

Itanium ABI passes the exception descriptor to _Unwind_Resume. We do not need it right now, but I feel like it might be better to take and ignore it. Changing it later might be more expensive, while it costs nothing to change for now.

author reviewer
#7 lore raw

PATCH v5 4/21 bpf: Add lookups for exception cleanup resumes and landing pads

In reply to Yonghong Song

Add two new files, exception.h and exception.c, to host the exception handling code. Only a few helpers so far: recognising a call to bpf_unwind_resume(), and asking which landing pad, if any, a call site unwinds to.

The pad of a call site is kept in insn_aux_data, so the two places that move instructions around -- bpf_patch_insn_data() and verifier_remove_insns() -- learn to keep it in step.

---
 include/linux/bpf_verifier.h |  6 ++++++
 kernel/bpf/Makefile          |  2 +-
 kernel/bpf/exception.c       | 23 +++++++++++++++++++++++
 kernel/bpf/exception.h       | 12 ++++++++++++
 kernel/bpf/fixups.c          | 26 +++++++++++++++++++++++++-
 5 files changed, 67 insertions(+), 2 deletions(-)
 create mode 100644 kernel/bpf/exception.c
 create mode 100644 kernel/bpf/exception.h
include/linux/bpf_verifier.h +6 -0
index 2b6b480c059e..666c73308b32 100644
@@ -710,6 +710,11 @@ struct bpf_insn_aux_data {
 	u64 jump_target:1;
 
 	unsigned int orig_idx; /* original instruction index, initialized once */
+	/*
+	 * 1 + the instruction index of the exception cleanup landing pad
+	 * this call site unwinds to, or 0 for none.
+	 */
+	u32 cleanup_pad;
 	/*
 	 * CFG strongly connected component this instruction belongs to,
 	 * zero if it is a singleton SCC.
@@ -1519,6 +1524,7 @@ u32 btf_func_arg_align(const struct btf *btf, const struct btf_type *t);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn);
 int bpf_compute_const_regs(struct bpf_verifier_env *env);
 int bpf_prune_dead_branches(struct bpf_verifier_env *env);
 int bpf_check_cfg(struct bpf_verifier_env *env);
kernel/bpf/Makefile +1 -1
index 9a92c348bbda..af9bc60428ad 100644
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o exception.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
kernel/bpf/exception.c new file +23 -0
new file mode 100644
index 000000000000..6989f31aa6c9
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
+#include "exception.h"
+
+BTF_ID_LIST_SINGLE(bpf_unwind_resume_id, func, bpf_unwind_resume)
+
+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn)
+{
+	return bpf_pseudo_kfunc_call(insn) && insn->off == 0 &&
+	       insn->imm == bpf_unwind_resume_id[0];
+}
+
+int bpf_exc_pad_of_call(struct bpf_verifier_env *env, u32 idx)
+{
+	u32 pad = env->insn_aux_data[idx].cleanup_pad;
+
+	return pad ? (int)pad - 1 : -1;
+}
kernel/bpf/exception.h new file +12 -0
new file mode 100644
index 000000000000..634cb2c15ffc
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef _LINUX_BPF_EXCEPTION_H
+#define _LINUX_BPF_EXCEPTION_H
+
+#include <linux/types.h>
+
+struct bpf_verifier_env;
+
+int bpf_exc_pad_of_call(struct bpf_verifier_env *env, u32 idx);
+
+#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +25 -1
index 2add8001c3ec..e3459123803c 100644
@@ -256,11 +256,18 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 			data[i].non_stack_access =
 				data[off + cnt - 1].non_stack_access;
 			data[off + cnt - 1].non_stack_access = false;
+			data[i].cleanup_pad = data[off + cnt - 1].cleanup_pad;
+			data[off + cnt - 1].cleanup_pad = 0;
 		} else if (bpf_is_mem_insn(insn + i)) {
 			data[i].non_stack_access = true;
 		}
 	}
 
+	if (env->cleanup_info_cnt)
+		for (i = 0; i < prog_len; i++)
+			if (data[i].cleanup_pad > off + 1)
+				data[i].cleanup_pad += cnt - 1;
+
 	/*
 	 * Last slot instruction could be a newly generated
 	 * BPF_ST/BPF_LDX/BPF_STX, systematically mark it for non-stack access
@@ -549,6 +556,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 	struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
 	unsigned int orig_prog_len = env->prog->len;
 	int err;
+	u32 i;
 
 	if (bpf_prog_is_offloaded(env->prog->aux))
 		bpf_prog_offload_remove_insns(env, off, cnt);
@@ -573,6 +581,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 		sizeof(*aux_data) * (orig_prog_len - off - cnt));
 	env->insn_aux_data_len -= cnt;
 
+	if (env->cleanup_info_cnt) {
+		for (i = 0; i < env->insn_aux_data_len; i++) {
+			u32 pad = aux_data[i].cleanup_pad;
+
+			if (pad > off + cnt)
+				aux_data[i].cleanup_pad = pad - cnt;
+			else if (pad > off)
+				aux_data[i].cleanup_pad = 0;
+		}
+	}
+
 	return 0;
 }
 
@@ -648,7 +667,7 @@ int bpf_opt_remove_nops(struct bpf_verifier_env *env)
 	struct bpf_insn *insn = env->prog->insnsi;
 	int insn_cnt = env->prog->len;
 	bool is_may_goto_0, is_ja;
-	int i, err;
+	int i, j, err;
 
 	for (i = 0; i < insn_cnt; i++) {
 		is_may_goto_0 = !memcmp(&insn[i], &MAY_GOTO_0, sizeof(MAY_GOTO_0));
@@ -657,6 +676,11 @@ int bpf_opt_remove_nops(struct bpf_verifier_env *env)
 		if (!is_may_goto_0 && !is_ja)
 			continue;
 
+		if (env->cleanup_info_cnt)
+			for (j = 0; j < insn_cnt; j++)
+				if (env->insn_aux_data[j].cleanup_pad == i + 1)
+					env->insn_aux_data[j].cleanup_pad = i + 2;
+
 		err = verifier_remove_insns(env, i, 1);
 		if (err)
 			return err;
Signature · 1 line
2.53.0-Meta
author reviewer
#8 lore raw

PATCH v5 5/21 bpf: Prepare for an exception cleanup table before the CFG walk

In reply to Yonghong Song

Record in insn_aux_data what the later passes need from the cleanup table:
throw_call for every bpf_throw() and resume_call for every bpf_unwind_resume() -- the two calls a JIT lowers its own way rather than as calls -- and cleanup_pad, the landing pad a frame resumes at, for every call within the [begin_off, end_off) range of a cleanup record. Subsequent commits consume all three.

Marking the two calls here, rather than recognising them in the JIT, is what makes the recognition exact: by the time a JIT runs, bpf_fixup_kfunc_call() has rewritten every other kfunc's imm into an offset from __bpf_call_base, and a BTF id compared against one of those offsets could match an unrelated call.

bpf_prepare_cleanup_exceptions() runs before bpf_check_cfg(), whose walk consumes what it produces. It refuses a table on an offloaded program, on one whose JIT cannot dispatch landing pads or was not asked to compile it, and on one that also installs an exception callback -- two different answers to what runs on the way out. What survives is marked jit_required:
the interpreter cannot dispatch a pad. bpf_jit_supports_cleanup_pads() is weak here and says no; the arch patches provide the real ones.

---
 include/linux/bpf_verifier.h |  2 ++
 include/linux/filter.h       |  1 +
 kernel/bpf/core.c            |  5 +++
 kernel/bpf/exception.c       | 62 ++++++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h       |  1 +
 kernel/bpf/fixups.c          |  6 ++++
 kernel/bpf/verifier.c        |  6 ++++
 7 files changed, 83 insertions(+)
include/linux/bpf_verifier.h +2 -0
index 666c73308b32..e12ec91b8150 100644
@@ -692,6 +692,8 @@ struct bpf_insn_aux_data {
 	 */
 	u64 fastcall_spills_num:3;
 	u64 arg_prog:4;
+	u64 throw_call:1; /* call to bpf_throw() */
+	u64 resume_call:1; /* call to bpf_unwind_resume() */
 
 	/* below flags are initialized once */
 	u64 jmp_point:1;
include/linux/filter.h +1 -0
index 422284b4fa96..2582a7606e46 100644
@@ -1242,6 +1242,7 @@ bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_arena_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
+bool bpf_jit_supports_cleanup_pads(void);
 bool bpf_jit_supports_ptr_xchg(void);
 bool bpf_jit_supports_arena(void);
 bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
kernel/bpf/core.c +5 -0
index 227211166dcc..a379cd1ec4c6 100644
@@ -3475,6 +3475,11 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp,
 {
 }
 
+bool __weak bpf_jit_supports_cleanup_pads(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_timed_may_goto(void)
 {
 	return false;
kernel/bpf/exception.c +62 -0

70 lines not on this page · raw message

kernel/bpf/exception.h +1 -0
index 634cb2c15ffc..b96b2c429e22 100644
@@ -7,6 +7,7 @@
 
 struct bpf_verifier_env;
 
+int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env);
 int bpf_exc_pad_of_call(struct bpf_verifier_env *env, u32 idx);
 
 #endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +6 -0
index e3459123803c..fed5b66ed3b7 100644
@@ -256,6 +256,12 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 			data[i].non_stack_access =
 				data[off + cnt - 1].non_stack_access;
 			data[off + cnt - 1].non_stack_access = false;
+			data[i].throw_call =
+				data[off + cnt - 1].throw_call;
+			data[off + cnt - 1].throw_call = false;
+			data[i].resume_call =
+				data[off + cnt - 1].resume_call;
+			data[off + cnt - 1].resume_call = false;
 			data[i].cleanup_pad = data[off + cnt - 1].cleanup_pad;
 			data[off + cnt - 1].cleanup_pad = 0;
 		} else if (bpf_is_mem_insn(insn + i)) {
kernel/bpf/verifier.c +6 -0
index c65103152320..8163c75ec3fa 100644
@@ -37,6 +37,7 @@
 
 #include "diagnostics.h"
 #include "disasm.h"
+#include "exception.h"
 
 static const struct bpf_verifier_ops * const bpf_verifier_ops[] = {
 #define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
@@ -21792,6 +21793,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/* The CFG needs an edge from a call in a cleanup range to its pad. */
+	ret = bpf_prepare_cleanup_exceptions(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	/* Validate instructions and resolve the program's referenced resources. */
 	ret = check_and_resolve_insns(env);
 	if (ret < 0)
Signature · 1 line
2.53.0-Meta
#9 lore raw

Re: [PATCH bpf-next v5 05/21] bpf: Prepare for an exception cleanup table before the CFG walk

This isn't a bug, but would it help to say "for every call that can unwind -- a BPF-to-BPF call or bpf_throw()" here, so the changelog matches the filter in mark_call_sites()? The code marks only…
Quoted 6 lines

Record in insn_aux_data what the later passes need from the cleanup table:
throw_call for every bpf_throw() and resume_call for every bpf_unwind_resume() -- the two calls a JIT lowers its own way rather than as calls -- and cleanup_pad, the landing pad a frame resumes at, for every call within the [begin_off, end_off) range of a cleanup record. Subsequent commits consume all three.

This isn't a bug, but would it help to say "for every call that can unwind -- a BPF-to-BPF call or bpf_throw()" here, so the changelog matches the filter in mark_call_sites()? The code marks only bpf_pseudo_call() and bpf_is_throw_kfunc() inside the range, skipping helper and kfunc calls that cannot unwind.

include/linux/bpf_verifier.h struct bpf_insn_aux_data { · line 699

Quoted 13 lines
include/linux/bpf_verifier.h +2 -0
index 666c73308b32..e12ec91b8150 100644
@@ -692,6 +692,8 @@ struct bpf_insn_aux_data {
 	 */
 	u64 fastcall_spills_num:3;
 	u64 arg_prog:4;
+	u64 throw_call:1; /* call to bpf_throw() */
+	u64 resume_call:1; /* call to bpf_unwind_resume() */
 
 	/* below flags are initialized once */
 	u64 jmp_point:1;

This isn't a bug, but would these two bits sit more naturally below the "initialized once" comment, next to cleanup_pad which the same pass fills in? Both throw_call and resume_call are set exactly once by bpf_prepare_cleanup_exceptions() before any verification pass.

[ ... ]

Quoted 7 lines
kernel/bpf/fixups.c +0 -0
index e3459123803c..fed5b66ed3b7 100644
@@ -256,6 +256,12 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 			data[i].non_stack_access =
 				data[off + cnt - 1].non_stack_access;

kernel/bpf/fixups.c static void adjust_insn_aux_data(struct bpf_verifier_env *env, · line 267

Quoted 10 lines
kernel/bpf/fixups.c +6 -0
 			data[off + cnt - 1].non_stack_access = false;
+			data[i].throw_call =
+				data[off + cnt - 1].throw_call;
+			data[off + cnt - 1].throw_call = false;
+			data[i].resume_call =
+				data[off + cnt - 1].resume_call;
+			data[off + cnt - 1].resume_call = false;
 			data[i].cleanup_pad = data[off + cnt - 1].cleanup_pad;
 			data[off + cnt - 1].cleanup_pad = 0;
 		} else if (bpf_is_mem_insn(insn + i)) {

This isn't a bug, but could these two assignments be single lines like the cleanup_pad one just below them? Unwrapped, they are 76 and 78 columns including the three tabs, so they fit.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35821138073

author reviewer
#10 lore raw

PATCH v5 6/21 bpf: Make exception landing pads reachable in the CFG

In reply to Yonghong Song

A bpf_throw() or a bpf2bpf call inside the [begin_off, end_off) range of a cleanup record can reach that record's landing pad. Add that edge to the CFG walk, which explores the pad and makes both ends prune points, and to bpf_insn_successors(), which liveness and the SCC passes walk.

Liveness needs one more thing. When bpf_stack_slot_alive()'s is_live_before() says an outer frame's slot has no reader after the call the frame is suspended at, the landing pad can still be one -- and usually is, since only four registers survive a call and a pad reloads the rest. Ask about the pad as well when the call site names one; otherwise clean_verifier_state() poisons the slot while the callee runs and the pad is rejected for reading it.

---
 kernel/bpf/cfg.c      | 54 ++++++++++++++++++++++++++++++++++++++++---
 kernel/bpf/liveness.c | 16 +++++++++++++
 2 files changed, 67 insertions(+), 3 deletions(-)
kernel/bpf/cfg.c +51 -3
index 842c7d1eabcc..aa56d38f2e37 100644
@@ -6,6 +6,7 @@
 #include <linux/sort.h>
 
 #include "diagnostics.h"
+#include "exception.h"
 
 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
 
@@ -158,17 +159,64 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
 	return DONE_EXPLORING;
 }
 
+static int visit_cleanup_pad_edge(int t, struct bpf_verifier_env *env)
+{
+	int *insn_stack = env->cfg.insn_stack;
+	int *insn_state = env->cfg.insn_state;
+	int w;
+
+	if (!env->cleanup_info_cnt)
+		return DONE_EXPLORING;
+	w = bpf_exc_pad_of_call(env, t);
+	if (w < 0)
+		return DONE_EXPLORING;
+
+	/*
+	 * @t is a call that may branch here, and @w is the target of that
+	 * branch, so both are prune points. @w especially: every covered call
+	 * site in a region unwinds to the same pad, and without a prune point
+	 * at its head the verifier walks the pad again for each of them.
+	 */
+	mark_prune_point(env, t);
+	mark_prune_point(env, w);
+	mark_jmp_point(env, w);
+	mark_jump_target(env, w);
+
+	if (insn_state[w])
+		return DONE_EXPLORING;
+	if (env->cfg.cur_stack >= env->prog->len)
+		return -E2BIG;
+	insn_stack[env->cfg.cur_stack++] = w;
+	insn_state[w] |= DISCOVERED;
+	return KEEP_EXPLORING;
+}
+
+static int merge_visit_ret(int a, int b)
+{
+	if (a < 0)
+		return a;
+	if (b < 0)
+		return b;
+	if (a == KEEP_EXPLORING || b == KEEP_EXPLORING)
+		return KEEP_EXPLORING;
+	return DONE_EXPLORING;
+}
+
 static int visit_func_call_insn(int t, struct bpf_insn *insns,
 				struct bpf_verifier_env *env,
 				bool visit_callee)
 {
-	int ret, insn_sz;
+	int ret, insn_sz, pad_ret;
 	int w;
 
+	pad_ret = visit_cleanup_pad_edge(t, env);
+	if (pad_ret < 0)
+		return pad_ret;
+
 	insn_sz = bpf_is_ldimm64(&insns[t]) ? 2 : 1;
 	ret = push_insn(t, t + insn_sz, FALLTHROUGH, env);
 	if (ret)
-		return ret;
+		return merge_visit_ret(pad_ret, ret);
 
 	mark_prune_point(env, t + insn_sz);
 	/* when we exit from subprog, we need to record non-linear history */
@@ -180,7 +228,7 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,
 		merge_callee_effects(env, t, w);
 		ret = push_insn(t, w, BRANCH, env);
 	}
-	return ret;
+	return merge_visit_ret(pad_ret, ret);
 }
 
 struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem)
kernel/bpf/liveness.c +16 -0
index 44ecdc5b4ec2..9d44a7dafe5f 100644
@@ -8,6 +8,8 @@
 #include <linux/slab.h>
 #include <linux/sort.h>
 
+#include "exception.h"
+
 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
 
 struct per_frame_masks {
@@ -264,6 +266,13 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 	if (opcode_info->can_jump)
 		succ->items[succ->cnt++] = idx + bpf_jmp_offset(insn) + 1;
 
+	if (unlikely(env->cleanup_info_cnt)) {
+		int pad = bpf_exc_pad_of_call(env, idx);
+
+		if (pad >= 0)
+			succ->items[succ->cnt++] = pad;
+	}
+
 	return succ;
 }
 
@@ -397,6 +406,13 @@ bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp
 		alive = bpf_calls_callback(env, callsite)
 			? is_live_before(instance, callsite, rel, half_spi)
 			: is_live_before(instance, callsite + 1, rel, half_spi);
+
+		if (!alive && unlikely(env->cleanup_info_cnt)) {
+			int pad = bpf_exc_pad_of_call(env, callsite);
+
+			if (pad >= 0)
+				alive = is_live_before(instance, pad, rel, half_spi);
+		}
 		if (alive)
 			return true;
 	}
Signature · 1 line
2.53.0-Meta
#11 lore raw

Re: [PATCH bpf-next v5 06/21] bpf: Make exception landing pads reachable in the CFG

This looks like a bug fix that completes exception handling support from an earlier commit. Should this include:
Quoted 15 lines

bpf: Make exception landing pads reachable in the CFG

A bpf_throw() or a bpf2bpf call inside the [begin_off, end_off) range of a cleanup record can reach that record's landing pad. Add that edge to the CFG walk, which explores the pad and makes both ends prune points, and to bpf_insn_successors(), which liveness and the SCC passes walk.

Liveness needs one more thing. When bpf_stack_slot_alive()'s is_live_before() says an outer frame's slot has no reader after the call the frame is suspended at, the landing pad can still be one -- and usually is, since only four registers survive a call and a pad reloads the rest. Ask about the pad as well when the call site names one; otherwise clean_verifier_state() poisons the slot while the callee runs and the pad is rejected for reading it.

Quoted 1 line

This looks like a bug fix that completes exception handling support from an earlier commit. Should this include:

Fixes 6e225ddc8aa6 ("bpf: Prepare for an exception cleanup table before the CFG walk")

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35821138073

maintainer
#12 lore raw

Re: [PATCH bpf-next v5 06/21] bpf: Make exception landing pads reachable in the CFG

In reply to Yonghong Song

On Tue, 2026-09-22 at 21:59 -0700, Yonghong Song wrote:

...

Quoted 4 lines
kernel/bpf/liveness.c +0 -0
index 44ecdc5b4ec2..9d44a7dafe5f 100644

...

kernel/bpf/liveness.c bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp · line 415

Quoted 11 lines
kernel/bpf/liveness.c +7 -0
@@ -397,6 +406,13 @@ bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp
 		alive = bpf_calls_callback(env, callsite)
 			? is_live_before(instance, callsite, rel, half_spi)
 			: is_live_before(instance, callsite + 1, rel, half_spi);
+
+		if (!alive && unlikely(env->cleanup_info_cnt)) {
+			int pad = bpf_exc_pad_of_call(env, callsite);
+
+			if (pad >= 0)
+				alive = is_live_before(instance, pad, rel, half_spi);
+		}

Why is this hunk necessary?
If bpf_insn_succ correctly traverses call successors including the landing pad the information should already be accounted for by the is_live_before(instance, callsite, rel, half_spi).

Quoted 3 lines
kernel/bpf/liveness.c +0 -0
 		if (alive)
 			return true;
 	}
author reviewer
#13 lore raw

PATCH v5 7/21 bpf: Explore the landing pads no call site reaches

In reply to Yonghong Song

A cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving bpf_check_cfg() to refuse the program over code its own frontend had no way not to emit:

0: call bpf_preempt_disable
1: call bpf_preempt_enable record = { begin = 1, end = 2, pad = 4 }
2: r0 = 0
3: exit
4: r1 = pads_ran ll landing pad
6: r2 = *(u64 *)(r1 + 0)
7: r2 |= RAN_NOUNWIND_REC
8: *(u64 *)(r1 + 0) = r2
9: call bpf_unwind_resume
10: exit

The range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it. mark_call_sites() marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.

Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other: do_check() never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.

---
 kernel/bpf/cfg.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
kernel/bpf/cfg.c +17 -0
index aa56d38f2e37..1c3b4165e18c 100644
@@ -640,6 +640,7 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
 	int insn_cnt = env->prog->len;
 	int *insn_stack, *insn_state;
 	int ex_insn_beg, i, ret = 0;
+	u32 pad_idx = 0;
 
 	insn_state = env->cfg.insn_state = kvzalloc_objs(int, insn_cnt,
 							 GFP_KERNEL_ACCOUNT);
@@ -695,6 +696,22 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
 		goto walk_cfg;
 	}
 
+	/*
+	 * A landing pad no call site was marked with -- a record whose range
+	 * holds no bpf2bpf call and no bpf_throw() -- is reached by nothing.
+	 * Walk it from here, and let the dead code sweep remove it.
+	 */
+	while (pad_idx < env->cleanup_info_cnt) {
+		u32 pad = env->cleanup_info[pad_idx++].landing_pad_off;
+
+		if (insn_state[pad] != EXPLORED) {
+			insn_state[pad] = DISCOVERED;
+			insn_stack[0] = pad;
+			env->cfg.cur_stack = 1;
+			goto walk_cfg;
+		}
+	}
+
 	for (i = 0; i < insn_cnt; i++) {
 		struct bpf_insn *insn = &env->prog->insnsi[i];
 
Signature · 1 line
2.53.0-Meta
maintainer
#14 lore raw

Re: [PATCH bpf-next v5 07/21] bpf: Explore the landing pads no call site reaches

Quoted 23 lines · Yonghong Song

A cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving bpf_check_cfg() to refuse the program over code its own frontend had no way not to emit:

   0: call bpf_preempt_disable
   1: call bpf_preempt_enable      record = { begin = 1, end = 2, pad = 4 }
   2: r0 = 0
   3: exit
   4: r1 = pads_ran ll             landing pad
   6: r2 = *(u64 *)(r1 + 0)
   7: r2 |= RAN_NOUNWIND_REC
   8: *(u64 *)(r1 + 0) = r2
   9: call bpf_unwind_resume

10: exit

The range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it. mark_call_sites() marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.

Quoted 8 lines

Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other: do_check() never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.

---

Yonghong, in v4 you said that rustc does not generate dead landing pads. Why do we need this patch?

...

author reviewer
#15 lore raw

Re: [PATCH bpf-next v5 07/21] bpf: Explore the landing pads no call site reaches

Quoted 30 lines · Eduard Zingerman
Quoted 29 lines · Yonghong Song

A cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving bpf_check_cfg() to refuse the program over code its own frontend had no way not to emit:

    0: call bpf_preempt_disable
    1: call bpf_preempt_enable      record = { begin = 1, end = 2, pad = 4 }
    2: r0 = 0
    3: exit
    4: r1 = pads_ran ll             landing pad
    6: r2 = *(u64 *)(r1 + 0)
    7: r2 |= RAN_NOUNWIND_REC
    8: *(u64 *)(r1 + 0) = r2
    9: call bpf_unwind_resume

10: exit

The range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it. mark_call_sites() marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.

Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other: do_check() never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.

Quoted 4 lines
Quoted 2 lines

---

Yonghong, in v4 you said that rustc does not generate dead landing pads. Why do we need this patch?

This patch is to avoid a verification failure. Let us say, we remove this patch and run the following
./test_progs -t exceptions_cleanup
and we will get the following failure:

test_exceptions_cleanup:PASS:open 0 nsec test_light_skeleton:PASS:light open_and_load 0 nsec test_light_skeleton:PASS:run 0 nsec test_light_skeleton:PASS:retval 0 nsec test_light_skeleton:PASS:pads_ran 0 nsec
libbpf: prog 'entry_nounwind_rec': BPF program load failed: -EINVAL
libbpf: prog 'entry_nounwind_rec': -- BEGIN PROG LOAD LOG -- unreachable insn 6

Verification failed: Program Structure: Unreachable instruction

Reason:
Instruction 6 is not reachable from the program entry point.

At:
nounwind_rec_frame @ exceptions_cleanup_shapes.c:663:2 Source context:

      661 | ...
      662 | ...
  >>> 663 |         asm volatile (
          |         ^-- error: unreachable instruction
      664 | ...
      665 | ...
  Instruction context:
       4 | (b7) r0 = 0
       5 | (95) exit
  >>>  6 | (18) r1 = 0xffa0000001d94010
       8 | (79) r2 = *(u64 *)(r1 +0)

Suggestion:
Remove the unreachable instruction or add valid control flow that reaches it.

processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0 -- END PROG LOAD LOG --
libbpf: prog 'entry_nounwind_rec': failed to load: -EINVAL
libbpf: failed to load object 'exceptions_cleanup_shapes'
libbpf: failed to load BPF skeleton 'exceptions_cleanup_shapes': -EINVAL test_shapes:FAIL:shapes open_and_load unexpected error: -22 tester_init:PASS:tester_log_buf 0 nsec
process_subtest:PASS:obj_open_mem 0 nsec
process_subtest:PASS:specs_alloc 0 nsec
#128/4 exceptions_cleanup/light_skeleton:FAIL
#128 exceptions_cleanup:FAIL

This happens in cfg.c:

        for (i = 0; i < insn_cnt; i++) {
                struct bpf_insn *insn = &env->prog->insnsi[i];
                        
                if (insn_state[i] != EXPLORED) {
                        verbose(env, "unreachable insn %d\n", i);
                        bpf_diag_program_structure(
                                env, i, "unreachable instruction",
                                "Remove the unreachable instruction or add valid control flow that reaches it.",
                                "Instruction %d is not reachable from the program entry point.", i);
                        ret = -EINVAL;
                        goto err_free;
                }
                if (bpf_is_ldimm64(insn)) {
                        if (insn_state[i + 1] != 0) {
                                verbose(env, "jump into the middle of ldimm64 insn %d\n", i);
                                bpf_diag_program_structure(
                                        env, i, "jump into ldimm64 immediate",
                                        "Target the first instruction of the ldimm64 pair, or restructure the jump target.",
                                        "Control flow reaches the second half of the ldimm64 instruction pair that starts at instruction %d.",
                                        i);
                                ret = -EINVAL;
                                goto err_free;
                        }
                        i++; /* skip second half of ldimm64 */
                }
        }

This patch intends to explore *unreachable* insns to avoid verification failure.

I think due to "record = { begin = 1, end = 2, pad = 4 }", it is considered that the code at 'pad = 4' is not dead, but actually it does dead later so we have the above failure.

So I add this patch to avoid failure. But maybe we should just remove this patch, and mark this test as failure as indeed landing pad is not reachable.

WDYT?

maintainer
#16 lore raw

Re: [PATCH bpf-next v5 07/21] bpf: Explore the landing pads no call site reaches

On Wed, 2026-09-23 at 10:27 -0700, Yonghong Song wrote:

...

Quoted 4 lines

This patch is to avoid a verification failure. Let us say, we remove this patch and run the following
./test_progs -t exceptions_cleanup
and we will get the following failure:

...

Quoted 9 lines

This patch intends to explore *unreachable* insns to avoid verification failure.

I think due to "record = { begin = 1, end = 2, pad = 4 }", it is considered that the code at 'pad = 4' is not dead, but actually it does dead later so we have the above failure.

So I add this patch to avoid failure. But maybe we should just remove this patch, and mark this test as failure as indeed landing pad is not reachable.

WDYT?

But that's a self-fulfilling thing, the test you are pointing at looks as follows:

/*
* 15. A region around a call the kernel knows cannot unwind: no call site is
* marked, nothing reaches the pad, and the sweep removes it. The program is
* otherwise ordinary and has to load.
*/
static __used __naked __noinline __u64 nounwind_rec_frame(void)

 {
         ...

}

So, the test is added to test this cfg pass, the cfg pass is added to make this test pass, meanwhile compiler does not generate such code. I don't see a point, tbh.

author reviewer
#17 lore raw

Re: [PATCH bpf-next v5 07/21] bpf: Explore the landing pads no call site reaches

Quoted 28 lines · Eduard Zingerman

On Wed, 2026-09-23 at 10:27 -0700, Yonghong Song wrote:

...

Quoted 4 lines

This patch is to avoid a verification failure. Let us say, we remove this patch and run the following
./test_progs -t exceptions_cleanup
and we will get the following failure:

...

Quoted 9 lines

This patch intends to explore *unreachable* insns to avoid verification failure.

I think due to "record = { begin = 1, end = 2, pad = 4 }", it is considered that the code at 'pad = 4' is not dead, but actually it does dead later so we have the above failure.

So I add this patch to avoid failure. But maybe we should just remove this patch, and mark this test as failure as indeed landing pad is not reachable.

WDYT?

But that's a self-fulfilling thing, the test you are pointing at looks as follows:

/*
* 15. A region around a call the kernel knows cannot unwind: no call site is
* marked, nothing reaches the pad, and the sweep removes it. The program is
* otherwise ordinary and has to load.
*/
static __used __naked __noinline __u64 nounwind_rec_frame(void)

Quoted 7 lines
  {
          ...

}

So, the test is added to test this cfg pass, the cfg pass is added to make this test pass, meanwhile compiler does not generate such code. I don't see a point, tbh.

Confirmed. Will remove this patch.

#18 lore raw

Re: [PATCH bpf-next v5 07/21] bpf: Explore the landing pads no call site reaches

In reply to Eduard Zingerman
Quoted 30 lines · Eduard Zingerman
Quoted 29 lines · Yonghong Song

A cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving bpf_check_cfg() to refuse the program over code its own frontend had no way not to emit:

   0: call bpf_preempt_disable
   1: call bpf_preempt_enable      record = { begin = 1, end = 2, pad = 4 }
   2: r0 = 0
   3: exit
   4: r1 = pads_ran ll             landing pad
   6: r2 = *(u64 *)(r1 + 0)
   7: r2 |= RAN_NOUNWIND_REC
   8: *(u64 *)(r1 + 0) = r2
   9: call bpf_unwind_resume

10: exit

The range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it. mark_call_sites() marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.

Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other: do_check() never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.

Quoted 5 lines
Quoted 2 lines

---

Yonghong, in v4 you said that rustc does not generate dead landing pads. Why do we need this patch?

It's not dead. commit log is incorrect.

I bet C code that caused such asm didn't mark bpf_preempt_enable as nothrow, so compiler correctly emitted exception table.
Now commit log argues that this helper doesn't throw in reality. Well, too late, the mistake was made earlier.

I don't like the unwind approach taken by later patches in the series.

imo the following is cleaner:
- do not repurpose bpf_throw() for this new thing.
Introduce new kfunc that will do the unwind, let's name it bpf_unwind() ?
- replace all 'call bpf_unwind_resume' with NOP by the verifier (or may with bpf_exit. tbd) Technically rustc or llvm can do that too, but it's cleaner to do in the verifier.
- In bpf_unwind() walk all exception tables and replace return addresses
in corresponding frames to landing_pad_ip-s.
- just return from bpf_unwind().
restoring callee saved registers will happen automatically by corresponding frames and there is no need to search exception tables at each step. That's what typical eh unwinder does, but it's doing it due to C++ logic that is more complex that Rust. For Rust unwinds we don't need all that. The above algorithm will do.

Also please add an example where [ip_start, ip_end] range is more than just a call insn. I mentioned it couple times as the reason why 'invoke' aka 16-byte insn approach doesn't work.
I'm pretty sure I saw such IP ranges generated by rustc.

maintainer
#19 lore raw

Re: [PATCH bpf-next v5 07/21] bpf: Explore the landing pads no call site reaches

On Wed, 2026-09-23 at 21:28 +0000, Alexei Starovoitov wrote:

...

Quoted 5 lines

imo the following is cleaner:
- do not repurpose bpf_throw() for this new thing.
Introduce new kfunc that will do the unwind, let's name it bpf_unwind() ?
- replace all 'call bpf_unwind_resume' with NOP by the verifier (or may with bpf_exit. tbd) Technically rustc or llvm can do that too, but it's cleaner to do in the verifier.

Quoted 8 lines

- In bpf_unwind() walk all exception tables and replace return addresses in corresponding frames to landing_pad_ip-s.
- just return from bpf_unwind().
restoring callee saved registers will happen automatically by corresponding frames and there is no need to search exception tables at each step. That's what typical eh unwinder does, but it's doing it due to C++ logic that is more complex that Rust. For Rust unwinds we don't need all that. The above algorithm will do.

+1, makes sense.
After return address rewrite the unwind would have to jump to it's epilogue, right? Why not reusing bpf_throw() though, is it because of the exception cb mechanics?

#20 lore raw

Re: [PATCH bpf-next v5 07/21] bpf: Explore the landing pads no call site reaches

Quoted 18 lines · Eduard Zingerman

On Wed, 2026-09-23 at 21:28 +0000, Alexei Starovoitov wrote:

...

Quoted 5 lines

imo the following is cleaner:
- do not repurpose bpf_throw() for this new thing.
Introduce new kfunc that will do the unwind, let's name it bpf_unwind() ?
- replace all 'call bpf_unwind_resume' with NOP by the verifier (or may with bpf_exit. tbd) Technically rustc or llvm can do that too, but it's cleaner to do in the verifier.

Quoted 8 lines

- In bpf_unwind() walk all exception tables and replace return addresses in corresponding frames to landing_pad_ip-s.
- just return from bpf_unwind().
restoring callee saved registers will happen automatically by corresponding frames and there is no need to search exception tables at each step. That's what typical eh unwinder does, but it's doing it due to C++ logic that is more complex that Rust. For Rust unwinds we don't need all that. The above algorithm will do.

Quoted 2 lines

+1, makes sense.
After return address rewrite the unwind would have to jump to it's epilogue, right?

yes. I feel there will be no need in asm() tricks in such kfunc. unwind will rewrite its own return address, and will return through normal C and will restore callee saved (whatever it needed),
then immediate callee in bpf prog should have had a landing pad for this kfunc, since this kfunc is throwing.
so bpf_unwind() will jump to whatever is necessary to unwind in that bpf prog. And so on till the end.

Quoted 1 line

Why not reusing bpf_throw() though, is it because of the exception cb mechanics?

yes. bpf_throw() is not seen as throwing from compiler pov.
I believe all kfuncs are not throwing. (that was a bug in this patch I alluded earlier). So we need a new kfunc and mark it attr(may_throw) or whatever the attr is called, so that compiler will generate exception tables for that bpf prog.
Last time I checked there were bpf_cleanup section for rust-c in that case, but that was back in April.

There is also a case of 'catching' abort in rust. iirc it only stops unwind and rust-c generates something else instead of 'call bpf_unwind_resume'. Memory is vague. I think that frame will continue as normal after processing landing pad drop()s.

#21 lore raw

Re: [PATCH bpf-next v5 07/21] bpf: Explore the landing pads no call site reaches

Quoted 30 lines · Alexei Starovoitov
Quoted 18 lines · Eduard Zingerman

On Wed, 2026-09-23 at 21:28 +0000, Alexei Starovoitov wrote:

...

Quoted 5 lines

imo the following is cleaner:
- do not repurpose bpf_throw() for this new thing.
Introduce new kfunc that will do the unwind, let's name it bpf_unwind() ?
- replace all 'call bpf_unwind_resume' with NOP by the verifier (or may with bpf_exit. tbd) Technically rustc or llvm can do that too, but it's cleaner to do in the verifier.

Quoted 8 lines

- In bpf_unwind() walk all exception tables and replace return addresses in corresponding frames to landing_pad_ip-s.
- just return from bpf_unwind().
restoring callee saved registers will happen automatically by corresponding frames and there is no need to search exception tables at each step. That's what typical eh unwinder does, but it's doing it due to C++ logic that is more complex that Rust. For Rust unwinds we don't need all that. The above algorithm will do.

Quoted 2 lines

+1, makes sense.
After return address rewrite the unwind would have to jump to it's epilogue, right?

yes. I feel there will be no need in asm() tricks in such kfunc. unwind will rewrite its own return address, and will return through normal C and will restore callee saved (whatever it needed),
then immediate callee in bpf prog should have had a landing pad for this kfunc, since this kfunc is throwing.
so bpf_unwind() will jump to whatever is necessary to unwind in that bpf prog. And so on till the end.

Quoted 8 lines
Quoted 1 line

Why not reusing bpf_throw() though, is it because of the exception cb mechanics?

yes. bpf_throw() is not seen as throwing from compiler pov.
I believe all kfuncs are not throwing. (that was a bug in this patch I alluded earlier). So we need a new kfunc and mark it attr(may_throw) or whatever the attr is called, so that compiler will generate exception tables for that bpf prog.
Last time I checked there were bpf_cleanup section for rust-c in that case, but that was back in April.

Hm, isn't this dependent on how you annotate the kfunc on the Rust side? Like, you could make the same annotation on bpf_throw() that you would for the new kfunc, no? I got confused by this part.

Quoted 5 lines

There is also a case of 'catching' abort in rust. iirc it only stops unwind and rust-c generates something else instead of 'call bpf_unwind_resume'. Memory is vague. I think that frame will continue as normal after processing landing pad drop()s.

#22 lore raw

Re: [PATCH bpf-next v5 07/21] bpf: Explore the landing pads no call site reaches

Quoted 40 lines · Kumar Kartikeya Dwivedi
Quoted 30 lines · Alexei Starovoitov
Quoted 18 lines · Eduard Zingerman

On Wed, 2026-09-23 at 21:28 +0000, Alexei Starovoitov wrote:

...

Quoted 5 lines

imo the following is cleaner:
- do not repurpose bpf_throw() for this new thing.
Introduce new kfunc that will do the unwind, let's name it bpf_unwind() ?
- replace all 'call bpf_unwind_resume' with NOP by the verifier (or may with bpf_exit. tbd) Technically rustc or llvm can do that too, but it's cleaner to do in the verifier.

Quoted 8 lines

- In bpf_unwind() walk all exception tables and replace return addresses in corresponding frames to landing_pad_ip-s.
- just return from bpf_unwind().
restoring callee saved registers will happen automatically by corresponding frames and there is no need to search exception tables at each step. That's what typical eh unwinder does, but it's doing it due to C++ logic that is more complex that Rust. For Rust unwinds we don't need all that. The above algorithm will do.

Quoted 2 lines

+1, makes sense.
After return address rewrite the unwind would have to jump to it's epilogue, right?

yes. I feel there will be no need in asm() tricks in such kfunc. unwind will rewrite its own return address, and will return through normal C and will restore callee saved (whatever it needed),
then immediate callee in bpf prog should have had a landing pad for this kfunc, since this kfunc is throwing.
so bpf_unwind() will jump to whatever is necessary to unwind in that bpf prog. And so on till the end.

Quoted 8 lines
Quoted 1 line

Why not reusing bpf_throw() though, is it because of the exception cb mechanics?

yes. bpf_throw() is not seen as throwing from compiler pov.
I believe all kfuncs are not throwing. (that was a bug in this patch I alluded earlier). So we need a new kfunc and mark it attr(may_throw) or whatever the attr is called, so that compiler will generate exception tables for that bpf prog.
Last time I checked there were bpf_cleanup section for rust-c in that case, but that was back in April.

Quoted 3 lines

Hm, isn't this dependent on how you annotate the kfunc on the Rust side? Like, you could make the same annotation on bpf_throw() that you would for the new kfunc, no? I got confused by this part.

bpf_throw() has a specific semantic already.
Like it suppose to call that bpf prog callback once it unwinds everything.
This new kfunc is different. It will initiate the unwind the way rust wants it to and may stop in the middle.
See below point about 'catching'.

Quoted 4 lines
Quoted 4 lines

There is also a case of 'catching' abort in rust. iirc it only stops unwind and rust-c generates something else instead of 'call bpf_unwind_resume'. Memory is vague. I think that frame will continue as normal after processing landing pad drop()s.

#23 lore raw

Re: [PATCH bpf-next v5 07/21] bpf: Explore the landing pads no call site reaches

Quoted 45 lines · Alexei Starovoitov
Quoted 40 lines · Kumar Kartikeya Dwivedi
Quoted 30 lines · Alexei Starovoitov
Quoted 18 lines · Eduard Zingerman

On Wed, 2026-09-23 at 21:28 +0000, Alexei Starovoitov wrote:

...

Quoted 5 lines

imo the following is cleaner:
- do not repurpose bpf_throw() for this new thing.
Introduce new kfunc that will do the unwind, let's name it bpf_unwind() ?
- replace all 'call bpf_unwind_resume' with NOP by the verifier (or may with bpf_exit. tbd) Technically rustc or llvm can do that too, but it's cleaner to do in the verifier.

Quoted 8 lines

- In bpf_unwind() walk all exception tables and replace return addresses in corresponding frames to landing_pad_ip-s.
- just return from bpf_unwind().
restoring callee saved registers will happen automatically by corresponding frames and there is no need to search exception tables at each step. That's what typical eh unwinder does, but it's doing it due to C++ logic that is more complex that Rust. For Rust unwinds we don't need all that. The above algorithm will do.

Quoted 2 lines

+1, makes sense.
After return address rewrite the unwind would have to jump to it's epilogue, right?

yes. I feel there will be no need in asm() tricks in such kfunc. unwind will rewrite its own return address, and will return through normal C and will restore callee saved (whatever it needed),
then immediate callee in bpf prog should have had a landing pad for this kfunc, since this kfunc is throwing.
so bpf_unwind() will jump to whatever is necessary to unwind in that bpf prog. And so on till the end.

Quoted 8 lines
Quoted 1 line

Why not reusing bpf_throw() though, is it because of the exception cb mechanics?

yes. bpf_throw() is not seen as throwing from compiler pov.
I believe all kfuncs are not throwing. (that was a bug in this patch I alluded earlier). So we need a new kfunc and mark it attr(may_throw) or whatever the attr is called, so that compiler will generate exception tables for that bpf prog.
Last time I checked there were bpf_cleanup section for rust-c in that case, but that was back in April.

Quoted 3 lines

Hm, isn't this dependent on how you annotate the kfunc on the Rust side? Like, you could make the same annotation on bpf_throw() that you would for the new kfunc, no? I got confused by this part.

Quoted 7 lines

bpf_throw() has a specific semantic already.
Like it suppose to call that bpf prog callback once it unwinds everything.
This new kfunc is different. It will initiate the unwind the way rust wants it to and may stop in the middle.
See below point about 'catching'.

Right, ok, that makes sense. I do think it is cleaner to have landing pad based cleanup initiated using a new entrypoint.

I just wasn't sure how much point there was in keeping the old unwinding scheme around, given without being able to release any resources it's not very useful in practice, so we could just drop the exception cb stuff completely and just change its behavior and make its working conditional on landing pads being present.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

There is also a case of 'catching' abort in rust. iirc it only stops unwind and rust-c generates something else instead of 'call bpf_unwind_resume'. Memory is vague. I think that frame will continue as normal after processing landing pad drop()s.

author reviewer
#24 lore raw

Re: [PATCH bpf-next v5 07/21] bpf: Explore the landing pads no call site reaches

In reply to Alexei Starovoitov
Quoted 58 lines · Alexei Starovoitov
Quoted 30 lines · Eduard Zingerman
Quoted 29 lines · Yonghong Song

A cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving bpf_check_cfg() to refuse the program over code its own frontend had no way not to emit:

    0: call bpf_preempt_disable
    1: call bpf_preempt_enable      record = { begin = 1, end = 2, pad = 4 }
    2: r0 = 0
    3: exit
    4: r1 = pads_ran ll             landing pad
    6: r2 = *(u64 *)(r1 + 0)
    7: r2 |= RAN_NOUNWIND_REC
    8: *(u64 *)(r1 + 0) = r2
    9: call bpf_unwind_resume

10: exit

The range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it. mark_call_sites() marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.

Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other: do_check() never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.

Quoted 4 lines
Quoted 2 lines

---

Yonghong, in v4 you said that rustc does not generate dead landing pads. Why do we need this patch?

It's not dead. commit log is incorrect.

I bet C code that caused such asm didn't mark bpf_preempt_enable as nothrow, so compiler correctly emitted exception table.
Now commit log argues that this helper doesn't throw in reality. Well, too late, the mistake was made earlier.

I don't like the unwind approach taken by later patches in the series.

imo the following is cleaner:
- do not repurpose bpf_throw() for this new thing.
Introduce new kfunc that will do the unwind, let's name it bpf_unwind() ?
- replace all 'call bpf_unwind_resume' with NOP by the verifier (or may with bpf_exit. tbd) Technically rustc or llvm can do that too, but it's cleaner to do in the verifier.
- In bpf_unwind() walk all exception tables and replace return addresses
in corresponding frames to landing_pad_ip-s.
- just return from bpf_unwind().
restoring callee saved registers will happen automatically by corresponding frames and there is no need to search exception tables at each step. That's what typical eh unwinder does, but it's doing it due to C++ logic that is more complex that Rust. For Rust unwinds we don't need all that. The above algorithm will do.

Quoted 5 lines

Also please add an example where [ip_start, ip_end] range is more than just a call insn. I mentioned it couple times as the reason why 'invoke' aka 16-byte insn approach doesn't work.
I'm pretty sure I saw such IP ranges generated by rustc.

Thanks for explaining what should be implemented in a simpler way. I will try to implement this as above steps.

author reviewer
#25 lore raw

PATCH v5 8/21 bpf: Walk the exception unwind in the verifier

In reply to Yonghong Song

bpf_throw() is about to stop discarding the frames it unwinds and start running their landing pads instead. Have the verifier walk the same thing, step for step:

at a throw, with the frame chain in env->cur_state->frame[]:
1. does a record cover the call site control is at?
2. yes: run that record's landing pad in this frame, and on reaching its resume, pop the frame and go back to 1
3. no: pop the frame and go back to 1
4. frame 0 asks with nothing left to pop: deliver

Doing it this way is what keeps the resource rules unchanged. Whatever a pad releases is released in the verifier state too, so by the time the walk reaches the boundary the state says exactly what the program will really hold there -- and check_resource_leak(), which used to fire the moment a throw was seen, simply moves to the end of the walk. A frame that no record covers contributes nothing, so anything it held is still held when the walk ends, and that is what gets reported.

A pad starts in the state the walker hands it: its own frame's stack and callee-saved registers, which the walker puts back, and nothing in the caller-saved ones. r0 is an unknown scalar: LLVM names it as both the exception pointer and the exception selector register, so a pad reads it before anything else, and the dispatchers leave a defined value there rather than whatever the kernel had.

Two things about that walk have to be said out loud, because the instruction stream does not say them.

A resume belongs to the frame whose landing pad the walker called:

- a JIT lowers it as the way back out of a pad -- a bare return on x86-64, a branch to the saved address on arm64 -- so it reaches the walker only from there
- an exception in flight does not say which frame that is: a pad may call a subprogram whose own pad reaches a resume by ordinary control flow
- so the state records the frame the unwind entered a pad in, states_equal() keeps two such states apart, a resume in any other frame is refused, and bpf_insn_successors() reports none for it

The edge from a throw to a pad crosses frames with no instruction in between, and mark_chain_precision() walks that history backwards:

- left alone it stays in the pad's frame while reading the callee's
instructions: a request for the pad frame's r6 is cleared by the callee's own write to r6, or reaches a call still set and trips "static subprog unexpected regs"
- so the history entry for a pad records how many frames the unwind popped, and the backtrack enters that many, the way it enters one for BPF_EXIT
- the pad's r0 is whatever the walker left rather than anything an instruction wrote, so a request for it ends at the edge
- a throwing global subprogram gets no frame of its own, so its exception arrives at the pad rather than the insn after the call, which backtrack_insn() has to allow for

This is also where bpf_unwind_resume() becomes callable: the kfunc is registered here, beside the rule that refuses it outside a pad. Earlier, nothing refused the call and a JIT rewrites a resume only in a pad, so a program could reach the kernel body -- a WARN_ONCE whose only job is to say it should never have been reached.

---
 include/linux/bpf_verifier.h |   8 ++-
 kernel/bpf/backtrack.c       |  41 +++++++++--
 kernel/bpf/exception.c       |   1 +
 kernel/bpf/helpers.c         |   1 +
 kernel/bpf/liveness.c        |   3 +
 kernel/bpf/states.c          |   6 ++
 kernel/bpf/verifier.c        | 133 +++++++++++++++++++++++++++--------
 7 files changed, 158 insertions(+), 35 deletions(-)
include/linux/bpf_verifier.h +7 -1
index e12ec91b8150..3146f707e030 100644
@@ -429,7 +429,8 @@ struct bpf_jmp_history_entry {
 	u32 prev_idx : 20;
 	/* special INSN_F_xxx flags */
 	u32 flags : 4;
-	u32 : 8;
+	u32 unwind_frames : 4;	/* frames popped to reach this landing pad */
+	u32 : 4;
 	/*
 	 * additional registers that need precision tracking when this
 	 * jump is backtracked, vector of five 11-bit records
@@ -510,6 +511,9 @@ struct bpf_verifier_state {
 	bool speculative;
 	bool in_sleepable;
 
+	bool unwinding; /* an exception is in flight */
+	u8 unwind_frameno; /* the frame whose landing pad is running */
+
 	/* first and last insn idx of this verifier state */
 	u32 first_insn_idx;
 	u32 last_insn_idx;
@@ -991,6 +995,8 @@ struct bpf_verifier_env {
 	} cfg;
 	struct backtrack_state bt;
 	struct bpf_jmp_history_entry *cur_hist_ent;
+	u8 cur_unwind_frames; /* frames popped to reach this insn, if a pad */
+	u8 unwind_frames; /* frames popped, staged for the next insn */
 	/* Per-callsite copy of parent's converged at_stack_in for cross-frame fills. */
 	struct arg_track **callsite_at_stack;
 	u32 pass_cnt; /* number of times do_check() was called */
kernel/bpf/backtrack.c +37 -4

88 lines not on this page · raw message

kernel/bpf/exception.c +1 -0
index 244ea1b5417c..3270e7ae2e7c 100644
@@ -5,6 +5,7 @@
 #include <linux/btf.h>
 #include <linux/btf_ids.h>
 #include <linux/filter.h>
+#include <linux/slab.h>
 #include "exception.h"
 
 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
kernel/bpf/helpers.c +1 -0
index 249418e60ee4..5eadab4dfa3f 100644
@@ -5094,6 +5094,7 @@ BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_throw)
+BTF_ID_FLAGS(func, bpf_unwind_resume)
 #ifdef CONFIG_BPF_EVENTS
 BTF_ID_FLAGS(func, bpf_send_signal_task)
 #endif
kernel/bpf/liveness.c +3 -0
index 9d44a7dafe5f..00ac8917a716 100644
@@ -258,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 	succ = env->succ;
 	succ->cnt = 0;
 
+	if (unlikely(bpf_is_unwind_resume_kfunc(insn)))
+		return succ;
+
 	opcode_info = &opcode_info_tbl[BPF_CLASS(insn->code) | BPF_OP(insn->code)];
 	insn_sz = bpf_is_ldimm64(insn) ? 2 : 1;
 	if (opcode_info->can_fallthrough)
kernel/bpf/states.c +6 -0
index 66fb11b6c6a7..b101baa43171 100644
@@ -996,6 +996,12 @@ static bool states_equal(struct bpf_verifier_env *env,
 	if (old->in_sleepable != cur->in_sleepable)
 		return false;
 
+	if (old->unwinding != cur->unwinding)
+		return false;
+
+	if (old->unwinding && old->unwind_frameno != cur->unwind_frameno)
+		return false;
+
 	if (!refsafe(old, cur, &env->idmap_scratch))
 		return false;
 
kernel/bpf/verifier.c +103 -30

209 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
maintainer
#26 lore raw

Re: [PATCH bpf-next v5 08/21] bpf: Walk the exception unwind in the verifier

On Tue, 2026-09-22 at 21:59 -0700, Yonghong Song wrote:

...

Quoted 25 lines
include/linux/bpf_verifier.h +5 -1
index e12ec91b8150..3146f707e030 100644
@@ -429,7 +429,8 @@ struct bpf_jmp_history_entry {
 	u32 prev_idx : 20;
 	/* special INSN_F_xxx flags */
 	u32 flags : 4;
-	u32 : 8;
+	u32 unwind_frames : 4;	/* frames popped to reach this landing pad */
+	u32 : 4;
 	/*
 	 * additional registers that need precision tracking when this
 	 * jump is backtracked, vector of five 11-bit records
@@ -510,6 +511,9 @@ struct bpf_verifier_state {
 	bool speculative;
 	bool in_sleepable;
 
+	bool unwinding; /* an exception is in flight */
+	u8 unwind_frameno; /* the frame whose landing pad is running */
+
 	/* first and last insn idx of this verifier state */
 	u32 first_insn_idx;
 	u32 last_insn_idx;

include/linux/bpf_verifier.h struct bpf_verifier_env { · line 999

Quoted 5 lines
include/linux/bpf_verifier.h +2 -0
@@ -991,6 +995,8 @@ struct bpf_verifier_env {
 	} cfg;
 	struct backtrack_state bt;
 	struct bpf_jmp_history_entry *cur_hist_ent;
+	u8 cur_unwind_frames; /* frames popped to reach this insn, if a pad */
+	u8 unwind_frames; /* frames popped, staged for the next insn */

Similarly to v4, stashing these variables into env in one place and reading them in another in order to push to jump history is an obfuscation step. Why don't you want to push to the jump history right away? Like in [1].

So far the only field in env that changes with current verifier state is insn_idx. I don't see why this feature requires adding two more such fields.

[1] https://github.com/kernel-patches/bpf/commit/fbc89e90cacd207b4271da25ed7a380f6b56125e#diff-edbb57adf10d1ce1fbb830a34fa92712fd01db1fbd9b6f2504001eb7bcc7b9d0R5604-R18577
(with corresponding changes in backtrack.c in the same commit)

...

author reviewer
#27 lore raw

Re: [PATCH bpf-next v5 08/21] bpf: Walk the exception unwind in the verifier

Quoted 39 lines · Eduard Zingerman

On Tue, 2026-09-22 at 21:59 -0700, Yonghong Song wrote:

...

Quoted 25 lines
include/linux/bpf_verifier.h +5 -1
index e12ec91b8150..3146f707e030 100644
@@ -429,7 +429,8 @@ struct bpf_jmp_history_entry {
  	u32 prev_idx : 20;
  	/* special INSN_F_xxx flags */
  	u32 flags : 4;
-	u32 : 8;
+	u32 unwind_frames : 4;	/* frames popped to reach this landing pad */
+	u32 : 4;
  	/*
  	 * additional registers that need precision tracking when this
  	 * jump is backtracked, vector of five 11-bit records
@@ -510,6 +511,9 @@ struct bpf_verifier_state {
  	bool speculative;
  	bool in_sleepable;
 
+	bool unwinding; /* an exception is in flight */
+	u8 unwind_frameno; /* the frame whose landing pad is running */
+
  	/* first and last insn idx of this verifier state */
  	u32 first_insn_idx;
  	u32 last_insn_idx;

include/linux/bpf_verifier.h struct bpf_verifier_env { · line 999

Quoted 5 lines
include/linux/bpf_verifier.h +2 -0
@@ -991,6 +995,8 @@ struct bpf_verifier_env {
  	} cfg;
  	struct backtrack_state bt;
  	struct bpf_jmp_history_entry *cur_hist_ent;
+	u8 cur_unwind_frames; /* frames popped to reach this insn, if a pad */
+	u8 unwind_frames; /* frames popped, staged for the next insn */

Similarly to v4, stashing these variables into env in one place and reading them in another in order to push to jump history is an obfuscation step. Why don't you want to push to the jump history right away? Like in [1].

Quoted 5 lines

So far the only field in env that changes with current verifier state is insn_idx. I don't see why this feature requires adding two more such fields.

[1] https://github.com/kernel-patches/bpf/commit/fbc89e90cacd207b4271da25ed7a380f6b56125e#diff-edbb57adf10d1ce1fbb830a34fa92712fd01db1fbd9b6f2504001eb7bcc7b9d0R5604-R18577
(with corresponding changes in backtrack.c in the same commit)

Okay, I looked at your backtrack.c and verifier.c related code. I will use jump history then.

Quoted 2 lines

...

author reviewer
#28 lore raw

PATCH v5 9/21 bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch

In reply to Yonghong Song

The verifier now walks the unwind, so at every instruction it knows whether an exception is in flight and, if it is, which frame's landing pad it entered. do_check() asks bpf_exc_check_insn() about each instruction of a program that carries a table, and it refuses:

- bpf_throw() in any frame while unwinding: a second unwind over frames the first is still discarding. A static callee is walked from the pad, so its own throw trips this too; a global one is not walked, which is what subprog_info.might_throw is for
- in the frame whose landing pad the unwind entered, where the code runs on the walker's stack: a plain exit, which is what a catch pad ends in, a tail call, an indirect jump, an outgoing on-stack call argument, and a BPF_LD_[ABS|IND], whose failure path leaves the frame through the epilogue
- an instruction reached both inside and outside a pad: the arm64 JIT addresses the frame through a register only a pad entry computes. This also turns away a jump into the middle of a pad

bpf_exc_check_callback() refuses a subprogram that may unwind as a helper callback, whose helper frame would end the walk before it found a boundary.

---
 include/linux/bpf_verifier.h |   2 +
 kernel/bpf/exception.c       | 101 +++++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h       |   3 ++
 kernel/bpf/fixups.c          |   1 +
 kernel/bpf/verifier.c        |  10 ++++
 5 files changed, 117 insertions(+)
include/linux/bpf_verifier.h +2 -0
index 3146f707e030..736304d912af 100644
@@ -698,6 +698,8 @@ struct bpf_insn_aux_data {
 	u64 arg_prog:4;
 	u64 throw_call:1; /* call to bpf_throw() */
 	u64 resume_call:1; /* call to bpf_unwind_resume() */
+	u64 in_cleanup_pad:1; /* runs with an exception in flight, in the pad's frame */
+	u64 outside_cleanup_pad:1; /* ... and the other way round */
 
 	/* below flags are initialized once */
 	u64 jmp_point:1;
kernel/bpf/exception.c +101 -0

113 lines not on this page · raw message

kernel/bpf/exception.h +3 -0
index b96b2c429e22..0872a3f70583 100644
@@ -5,9 +5,12 @@
 
 #include <linux/types.h>
 
+struct bpf_insn;
 struct bpf_verifier_env;
 
 int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env);
+int bpf_exc_check_insn(struct bpf_verifier_env *env, struct bpf_insn *insn);
+int bpf_exc_check_callback(struct bpf_verifier_env *env, int subprog);
 int bpf_exc_pad_of_call(struct bpf_verifier_env *env, u32 idx);
 
 #endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +1 -0
index fed5b66ed3b7..53a737c88dcc 100644
@@ -252,6 +252,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 		/* Expand insni[off]'s seen count to the patched range. */
 		data[i].seen = old_seen;
 		data[i].zext_dst = bpf_insn_def32(new_prog, insn + i) >= 0;
+		data[i].in_cleanup_pad = data[off + cnt - 1].in_cleanup_pad;
 		if (!memcmp(insn + i, original_insn, sizeof(struct bpf_insn))) {
 			data[i].non_stack_access =
 				data[off + cnt - 1].non_stack_access;
kernel/bpf/verifier.c +10 -0
index 53edce3d64ea..7b74dd9b9f48 100644
@@ -10549,6 +10549,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
 	 * callbacks
 	 */
 	env->subprog_info[subprog].is_cb = true;
+	err = bpf_exc_check_callback(env, subprog);
+	if (err)
+		return err;
+
 	if (bpf_pseudo_kfunc_call(insn) &&
 	    !is_callback_calling_kfunc(insn->imm)) {
 		verifier_bug(env, "kfunc %s#%d not marked as callback-calling",
@@ -19041,6 +19045,12 @@ static int do_check(struct bpf_verifier_env *env)
 				return err;
 		}
 
+		if (unlikely(env->cleanup_info_cnt)) {
+			err = bpf_exc_check_insn(env, insn);
+			if (err)
+				return err;
+		}
+
 		sanitize_mark_insn_seen(env);
 		prev_insn_idx = env->insn_idx;
 
Signature · 1 line
2.53.0-Meta
author reviewer
#29 lore raw

PATCH v5 10/21 bpf: Refuse a private stack for a program with an exception cleanup table

In reply to Yonghong Song

A private stack is hard to support together with an exception cleanup table on x86-64: a landing pad runs on the bpf_throw() walker's stack with its frame's registers put back from a spill area, and carrying the private stack's frame pointer through there as well would cost a register that architecture cannot spare.

Force NO_PRIV_STACK in check_max_stack_depth(), where the choice is made, and do it on every architecture for now rather than just x86-64. The program still loads, but more than an optimization can be lost:
check_max_stack_depth_subprog() checks a PRIV_STACK_ADAPTIVE subprogram against MAX_BPF_STACK on its own, and adds a NO_PRIV_STACK one to the depth its callers have to fit in -- so a bpf2bpf chain deep enough to need the private stack is now refused with "combined stack size of %d calls is %d. Too large".

---
 kernel/bpf/verifier.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
kernel/bpf/verifier.c +11 -0
index 7b74dd9b9f48..6462e9f2ffdc 100644
@@ -5602,6 +5602,17 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
 		}
 	}
 
+	/*
+	 * A pad rebuilds its frame from a spill area, and on x86-64 a private
+	 * stack's frame pointer lives in r9, which no spill area holds.
+	 * Refused on every arch rather than just that one. The subprograms
+	 * below are then checked against MAX_BPF_STACK together rather than
+	 * one at a time, so this can turn a program that would have loaded
+	 * with a private stack into one that is too deep.
+	 */
+	if (env->cleanup_info_cnt)
+		priv_stack_mode = NO_PRIV_STACK;
+
 	if (priv_stack_mode == PRIV_STACK_UNKNOWN)
 		priv_stack_mode = bpf_enable_priv_stack(env->prog);
 
Signature · 1 line
2.53.0-Meta
author reviewer
#30 lore raw

PATCH v5 11/21 bpf: Dispatch exception cleanup pads from bpf_throw()

In reply to Yonghong Song

Stop discarding the frames an exception unwinds through. bpf_throw() already walks the BPF call stack with arch_bpf_stack_walk() to find the exception boundary; have it look each frame's return address up in that (sub)program's cleanup table on the way and run the landing pad a matching record names.

A pad is run as a subroutine of the walker, not jumped to. It executes with the unwinding frame's frame pointer and BPF callee-saved registers, so everything it reads is that frame's, but on the current stack far below it, so nothing it calls can disturb the frame it is cleaning up after. It ends in what the JIT emits for its bpf_unwind_resume(), which hands control back to the walker rather than to the frame's caller. That is exactly the procedure the LLVM commit emitting the section describes.

Restoring the frame's r6-r9 is what makes this work, and it is only possible because the callee about to be discarded spilled them in its own prologue. bpf_exc_force_spill() tells a JIT to make that spill unconditional and of a known shape for every subprogram of a program carrying a cleanup table, and aux->exc->spill_off records where it starts, so the walker needs no per-frame metadata. The frame that called bpf_throw() has no callee to have spilled anything and never runs its own epilogue, so the JIT spills that frame's registers at the throw site instead, in the area aux->exc->throw_spill_off names.

The main program needs the table handed to it rather than built for it. jit_subprogs() compiles it as func[0], but the ksym covering that image is the one bpf_prog_load() registers for the outer bpf_prog, and that is what the walker finds -- so without the handover a landing pad in the main program's own frame is never dispatched, silently, since the exception still reaches the boundary and the cookie still comes back.

Which instructions a JIT has to lower its own way -- a bpf_throw(), a resume, the head and the body of a pad -- it reads from insn_aux_data, which bpf_int_jit_compile() already has and which instruction patching keeps in step. A subprogram's instructions are indexed from its aux->subprog_start.

---
 include/linux/bpf.h          |  88 ++++++++++++++++++++++
 include/linux/bpf_verifier.h |   1 +
 include/linux/filter.h       |   1 +
 kernel/bpf/core.c            |  30 +++++++-
 kernel/bpf/exception.c       | 141 +++++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h       |   7 ++
 kernel/bpf/fixups.c          |  65 ++++++++++++++++
 kernel/bpf/helpers.c         |  34 +++++++++
 8 files changed, 364 insertions(+), 3 deletions(-)
include/linux/bpf.h +88 -0
index e7c5e203eddd..c11dc224973e 100644
@@ -1787,6 +1787,93 @@ enum bpf_sig_keyring {
 	BPF_SIG_KEYRING_BPF,
 };
 
+/*
+ * One cleanup region of a JITed (sub)program: @pad is the landing pad to run
+ * for a return address in (begin, end], the native code of its call sites.
+ */
+struct bpf_cleanup_range {
+	u64 begin;
+	u64 end;
+	u64 pad;
+};
+
+struct bpf_exception_info {
+	struct bpf_cleanup_info *info;
+	struct bpf_cleanup_range *ranges;
+	u32 nr_info;
+	u32 nr_ranges;
+	/* Set if any instruction of this subprogram calls bpf_throw(). */
+	bool has_throw;
+	/* Offset from a frame's FP to the caller's spilled r6-r9. */
+	s32 spill_off;
+	/* Likewise, to the registers a frame spills before calling bpf_throw(). */
+	s32 throw_spill_off;
+};
+
+#ifdef CONFIG_BPF_SYSCALL
+bool bpf_exc_force_spill(const struct bpf_prog *prog);
+bool bpf_exc_needs_throw_spill(const struct bpf_prog *prog);
+bool bpf_exc_insn_is_pad(const struct bpf_verifier_env *env,
+			 const struct bpf_prog *prog, u32 idx);
+bool bpf_exc_insn_in_pad(const struct bpf_verifier_env *env,
+			 const struct bpf_prog *prog, u32 idx);
+bool bpf_exc_insn_is_throw(const struct bpf_verifier_env *env,
+			   const struct bpf_prog *prog, u32 idx);
+bool bpf_exc_insn_is_resume(const struct bpf_verifier_env *env,
+			    const struct bpf_prog *prog, u32 idx);
+int bpf_exc_attach_main_prog(struct bpf_verifier_env *env, struct bpf_prog *prog);
+void bpf_exc_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image);
+void bpf_exc_free_info(struct bpf_prog_aux *aux);
+#else
+static inline bool bpf_exc_force_spill(const struct bpf_prog *prog)
+{
+	return false;
+}
+
+static inline bool bpf_exc_needs_throw_spill(const struct bpf_prog *prog)
+{
+	return false;
+}
+
+static inline bool bpf_exc_insn_is_pad(const struct bpf_verifier_env *env,
+				       const struct bpf_prog *prog, u32 idx)
+{
+	return false;
+}
+
+static inline bool bpf_exc_insn_in_pad(const struct bpf_verifier_env *env,
+				       const struct bpf_prog *prog, u32 idx)
+{
+	return false;
+}
+
+static inline bool bpf_exc_insn_is_throw(const struct bpf_verifier_env *env,
+					 const struct bpf_prog *prog, u32 idx)
+{
+	return false;
+}
+
+static inline bool bpf_exc_insn_is_resume(const struct bpf_verifier_env *env,
+					  const struct bpf_prog *prog, u32 idx)
+{
+	return false;
+}
+
+static inline int bpf_exc_attach_main_prog(struct bpf_verifier_env *env,
+					   struct bpf_prog *prog)
+{
+	return 0;
+}
+
+static inline void bpf_exc_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image)
+{
+}
+
+static inline void bpf_exc_free_info(struct bpf_prog_aux *aux)
+{
+}
+#endif
+
 struct bpf_prog_aux {
 	atomic64_t refcnt;
 	u32 used_map_cnt;
@@ -1867,6 +1954,7 @@ struct bpf_prog_aux {
 	char name[BPF_OBJ_NAME_LEN];
 	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
 	u16 stack_arg_sp_adjust;
+	struct bpf_exception_info *exc;
 #ifdef CONFIG_SECURITY
 	void *security;
 #endif
include/linux/bpf_verifier.h +1 -0
index 736304d912af..500a1e253579 100644
@@ -700,6 +700,7 @@ struct bpf_insn_aux_data {
 	u64 resume_call:1; /* call to bpf_unwind_resume() */
 	u64 in_cleanup_pad:1; /* runs with an exception in flight, in the pad's frame */
 	u64 outside_cleanup_pad:1; /* ... and the other way round */
+	u64 cleanup_pad_head:1; /* first insn of a landing pad */
 
 	/* below flags are initialized once */
 	u64 jmp_point:1;
include/linux/filter.h +1 -0
index 2582a7606e46..b0c495e94f74 100644
@@ -1243,6 +1243,7 @@ bool bpf_jit_supports_arena_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
 bool bpf_jit_supports_cleanup_pads(void);
+void arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base);
 bool bpf_jit_supports_ptr_xchg(void);
 bool bpf_jit_supports_arena(void);
 bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
kernel/bpf/core.c +27 -3
index a379cd1ec4c6..16e5210a56d5 100644
@@ -292,6 +292,7 @@ void __bpf_prog_free(struct bpf_prog *fp)
 		mutex_destroy(&fp->aux->dst_mutex);
 		mutex_destroy(&fp->aux->st_ops_assoc_mutex);
 		kfree(fp->aux->poke_tab);
+		bpf_exc_free_info(fp->aux);
 		kfree(fp->aux);
 	}
 	free_percpu(fp->stats);
@@ -2625,13 +2626,21 @@ static bool bpf_prog_select_interpreter(struct bpf_prog *fp)
 	return select_interpreter;
 }
 
-static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
+static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog,
+					     int *err)
 {
 #ifdef CONFIG_BPF_JIT
 	struct bpf_prog *orig_prog;
+	int ret;
 
-	if (!bpf_prog_need_blind(prog))
+	if (!bpf_prog_need_blind(prog)) {
+		ret = bpf_exc_attach_main_prog(env, prog);
+		if (ret) {
+			*err = ret;
+			return prog;
+		}
 		return bpf_int_jit_compile(env, prog);
+	}
 
 	orig_prog = prog;
 	prog = bpf_jit_blind_constants(env, prog);
@@ -2642,6 +2651,13 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
 	if (IS_ERR(prog))
 		goto out_restore;
 
+	ret = bpf_exc_attach_main_prog(env, prog);
+	if (ret) {
+		*err = ret;
+		bpf_jit_prog_release_other(orig_prog, prog);
+		goto out_restore;
+	}
+
 	prog = bpf_int_jit_compile(env, prog);
 	if (prog->jited) {
 		bpf_jit_prog_release_other(prog, orig_prog);
@@ -2681,8 +2697,10 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
 		if (*err)
 			return fp;
 
-		fp = bpf_prog_jit_compile(env, fp);
+		fp = bpf_prog_jit_compile(env, fp, err);
 		bpf_prog_jit_attempt_done(fp);
+		if (*err)
+			return fp;
 		if (!fp->jited && jit_needed) {
 			*err = -ENOTSUPP;
 			return fp;
@@ -3480,6 +3498,12 @@ bool __weak bpf_jit_supports_cleanup_pads(void)
 	return false;
 }
 
+/* Call @pad with the frame pointer @frame_fp and r6-r9 spilled at @spill_base. */
+void __weak arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base)
+{
+	WARN_ON_ONCE(1);
+}
+
 bool __weak bpf_jit_supports_timed_may_goto(void)
 {
 	return false;
kernel/bpf/exception.c +141 -0

155 lines not on this page · raw message

kernel/bpf/exception.h +7 -0
index 0872a3f70583..7a4bfc201e1c 100644
@@ -5,6 +5,10 @@
 
 #include <linux/types.h>
 
+struct bpf_cleanup_info;
+struct bpf_cleanup_range;
+struct bpf_prog;
+struct bpf_prog_aux;
 struct bpf_insn;
 struct bpf_verifier_env;
 
@@ -12,5 +16,8 @@ int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env);
 int bpf_exc_check_insn(struct bpf_verifier_env *env, struct bpf_insn *insn);
 int bpf_exc_check_callback(struct bpf_verifier_env *env, int subprog);
 int bpf_exc_pad_of_call(struct bpf_verifier_env *env, u32 idx);
+int bpf_exc_alloc_info(struct bpf_prog_aux *aux);
+int bpf_exc_attach_info(struct bpf_prog_aux *aux, struct bpf_cleanup_info *recs, u32 cnt);
+const struct bpf_cleanup_range *bpf_exc_pad_for_ip(const struct bpf_prog *prog, u64 ip);
 
 #endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +65 -0

100 lines not on this page · raw message

kernel/bpf/helpers.c +34 -0
index 5eadab4dfa3f..03af739b9909 100644
@@ -31,6 +31,7 @@
 #include <linux/buildid.h>
 
 #include "../../lib/kstrtox.h"
+#include "exception.h"
 
 /* If kernel subsystem is allowing eBPF programs to call this function,
  * inside its own verifier_ops->get_func_proto() callback it should return
@@ -3398,8 +3399,36 @@ struct bpf_throw_ctx {
 	u64 sp;
 	u64 bp;
 	int cnt;
+	const struct bpf_prog *callee;
+	u64 callee_fp;
 };
 
+static void bpf_run_cleanup_pad(struct bpf_throw_ctx *ctx, const struct bpf_prog *prog,
+				u64 ip, u64 fp)
+{
+	const struct bpf_exception_info *exc = prog->aux->exc;
+	const struct bpf_cleanup_range *rec;
+	u64 spill_base;
+
+	if (!exc || !exc->nr_ranges)
+		return;
+	rec = bpf_exc_pad_for_ip(prog, ip);
+	if (!rec)
+		return;
+
+	/*
+	 * The callee is always another subprogram of this program -- the walk
+	 * ends at any frame that is not one -- so its prologue spilled these
+	 * registers and its exc is there to say where.
+	 */
+	if (ctx->callee)
+		spill_base = ctx->callee_fp + ctx->callee->aux->exc->spill_off;
+	else
+		spill_base = fp + exc->throw_spill_off;
+
+	arch_bpf_run_cleanup_pad(rec->pad, fp, spill_base);
+}
+
 static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
 {
 	struct bpf_throw_ctx *ctx = cookie;
@@ -3416,6 +3445,11 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
 	if (!prog)
 		return !ctx->cnt;
 	ctx->cnt++;
+
+	bpf_run_cleanup_pad(ctx, prog, ip, bp);
+	ctx->callee = prog;
+	ctx->callee_fp = bp;
+
 	if (bpf_is_subprog(prog))
 		return true;
 	ctx->aux = prog->aux;
Signature · 1 line
2.53.0-Meta
author reviewer
#31 lore raw

PATCH v5 12/21 bpf, x86: Dispatch exception cleanup pads at run time

In reply to Yonghong Song

Provide the arch half for x86-64: force the full callee-saved spill for a program carrying a cleanup table so the walker can find a frame's r6-r9 in its callee's prologue, record where that spill area starts, build the native cleanup table from the JIT's addrs[], emit a bare return for a pad's bpf_unwind_resume(), and hand control to a pad from arch_bpf_run_cleanup_pad().

Support is gated on CONFIG_UNWINDER_ORC, the same requirement arch_bpf_stack_walk() and therefore bpf_throw() already have here, and bpf_cleanup_pad.o is built only there. Nothing else can call it, and a frame-pointer build would have objtool validate a routine that has to carry BPF r10 in rbp rather than a frame pointer ("call without frame pointer save/setup", an error under CONFIG_OBJTOOL_WERROR).

---
 arch/x86/net/Makefile          |  3 ++
 arch/x86/net/bpf_cleanup_pad.S | 79 +++++++++++++++++++++++++++++++++
 arch/x86/net/bpf_jit_comp.c    | 81 +++++++++++++++++++++++++++++-----
 3 files changed, 151 insertions(+), 12 deletions(-)
 create mode 100644 arch/x86/net/bpf_cleanup_pad.S
arch/x86/net/Makefile +3 -0
index dddbefc0f439..8bb22644ba10 100644
@@ -7,4 +7,7 @@ ifeq ($(CONFIG_X86_32),y)
         obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o
 else
         obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+        ifdef CONFIG_UNWINDER_ORC
+                obj-$(CONFIG_BPF_JIT) += bpf_cleanup_pad.o
+        endif
 endif
arch/x86/net/bpf_cleanup_pad.S new file +79 -0

79 lines not on this page · raw message

arch/x86/net/bpf_jit_comp.c +69 -12

173 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
maintainer
#32 lore raw

Re: [PATCH bpf-next v5 12/21] bpf, x86: Dispatch exception cleanup pads at run time

On Tue, 2026-09-22 at 21:59 -0700, Yonghong Song wrote:

...

Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0 -0
index d4a980140b48..26df52050cd7 100644

...

arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · line 2037

Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +1 -1
@@ -2024,7 +2034,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
  	/* Exception callback will clobber callee regs for its own use, and
  	 * restore the original callee regs from main prog's stack frame.
  	 */
-	if (bpf_prog->aux->exception_boundary) {
+	if (bpf_prog->aux->exception_boundary || bpf_exc_force_spill(bpf_prog)) {

On option to avoid spilling all the registers would be to use register liveness information at call sites and collect a union of registers alive at landing pads for all callers of this particular function. Wdyt?

Quoted 3 lines
arch/x86/net/bpf_jit_comp.c +0 -0
  		/* We also need to save r12, which is not mapped to any BPF
  		 * register, as we throw after entry into the kernel, which may
  		 * overwrite r12.

...

author reviewer
#33 lore raw

Re: [PATCH bpf-next v5 12/21] bpf, x86: Dispatch exception cleanup pads at run time

Quoted 10 lines · Eduard Zingerman

On Tue, 2026-09-22 at 21:59 -0700, Yonghong Song wrote:

...

Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0 -0
index d4a980140b48..26df52050cd7 100644

...

Quoted 10 lines

arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · line 2037

Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +1 -1
@@ -2024,7 +2034,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
  	/* Exception callback will clobber callee regs for its own use, and
  	 * restore the original callee regs from main prog's stack frame.
  	 */
-	if (bpf_prog->aux->exception_boundary) {
+	if (bpf_prog->aux->exception_boundary || bpf_exc_force_spill(bpf_prog)) {

On option to avoid spilling all the registers would be to use register liveness information at call sites and collect a union of registers alive at landing pads for all callers of this particular function. Wdyt?

I will try to fix the above to avoid saving unnecessary callee saved registers. The below is the code in x86 bpf_jit_comp.c:

	push_callee_regs(&prog, callee_regs_used);
Quoted 5 lines
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0 -0
  		/* We also need to save r12, which is not mapped to any BPF
  		 * register, as we throw after entry into the kernel, which may
  		 * overwrite r12.

...

author reviewer
#34 lore raw

PATCH v5 13/21 bpf, arm64: Dispatch exception cleanup pads at run time

In reply to Yonghong Song

The same arch half for arm64: force the full callee-saved spill, record where it starts, build the native table from the JIT's byte offsets, and hand control to a pad from arch_bpf_run_cleanup_pad().

Unlike x86-64 a pad cannot simply return: every call it makes clobbers x30, so no return address survives to its resume. x23 carries it instead. bpf2a64[] maps nothing to x23 or x24 -- they are pushed only to keep the frame shape the exception callback expects -- so nothing else in generated code touches them, and being callee-saved they survive every kfunc the pad calls. The JIT emits "br x23" for a pad's bpf_unwind_resume().

x24 is the other one, and it is what makes a pad able to touch its own frame at all. Generated code addresses the BPF frame through the stack pointer -- the frame sits directly on top of it, which turns every offset positive and each access into one instruction -- and in a pad the stack pointer is the walker's. So the JIT has a pad recompute the equivalent of its frame's stack pointer from BPF r10 on entry, into x24, and addresses the frame off x24 for every instruction the previous patches marked as running only while unwinding. One instruction per pad, and the accesses keep the shape and the immediate range they have everywhere else.

---
 arch/arm64/net/Makefile          |  2 +-
 arch/arm64/net/bpf_cleanup_pad.S | 94 ++++++++++++++++++++++++++++++++
 arch/arm64/net/bpf_jit_comp.c    | 92 +++++++++++++++++++++++++++++--
 3 files changed, 181 insertions(+), 7 deletions(-)
 create mode 100644 arch/arm64/net/bpf_cleanup_pad.S
arch/arm64/net/Makefile +1 -1
index 3ae382bfca87..ebec2a44a52b 100644
@@ -2,4 +2,4 @@
 #
 # ARM64 networking code
 #
-obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o
arch/arm64/net/bpf_cleanup_pad.S new file +94 -0

94 lines not on this page · raw message

arch/arm64/net/bpf_jit_comp.c +86 -6

190 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
author reviewer
#35 lore raw

PATCH v5 14/21 libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc

In reply to Yonghong Song

LLVM terminates a cleanup landing pad with a call to _Unwind_Resume: the base unwind ABI's entry point for carrying an unwind on once a frame's cleanups have run. The kernel provides that terminator as a kfunc, but not under that name: claiming _Unwind_Resume in the kernel's own symbol table, for a function whose body never runs, would be needlessly confusing, so it is called bpf_unwind_resume.

Both load paths take the detour. A direct load resolves the name against the kernel's BTF while libbpf runs. A light skeleton instead writes the name into the loader program's blob of bytes, for that program to resolve when it runs, so the name recorded there has to be translated as well.

---
 tools/lib/bpf/libbpf.c | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)
tools/lib/bpf/libbpf.c +18 -5
index cd1ea1bb53cb..2d0081784445 100644
@@ -8379,6 +8379,16 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s
 	}
 }
 
+/*
+ * LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the
+ * base unwind ABI's entry point for carrying an unwind on once a frame's
+ * cleanups have run. The kernel knows it as bpf_unwind_resume.
+ */
+static const char *kern_extern_name(const char *name)
+{
+	return strcmp(name, "_Unwind_Resume") ? name : "bpf_unwind_resume";
+}
+
 static int bpf_program_record_relos(struct bpf_program *prog)
 {
 	struct bpf_object *obj = prog->obj;
@@ -8395,12 +8405,12 @@ static int bpf_program_record_relos(struct bpf_program *prog)
 				continue;
 			kind = btf_is_var(btf__type_by_id(obj->btf, ext->btf_id)) ?
 				BTF_KIND_VAR : BTF_KIND_FUNC;
-			bpf_gen__record_extern(obj->gen_loader, ext->name,
+			bpf_gen__record_extern(obj->gen_loader, kern_extern_name(ext->name),
 					       ext->is_weak, !ext->ksym.type_id,
 					       true, kind, relo->insn_idx);
 			break;
 		case RELO_EXTERN_CALL:
-			bpf_gen__record_extern(obj->gen_loader, ext->name,
+			bpf_gen__record_extern(obj->gen_loader, kern_extern_name(ext->name),
 					       ext->is_weak, false, false, BTF_KIND_FUNC,
 					       relo->insn_idx);
 			break;
@@ -8876,17 +8886,20 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,
 	struct module_btf *mod_btf = NULL;
 	const struct btf_type *kern_func;
 	struct btf *kern_btf = NULL;
+	const char *local_name, *kern_name;
 	int ret;
 
 	local_func_proto_id = ext->ksym.type_id;
 
-	kfunc_id = find_ksym_btf_id(obj, ext->essent_name ?: ext->name, BTF_KIND_FUNC, &kern_btf,
-				    &mod_btf);
+	local_name = ext->essent_name ?: ext->name;
+	kern_name = kern_extern_name(local_name);
+
+	kfunc_id = find_ksym_btf_id(obj, kern_name, BTF_KIND_FUNC, &kern_btf, &mod_btf);
 	if (kfunc_id < 0) {
 		if (kfunc_id == -ESRCH && ext->is_weak)
 			return 0;
 		pr_warn("extern (func ksym) '%s': not found in kernel or module BTFs\n",
-			ext->name);
+			kern_name != local_name ? kern_name : ext->name);
 		return kfunc_id;
 	}
 
Signature · 1 line
2.53.0-Meta
#36 lore raw

Re: [PATCH bpf-next v5 14/21] libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc

Quoted 58 lines · Yonghong Song

LLVM terminates a cleanup landing pad with a call to _Unwind_Resume: the base unwind ABI's entry point for carrying an unwind on once a frame's cleanups have run. The kernel provides that terminator as a kfunc, but not under that name: claiming _Unwind_Resume in the kernel's own symbol table, for a function whose body never runs, would be needlessly confusing, so it is called bpf_unwind_resume.

Both load paths take the detour. A direct load resolves the name against the kernel's BTF while libbpf runs. A light skeleton instead writes the name into the loader program's blob of bytes, for that program to resolve when it runs, so the name recorded there has to be translated as well.

---
 tools/lib/bpf/libbpf.c | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)
tools/lib/bpf/libbpf.c +13 -2
index cd1ea1bb53cb..2d0081784445 100644
@@ -8379,6 +8379,16 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s
 	}
 }
 
+/*
+ * LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the
+ * base unwind ABI's entry point for carrying an unwind on once a frame's
+ * cleanups have run. The kernel knows it as bpf_unwind_resume.
+ */
+static const char *kern_extern_name(const char *name)
+{
+	return strcmp(name, "_Unwind_Resume") ? name : "bpf_unwind_resume";
+}
+
 static int bpf_program_record_relos(struct bpf_program *prog)
 {
 	struct bpf_object *obj = prog->obj;
@@ -8395,12 +8405,12 @@ static int bpf_program_record_relos(struct bpf_program *prog)
 				continue;
 			kind = btf_is_var(btf__type_by_id(obj->btf, ext->btf_id)) ?
 				BTF_KIND_VAR : BTF_KIND_FUNC;
-			bpf_gen__record_extern(obj->gen_loader, ext->name,
+			bpf_gen__record_extern(obj->gen_loader, kern_extern_name(ext->name),
 					       ext->is_weak, !ext->ksym.type_id,
 					       true, kind, relo->insn_idx);
 			break;
 		case RELO_EXTERN_CALL:
-			bpf_gen__record_extern(obj->gen_loader, ext->name,
+			bpf_gen__record_extern(obj->gen_loader, kern_extern_name(ext->name),
 					       ext->is_weak, false, false, BTF_KIND_FUNC,
 					       relo->insn_idx);
 			break;
@@ -8876,17 +8886,20 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,
 	struct module_btf *mod_btf = NULL;
 	const struct btf_type *kern_func;
 	struct btf *kern_btf = NULL;
+	const char *local_name, *kern_name;

tools/lib/bpf/libbpf.c static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj, · line 8897

Quoted 10 lines
tools/lib/bpf/libbpf.c +4 -2
 	int ret;
 
 	local_func_proto_id = ext->ksym.type_id;
 
-	kfunc_id = find_ksym_btf_id(obj, ext->essent_name ?: ext->name, BTF_KIND_FUNC, &kern_btf,
-				    &mod_btf);
+	local_name = ext->essent_name ?: ext->name;
+	kern_name = kern_extern_name(local_name);
+
+	kfunc_id = find_ksym_btf_id(obj, kern_name, BTF_KIND_FUNC, &kern_btf, &mod_btf);

Compiler doesn't directly emit it in .ksyms section, right? How does this work in practice? It's a bit confusing and not clear from the commit log.

Quoted 1 line

[...]

author reviewer
#37 lore raw

PATCH v5 15/21 libbpf: Add cleanup_info to bpf_prog_load_opts

In reply to Yonghong Song

Let a caller hand the kernel an exception cleanup table: struct bpf_prog_load_opts grows cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, and bpf_prog_load() passes all three on to BPF_PROG_LOAD.

The attr size it computes grows with them, by one field rather than
three: cleanup_info_cnt is the last of the three in the BPF_PROG_LOAD attr, so offsetofend() on it already covers cleanup_info and cleanup_info_rec_size.

---
 tools/lib/bpf/bpf.c | 6 +++++-
 tools/lib/bpf/bpf.h | 7 ++++++-
 2 files changed, 11 insertions(+), 2 deletions(-)
tools/lib/bpf/bpf.c +5 -1
index a9de7f107cf7..03315b3a3a19 100644
@@ -295,7 +295,7 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
 		  const struct bpf_insn *insns, size_t insn_cnt,
 		  struct bpf_prog_load_opts *opts)
 {
-	const size_t attr_sz = offsetofend(union bpf_attr, keyring_id);
+	const size_t attr_sz = offsetofend(union bpf_attr, cleanup_info_cnt);
 	void *finfo = NULL, *linfo = NULL;
 	const char *func_info, *line_info;
 	__u32 log_size, log_level, attach_prog_fd, attach_btf_obj_fd;
@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
 	attr.fd_array = ptr_to_u64(OPTS_GET(opts, fd_array, NULL));
 	attr.fd_array_cnt = OPTS_GET(opts, fd_array_cnt, 0);
 
+	attr.cleanup_info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);
+	attr.cleanup_info = ptr_to_u64(OPTS_GET(opts, cleanup_info, NULL));
+	attr.cleanup_info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+
 	if (log_level) {
 		attr.log_buf = ptr_to_u64(log_buf);
 		attr.log_size = log_size;
tools/lib/bpf/bpf.h +6 -1
index 490e8cb4ba53..ec542fd68f63 100644
@@ -128,9 +128,14 @@ struct bpf_prog_load_opts {
 
 	/* if set, provides the length of fd_array */
 	__u32 fd_array_cnt;
+
+	/* exception cleanup table, from the .bpf_cleanup section */
+	const void *cleanup_info;
+	__u32 cleanup_info_cnt;
+	__u32 cleanup_info_rec_size;
 	size_t :0;
 };
-#define bpf_prog_load_opts__last_field fd_array_cnt
+#define bpf_prog_load_opts__last_field cleanup_info_rec_size
 
 LIBBPF_API int bpf_prog_load(enum bpf_prog_type prog_type,
 			     const char *prog_name, const char *license,
Signature · 1 line
2.53.0-Meta
author reviewer
#38 lore raw

PATCH v5 16/21 libbpf: Collect .bpf_cleanup records and pass them to the kernel

In reply to Yonghong Song

Parse the compiler-emitted .bpf_cleanup section and hand the resulting table to BPF_PROG_LOAD.

Each record is three 4-byte fields, and each field is a byte offset into some code section named by a matching .rel.bpf_cleanup relocation. bpf_object__init_cleanup_info() resolves both halves once at open time and keeps (section, instruction index) pairs; it rejects a record whose field has no relocation, whose relocation is not one of the two 32-bit types that spell a data reference to a code section -- R_BPF_64_NODYLD32 from LLVM, R_BPF_64_ABS32 from GNU as -- or whose offset is not instruction aligned.

The records are sorted by begin_off once the offsets are final: the kernel wants the table sorted with disjoint ranges so that it can find the record covering a call site with a binary search, and records arrive in .bpf_cleanup order, which says nothing about where the subprograms they describe were appended. Overlapping ranges are reported here, where the program name and both regions are still at hand.

bpf_object_load_prog() then passes the per-program table through the bpf_prog_load() options added in the previous patch, with the record size carried on the program the way func_info and line_info carry theirs rather than taken from a sizeof() at the call site. bpf_program__clone() carries it too. That is a second load path -- the one veristat uses -- and without the table the kernel sees landing pads nothing reaches and refuses the program with "unreachable insn".

---
 tools/lib/bpf/libbpf.c          | 309 ++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |   3 +
 2 files changed, 312 insertions(+)
tools/lib/bpf/libbpf.c +309 -0
index 2d0081784445..d0e601b3499c 100644
@@ -514,6 +514,11 @@ struct bpf_program {
 	void *line_info;
 	__u32 line_info_rec_size;
 	__u32 line_info_cnt;
+
+	struct bpf_cleanup_info *cleanup_info;
+	__u32 cleanup_info_rec_size;
+	__u32 cleanup_info_cnt;
+
 	__u32 prog_flags;
 	__u8  hash[SHA256_DIGEST_LENGTH];
 
@@ -549,6 +554,7 @@ struct bpf_struct_ops {
 #define STRUCT_OPS_SEC ".struct_ops"
 #define STRUCT_OPS_LINK_SEC ".struct_ops.link"
 #define ARENA_SEC ".addr_space.1"
+#define CLEANUP_SEC ".bpf_cleanup"
 
 enum libbpf_map_type {
 	LIBBPF_MAP_UNSPEC,
@@ -677,6 +683,26 @@ struct elf_sec_desc {
 	Elf_Data *data;
 };
 
+#define CLEANUP_REC_FIELDS	(sizeof(struct bpf_cleanup_info) / sizeof(__u32))
+
+/* Index of each field of struct bpf_cleanup_info, read as an array of __u32. */
+enum {
+	CLEANUP_REC_BEGIN,
+	CLEANUP_REC_END,
+	CLEANUP_REC_PAD,
+};
+
+/*
+ * One (begin, end, landing_pad) triple from .bpf_cleanup, with each field
+ * resolved from its relocation to an ELF section plus a section-relative
+ * instruction index. The mapping to final program instruction indices can only
+ * happen after subprogram placement, which differs per main program.
+ */
+struct cleanup_raw_rec {
+	int sec_idx[CLEANUP_REC_FIELDS];
+	size_t insn_idx[CLEANUP_REC_FIELDS];
+};
+
 struct elf_state {
 	int fd;
 	const void *obj_buf;
@@ -696,6 +722,8 @@ struct elf_state {
 	bool has_st_ops;
 	int arena_data_shndx;
 	int jumptables_data_shndx;
+	Elf_Data *cleanup_data;
+	int cleanup_shndx;
 };
 
 struct usdt_manager;
@@ -773,6 +801,9 @@ struct bpf_object {
 	void *jumptables_data;
 	size_t jumptables_data_sz;
 
+	struct cleanup_raw_rec *cleanup_recs;
+	size_t cleanup_rec_cnt;
+
 	struct {
 		struct bpf_program *prog;
 		unsigned int sym_off;
@@ -819,7 +850,10 @@ static void bpf_program__exit(struct bpf_program *prog)
 	zfree(&prog->sec_name);
 	zfree(&prog->insns);
 	zfree(&prog->reloc_desc);
+	zfree(&prog->cleanup_info);
 
+	prog->cleanup_info_rec_size = 0;
+	prog->cleanup_info_cnt = 0;
 	prog->nr_reloc = 0;
 	prog->insns_cnt = 0;
 	prog->sec_idx = -1;
@@ -1561,6 +1595,7 @@ static struct bpf_object *bpf_object__new(const char *path,
 	obj->efile.obj_buf = obj_buf;
 	obj->efile.obj_buf_sz = obj_buf_sz;
 	obj->efile.btf_maps_shndx = -1;
+	obj->efile.cleanup_shndx = -1;
 	obj->kconfig_map_idx = -1;
 	obj->arena_map_idx = -1;
 
@@ -4047,6 +4082,9 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 				sec_desc->shdr = sh;
 				sec_desc->data = data;
 				obj->efile.has_st_ops = true;
+			} else if (strcmp(name, CLEANUP_SEC) == 0) {
+				obj->efile.cleanup_data = data;
+				obj->efile.cleanup_shndx = idx;
 			} else if (strcmp(name, ARENA_SEC) == 0) {
 				obj->efile.arena_data = data;
 				obj->efile.arena_data_shndx = idx;
@@ -4074,6 +4112,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 			    strcmp(name, ".rel" STRUCT_OPS_LINK_SEC) &&
 			    strcmp(name, ".rel?" STRUCT_OPS_SEC) &&
 			    strcmp(name, ".rel?" STRUCT_OPS_LINK_SEC) &&
+			    strcmp(name, ".rel" CLEANUP_SEC) &&
 			    strcmp(name, ".rel" MAPS_ELF_SEC)) {
 				pr_info("elf: skipping relo section(%d) %s for section(%d) %s\n",
 					idx, name, targ_sec_idx,
@@ -4854,6 +4893,238 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,
 	return NULL;
 }
 
+static int bpf_object__init_cleanup_info(struct bpf_object *obj)
+{
+	Elf_Data *data = obj->efile.cleanup_data;
+	Elf_Data *relo = NULL;
+	size_t i, nrels, nslots, nrecs;
+	struct cleanup_raw_rec *recs;
+	int *slot_sec, ret = 0;
+	size_t *slot_val;
+	const __u32 *vals;
+	bool native;
+
+	if (!data || obj->efile.cleanup_shndx < 0)
+		return 0;
+
+	native = is_native_endianness(obj);
+
+	for (i = 0; i < obj->efile.sec_cnt; i++) {
+		struct elf_sec_desc *sd = &obj->efile.secs[i];
+
+		if (sd->sec_type == SEC_RELO && sd->shdr &&
+		    sd->shdr->sh_info == (Elf64_Word)obj->efile.cleanup_shndx) {
+			relo = sd->data;
+			break;
+		}
+	}
+	if (!relo) {
+		pr_warn("%s present without relocations\n", CLEANUP_SEC);
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+	if (data->d_size % sizeof(struct bpf_cleanup_info)) {
+		pr_warn("%s size %zu is not a multiple of the record size %zu\n",
+			CLEANUP_SEC, data->d_size, sizeof(struct bpf_cleanup_info));
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+
+	vals = data->d_buf;
+	nslots = data->d_size / sizeof(__u32);
+	nrecs = data->d_size / sizeof(struct bpf_cleanup_info);
+
+	slot_sec = calloc(nslots, sizeof(*slot_sec));
+	slot_val = calloc(nslots, sizeof(*slot_val));
+	recs = calloc(nrecs ?: 1, sizeof(*recs));
+	if (!slot_sec || !slot_val || !recs) {
+		ret = -ENOMEM;
+		goto out;
+	}
+	for (i = 0; i < nslots; i++)
+		slot_sec[i] = -1;
+
+	/* One relocation per 4-byte field, naming the section it points into. */
+	nrels = relo->d_size / sizeof(Elf64_Rel);
+	for (i = 0; i < nrels; i++) {
+		Elf64_Rel *rel = elf_rel_by_idx(relo, i);
+		Elf64_Sym *sym = elf_sym_by_idx(obj, ELF64_R_SYM(rel->r_info));
+		size_t type = ELF64_R_TYPE(rel->r_info);
+		size_t slot = rel->r_offset / sizeof(__u32);
+
+		if (type != R_BPF_64_NODYLD32 && type != R_BPF_64_ABS32) {
+			pr_warn("%s: relocation %zu has unexpected type %zu\n",
+				CLEANUP_SEC, i, type);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		if (!sym || slot >= nslots || rel->r_offset % sizeof(__u32)) {
+			pr_warn("%s: bad relocation %zu\n", CLEANUP_SEC, i);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		slot_sec[slot] = sym->st_shndx;
+		/*
+		 * The addend lives in the section data, which libelf leaves in
+		 * the object's byte order; a non-section symbol additionally
+		 * contributes its own value.
+		 */
+		slot_val[slot] = (native ? vals[slot] : bswap_32(vals[slot])) +
+				 sym->st_value;
+	}
+
+	for (i = 0; i < nslots; i++) {
+		struct cleanup_raw_rec *rec = &recs[i / CLEANUP_REC_FIELDS];
+		size_t field = i % CLEANUP_REC_FIELDS;
+
+		if (slot_sec[i] < 0) {
+			pr_warn("%s: field %zu has no relocation\n", CLEANUP_SEC, i);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		if (slot_val[i] % BPF_INSN_SZ) {
+			pr_warn("%s: field %zu offset %zu is not instruction aligned\n",
+				CLEANUP_SEC, i, slot_val[i]);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		rec->sec_idx[field] = slot_sec[i];
+		rec->insn_idx[field] = slot_val[i] / BPF_INSN_SZ;
+	}
+
+	obj->cleanup_recs = recs;
+	obj->cleanup_rec_cnt = nrecs;
+	recs = NULL;
+out:
+	free(recs);
+	free(slot_val);
+	free(slot_sec);
+	return ret;
+}
+
+static int cmp_cleanup_info(const void *a, const void *b)
+{
+	const struct bpf_cleanup_info *x = a, *y = b;
+
+	if (x->begin_off == y->begin_off)
+		return 0;
+	return x->begin_off < y->begin_off ? -1 : 1;
+}
+
+static int bpf_prog_collect_cleanup_info(struct bpf_object *obj,
+					 struct bpf_program *prog)
+{
+	size_t i;
+	int j;
+
+	for (i = 0; i < obj->cleanup_rec_cnt; i++) {
+		struct cleanup_raw_rec *raw = &obj->cleanup_recs[i];
+		struct bpf_program *owner = NULL;
+		struct bpf_cleanup_info ci = {};
+		__u32 fields[CLEANUP_REC_FIELDS];
+		void *tmp;
+
+		if (raw->sec_idx[CLEANUP_REC_BEGIN] == raw->sec_idx[CLEANUP_REC_END] &&
+		    raw->insn_idx[CLEANUP_REC_BEGIN] >= raw->insn_idx[CLEANUP_REC_END]) {
+			pr_warn("%s: record %zu is an empty range [%zu,%zu)\n",
+				CLEANUP_SEC, i, raw->insn_idx[CLEANUP_REC_BEGIN],
+				raw->insn_idx[CLEANUP_REC_END]);
+			return -LIBBPF_ERRNO__FORMAT;
+		}
+
+		for (j = 0; j < CLEANUP_REC_FIELDS; j++) {
+			size_t idx = raw->insn_idx[j], final;
+			struct bpf_program *p;
+
+			/*
+			 * The end of a range is exclusive, so it may name the
+			 * instruction just past the last one of a function,
+			 * which belongs to the next function or to nothing at
+			 * all. Ask about the last instruction the range covers,
+			 * the way the kernel does.
+			 */
+			if (j == CLEANUP_REC_END) {
+				if (!idx) {
+					pr_warn("%s: record %zu ends at instruction 0\n",
+						CLEANUP_SEC, i);
+					return -LIBBPF_ERRNO__FORMAT;
+				}
+				idx--;
+			}
+
+			p = find_prog_by_sec_insn(obj, raw->sec_idx[j], idx);
+			if (!p) {
+				pr_warn("%s: record %zu field %d is not inside a function\n",
+					CLEANUP_SEC, i, j);
+				return -LIBBPF_ERRNO__FORMAT;
+			}
+			if (!owner) {
+				owner = p;
+			} else if (owner != p) {
+				pr_warn("%s: record %zu spans functions '%s' and '%s'\n",
+					CLEANUP_SEC, i, owner->name, p->name);
+				return -LIBBPF_ERRNO__FORMAT;
+			}
+
+			if (owner == prog) {
+				final = raw->insn_idx[j] - prog->sec_insn_off;
+			} else if (prog_is_subprog(obj, owner) && owner->sub_insn_off) {
+				/*
+				 * sub_insn_off is where this subprogram was
+				 * appended to the main program being relocated;
+				 * zero means it is not part of it.
+				 */
+				final = owner->sub_insn_off +
+					raw->insn_idx[j] - owner->sec_insn_off;
+			} else {
+				owner = NULL;
+				break;
+			}
+			fields[j] = final;
+		}
+		if (!owner)
+			continue;
+
+		ci.begin_off = fields[CLEANUP_REC_BEGIN];
+		ci.end_off = fields[CLEANUP_REC_END];
+		ci.landing_pad_off = fields[CLEANUP_REC_PAD];
+
+		tmp = libbpf_reallocarray(prog->cleanup_info, prog->cleanup_info_cnt + 1,
+					  sizeof(*prog->cleanup_info));
+		if (!tmp)
+			return -ENOMEM;
+		prog->cleanup_info = tmp;
+		prog->cleanup_info_rec_size = sizeof(struct bpf_cleanup_info);
+		prog->cleanup_info[prog->cleanup_info_cnt++] = ci;
+
+		pr_debug("prog '%s': cleanup region [%u,%u) -> landing pad %u\n",
+			 prog->name, ci.begin_off, ci.end_off, ci.landing_pad_off);
+	}
+
+	if (!prog->cleanup_info_cnt)
+		return 0;
+
+	if (prog->cleanup_info_cnt > INT32_MAX / sizeof(struct bpf_cleanup_info)) {
+		pr_warn("prog '%s': too many cleanup records: %u\n",
+			prog->name, prog->cleanup_info_cnt);
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+
+	qsort(prog->cleanup_info, prog->cleanup_info_cnt,
+	      sizeof(*prog->cleanup_info), cmp_cleanup_info);
+	for (i = 1; i < prog->cleanup_info_cnt; i++) {
+		struct bpf_cleanup_info *prev = &prog->cleanup_info[i - 1];
+		struct bpf_cleanup_info *cur = &prog->cleanup_info[i];
+
+		if (cur->begin_off < prev->end_off) {
+			pr_warn("prog '%s': overlapping cleanup regions [%u,%u) and [%u,%u)\n",
+				prog->name, prev->begin_off, prev->end_off,
+				cur->begin_off, cur->end_off);
+			return -LIBBPF_ERRNO__FORMAT;
+		}
+	}
+
+	return 0;
+}
+
 static int
 bpf_object__collect_prog_relos(struct bpf_object *obj, Elf64_Shdr *shdr, Elf_Data *data)
 {
@@ -7589,6 +7860,13 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_pat
 					return err;
 			}
 		}
+
+		err = bpf_prog_collect_cleanup_info(obj, prog);
+		if (err) {
+			pr_warn("prog '%s': failed to collect cleanup info: %s\n",
+				prog->name, errstr(err));
+			return err;
+		}
 	}
 	for (i = 0; i < obj->nr_programs; i++) {
 		prog = &obj->programs[i];
@@ -7779,6 +8057,9 @@ static int bpf_object__collect_relos(struct bpf_object *obj)
 			return -LIBBPF_ERRNO__INTERNAL;
 		}
 
+		if (idx == obj->efile.cleanup_shndx)
+			continue;
+
 		if (obj->efile.secs[idx].sec_type == SEC_ST_OPS)
 			err = bpf_object__collect_st_ops_relos(obj, shdr, data);
 		else if (idx == obj->efile.btf_maps_shndx)
@@ -8051,6 +8332,11 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
 		load_attr.line_info_rec_size = prog->line_info_rec_size;
 		load_attr.line_info_cnt = prog->line_info_cnt;
 	}
+	if (prog->cleanup_info_cnt) {
+		load_attr.cleanup_info = prog->cleanup_info;
+		load_attr.cleanup_info_cnt = prog->cleanup_info_cnt;
+		load_attr.cleanup_info_rec_size = prog->cleanup_info_rec_size;
+	}
 	load_attr.log_level = log_level;
 	load_attr.prog_flags = prog->prog_flags;
 	load_attr.fd_array = obj->fd_array;
@@ -8644,6 +8930,7 @@ static struct bpf_object *bpf_object_open(const char *path, const void *obj_buf,
 	err = err ? : bpf_object__init_maps(obj, opts);
 	err = err ? : bpf_object_init_progs(obj, opts);
 	err = err ? : bpf_object__collect_relos(obj);
+	err = err ? : bpf_object__init_cleanup_info(obj);
 	if (err)
 		goto out;
 
@@ -9758,6 +10045,9 @@ void bpf_object__close(struct bpf_object *obj)
 	zfree(&obj->jumptables_data);
 	obj->jumptables_data_sz = 0;
 
+	zfree(&obj->cleanup_recs);
+	obj->cleanup_rec_cnt = 0;
+
 	for (i = 0; i < obj->jumptable_map_cnt; i++)
 		close(obj->jumptable_maps[i].fd);
 	zfree(&obj->jumptable_maps);
@@ -10156,6 +10446,25 @@ int bpf_program__clone(struct bpf_program *prog, const struct bpf_prog_load_opts
 		attr.line_info_rec_size = info ? info_rec_size : prog->line_info_rec_size;
 	}
 
+	/* exception cleanup table */
+	info = OPTS_GET(opts, cleanup_info, NULL);
+	info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+	info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);
+	if (!!info != !!info_cnt || !!info != !!info_rec_size) {
+		pr_warn("prog '%s': cleanup_info, cleanup_info_cnt, and cleanup_info_rec_size must all be specified or all omitted\n",
+			prog->name);
+		return libbpf_err(-EINVAL);
+	}
+	if (info) {
+		attr.cleanup_info = info;
+		attr.cleanup_info_cnt = info_cnt;
+		attr.cleanup_info_rec_size = info_rec_size;
+	} else if (prog->cleanup_info_cnt) {
+		attr.cleanup_info = prog->cleanup_info;
+		attr.cleanup_info_cnt = prog->cleanup_info_cnt;
+		attr.cleanup_info_rec_size = prog->cleanup_info_rec_size;
+	}
+
 	/* Logging is caller-controlled; no fallback to prog/obj log settings */
 	attr.log_buf = OPTS_GET(opts, log_buf, NULL);
 	attr.log_size = OPTS_GET(opts, log_size, 0);
tools/lib/bpf/libbpf_internal.h +3 -0
index cb4d96233844..3ba6d9090368 100644
@@ -56,6 +56,9 @@
 #ifndef R_BPF_64_ABS32
 #define R_BPF_64_ABS32 3
 #endif
+#ifndef R_BPF_64_NODYLD32
+#define R_BPF_64_NODYLD32 4
+#endif
 #ifndef R_BPF_64_32
 #define R_BPF_64_32 10
 #endif
Signature · 1 line
2.53.0-Meta
#39 lore raw

Re: [PATCH bpf-next v5 16/21] libbpf: Collect .bpf_cleanup records and pass them to the kernel

Quoted 26 lines · Yonghong Song

Parse the compiler-emitted .bpf_cleanup section and hand the resulting table to BPF_PROG_LOAD.

Each record is three 4-byte fields, and each field is a byte offset into some code section named by a matching .rel.bpf_cleanup relocation. bpf_object__init_cleanup_info() resolves both halves once at open time and keeps (section, instruction index) pairs; it rejects a record whose field has no relocation, whose relocation is not one of the two 32-bit types that spell a data reference to a code section -- R_BPF_64_NODYLD32 from LLVM, R_BPF_64_ABS32 from GNU as -- or whose offset is not instruction aligned.

The records are sorted by begin_off once the offsets are final: the kernel wants the table sorted with disjoint ranges so that it can find the record covering a call site with a binary search, and records arrive in .bpf_cleanup order, which says nothing about where the subprograms they describe were appended. Overlapping ranges are reported here, where the program name and both regions are still at hand.

bpf_object_load_prog() then passes the per-program table through the bpf_prog_load() options added in the previous patch, with the record size carried on the program the way func_info and line_info carry theirs rather than taken from a sizeof() at the call site. bpf_program__clone() carries it too. That is a second load path -- the one veristat uses -- and without the table the kernel sees landing pads nothing reaches and refuses the program with "unreachable insn".

Quoted 2 lines

---

Should we treat sh_entsize == 0 as 12 (or change LLVM to set it), just so that wider records, if necessary for any reason in the future, end up being easier to reject?

Quoted 1 line

[...]

author reviewer
#40 lore raw

PATCH v5 17/21 libbpf: Carry the exception cleanup table through the light skeleton

In reply to Yonghong Song

A light skeleton does not call bpf_prog_load(). bpf_gen__prog_load() builds its own union bpf_attr field by field, and the loader program it emits is what issues BPF_PROG_LOAD when the skeleton runs -- so a program loaded this way reached the kernel without the table the previous patch collected for it. The load then fails, because the landing pads are code nothing reaches and the verifier says so. It says "unreachable insn", which names neither the skeleton nor the table.

Carry it the way func_info and line_info are carried: the records go into the loader's blob of bytes, the count and record size into the attr, and a relocation stores the blob's address into attr.cleanup_info once that address is known. The attr grows to its new last field, cleanup_info_cnt. Records are 4-byte fields like the other info blobs, so a cross-endian build has to swap them too.

---
 tools/lib/bpf/gen_loader.c      | 29 +++++++++++++++++++++++++----
 tools/lib/bpf/libbpf_internal.h |  7 +++++++
 2 files changed, 32 insertions(+), 4 deletions(-)
tools/lib/bpf/gen_loader.c +25 -4
index af3a04f161ac..2345fbdd46f5 100644
@@ -981,13 +981,15 @@ static void cleanup_relos(struct bpf_gen *gen, int insns)
 	cleanup_core_relo(gen);
 }
 
-/* Convert func, line, and core relo info blobs to target endianness */
+/* Convert func, line, core relo and cleanup info blobs to target endianness */
 static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
-			    int core_relos, struct bpf_prog_load_opts *load_attr)
+			    int core_relos, int cleanup_info,
+			    struct bpf_prog_load_opts *load_attr)
 {
 	struct bpf_func_info *fi = gen->data_start + func_info;
 	struct bpf_line_info *li = gen->data_start + line_info;
 	struct bpf_core_relo *cr = gen->data_start + core_relos;
+	struct bpf_cleanup_info *ci = gen->data_start + cleanup_info;
 	int i;
 
 	for (i = 0; i < load_attr->func_info_cnt; i++)
@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
 
 	for (i = 0; i < gen->core_relo_cnt; i++)
 		bpf_core_relo_bswap(cr++);
+
+	for (i = 0; i < load_attr->cleanup_info_cnt; i++)
+		bpf_cleanup_info_bswap(ci++);
 }
 
 void bpf_gen__prog_load(struct bpf_gen *gen,
@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 			       load_attr->line_info_rec_size;
 	int core_relo_tot_sz = gen->core_relo_cnt *
 			       sizeof(struct bpf_core_relo);
+	int cleanup_info_tot_sz = load_attr->cleanup_info_cnt *
+				  load_attr->cleanup_info_rec_size;
 	int prog_load_attr, license_off, insns_off, func_info, line_info, core_relos;
-	int attr_size = offsetofend(union bpf_attr, core_relo_rec_size);
+	int attr_size = offsetofend(union bpf_attr, cleanup_info_cnt);
+	int cleanup_info;
 	union bpf_attr attr;
 
 	memset(&attr, 0, attr_size);
@@ -1061,9 +1069,17 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 		 core_relos, gen->core_relo_cnt,
 		 sizeof(struct bpf_core_relo));
 
+	attr.cleanup_info_rec_size = tgt_endian(load_attr->cleanup_info_rec_size);
+	attr.cleanup_info_cnt = tgt_endian(load_attr->cleanup_info_cnt);
+	cleanup_info = add_data(gen, load_attr->cleanup_info, cleanup_info_tot_sz);
+	pr_debug("gen: prog_load: cleanup_info: off %d cnt %u rec size %u\n",
+		 cleanup_info, load_attr->cleanup_info_cnt,
+		 load_attr->cleanup_info_rec_size);
+
 	/* convert all info blobs to target endianness */
 	if (gen->swapped_endian && !gen->error)
-		info_blob_bswap(gen, func_info, line_info, core_relos, load_attr);
+		info_blob_bswap(gen, func_info, line_info, core_relos, cleanup_info,
+				load_attr);
 
 	libbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name));
 	prog_load_attr = add_data(gen, &attr, attr_size);
@@ -1085,6 +1101,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 	/* populate union bpf_attr with a pointer to core_relos */
 	emit_rel_store(gen, attr_field(prog_load_attr, core_relos), core_relos);
 
+	/* populate union bpf_attr with a pointer to cleanup_info, if there is one */
+	if (load_attr->cleanup_info_cnt)
+		emit_rel_store(gen, attr_field(prog_load_attr, cleanup_info),
+			       cleanup_info);
+
 	/* populate union bpf_attr fd_array with a pointer to data where map_fds are saved */
 	emit_rel_store(gen, attr_field(prog_load_attr, fd_array), gen->fd_array);
 
tools/lib/bpf/libbpf_internal.h +7 -0
index 3ba6d9090368..78519f24fb40 100644
@@ -572,6 +572,13 @@ static inline void bpf_core_relo_bswap(struct bpf_core_relo *i)
 	i->kind = bswap_32(i->kind);
 }
 
+static inline void bpf_cleanup_info_bswap(struct bpf_cleanup_info *i)
+{
+	i->begin_off = bswap_32(i->begin_off);
+	i->end_off = bswap_32(i->end_off);
+	i->landing_pad_off = bswap_32(i->landing_pad_off);
+}
+
 enum btf_field_iter_kind {
 	BTF_FIELD_ITER_IDS,
 	BTF_FIELD_ITER_STRS,
Signature · 1 line
2.53.0-Meta
author reviewer
#41 lore raw

PATCH v5 18/21 libbpf: Let the static linker carry .bpf_cleanup relocations

In reply to Yonghong Song

An object that carries a compiler-emitted exception cleanup table cannot be linked today. The table's fields are byte offsets into a code section, materialised by a 32-bit relocation against that section's symbol with the offset itself as the implicit addend, and the linker rejects both halves of
that: the relocation type is not in the list it accepts, and a relocation against an STT_SECTION symbol from a non-executable section is an outright error.

Both spellings of that relocation have to be taken. LLVM emits R_BPF_64_NODYLD32 for a .long against a section symbol; GNU as emits R_BPF_64_ABS32, which is what bpf_reloc_type_lookup() maps BFD_RELOC_32 to. They describe the same value, and the selftests are built with both compilers.

Taking them is keyed on the relocation type rather than on the section name, so any non-executable section could reach the new arm, and what it replaces is an unconditional refusal. Check what the refusal used to make
unnecessary: a relocated section may be SHT_NOBITS, which extend_sec() leaves with no raw_data, and r_offset is checked for alignment only where the section holds instructions. Refuse those rather than write through them.

---
 tools/lib/bpf/linker.c | 38 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 37 insertions(+), 1 deletion(-)
tools/lib/bpf/linker.c +37 -1
index 78f92c39290a..95f2a5243eda 100644
@@ -1036,7 +1036,8 @@ static int linker_sanity_check_elf_relos(struct src_obj *obj, struct src_sec *se
 		size_t sym_type = ELF64_R_TYPE(relo->r_info);
 
 		if (sym_type != R_BPF_64_64 && sym_type != R_BPF_64_32 &&
-		    sym_type != R_BPF_64_ABS64 && sym_type != R_BPF_64_ABS32) {
+		    sym_type != R_BPF_64_ABS64 && sym_type != R_BPF_64_ABS32 &&
+		    sym_type != R_BPF_64_NODYLD32) {
 			pr_warn("ELF relo #%d in section #%zu has unexpected type %zu in %s\n",
 				i, sec->sec_idx, sym_type, obj->filename);
 			return -EINVAL;
@@ -2258,6 +2259,7 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
 			if (ELF64_ST_TYPE(src_sym->st_info) == STT_SECTION) {
 				struct src_sec *sec = &obj->secs[src_sym->st_shndx];
 				struct bpf_insn *insn;
+				__u32 *val;
 
 				if (src_linked_sec->shdr->sh_flags & SHF_EXECINSTR) {
 					/* calls to the very first static function inside
@@ -2274,6 +2276,40 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
 						insn->imm += sec->dst_off / sizeof(struct bpf_insn);
 					else
 						insn->imm += sec->dst_off;
+				} else if (sym_type == R_BPF_64_NODYLD32 ||
+					   sym_type == R_BPF_64_ABS32) {
+					/*
+					 * Two spellings of the one thing: LLVM
+					 * emits NODYLD32 for a .long against a
+					 * section symbol, GNU as emits ABS32
+					 * (bpf_reloc_type_lookup() maps
+					 * BFD_RELOC_32 to it), and the value
+					 * they describe is the same.
+					 */
+
+					/*
+					 * Only an executable section has its
+					 * r_offset checked, and even there only
+					 * for alignment; SHT_NOBITS has no
+					 * raw_data at all. Bound it here,
+					 * subtracting so it cannot wrap.
+					 */
+					if (!dst_linked_sec->raw_data ||
+					    dst_linked_sec->sec_sz < (int)sizeof(*val) ||
+					    dst_rel->r_offset % sizeof(*val) ||
+					    dst_rel->r_offset >
+					    (size_t)dst_linked_sec->sec_sz - sizeof(*val)) {
+						pr_warn("ELF relo #%d in section #%zu points outside the data of section '%s' in %s\n",
+							j, src_sec->sec_idx,
+							dst_linked_sec->sec_name,
+							obj->filename);
+						return -EINVAL;
+					}
+					val = dst_linked_sec->raw_data + dst_rel->r_offset;
+					if (linker->swapped_endian)
+						*val = bswap_32(bswap_32(*val) + sec->dst_off);
+					else
+						*val += sec->dst_off;
 				} else {
 					pr_warn("relocation against STT_SECTION in non-exec section is not supported!\n");
 					return -EINVAL;
Signature · 1 line
2.53.0-Meta
#42 lore raw

Re: [PATCH bpf-next v5 18/21] libbpf: Let the static linker carry .bpf_cleanup relocations

This isn't a bug, but could this paragraph state what the code does rather than walking through the review reasoning?
Quoted 7 lines

Taking them is keyed on the relocation type rather than on the section name, so any non-executable section could reach the new arm, and what it replaces is an unconditional refusal. Check what the refusal used to make
unnecessary: a relocated section may be SHT_NOBITS, which extend_sec() leaves with no raw_data, and r_offset is checked for alignment only where the section holds instructions. Refuse those rather than write through them.

This isn't a bug, but could this paragraph state what the code does rather than walking through the review reasoning?

The paragraph opens with an imperative ("Check what...") and describes the thought process that led to the bounds check. The facts it conveys are
correct: an SHT_NOBITS target has no raw_data, and r_offset alignment is only checked for executable sections. Would it read more like neighbouring changelogs if it stated the conclusion directly, something like: "the unconditional refusal also covered two cases the new arm must handle
itself: an SHT_NOBITS target has no raw_data, and r_offset is only alignment-checked for executable sections, so both are now rejected explicitly"?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35821138073

author reviewer
#43 lore raw

PATCH v5 19/21 selftests/bpf: Add an end-to-end .bpf_cleanup exception test

In reply to Yonghong Song

C has no unwinding, so nothing here comes out of the frontend: the frames that own a resource are written as __naked inline assembly, which spells out by hand exactly what a frontend emits -- a call site bracketed by two labels, a landing pad unreachable in the compiler's CFG, and a .bpf_cleanup record tying them together. The assembler turns ".long <text label>" into the same R_BPF_64_NODYLD32 relocation the BPF AsmPrinter emits, so libbpf and the kernel see an object indistinguishable from a compiler-generated one.

Call chain: entry -> foo1 -> foo1v -> foo2 -> foo3. foo3 holds a non-preemptible section and throws inside it; foo2 holds an RCU read lock and has two call sites sharing one pad, one of them its own throw; foo1v is a void frame whose pad ends in a jump to a resume block placed after an unrelated block that ends in a plain exit; foo1 owns nothing and gets no record; entry is the boundary.

There are also some shapes the kernel refuses -- the ones with no correct answer, and the ones a pad running on the bpf_throw() walker's stack cannot express -- plus the return-value rejection that delivering an exception at the boundary of a program type which constrains its return value produces. The test skips rather than fails where the JIT cannot dispatch a landing pad at all.

---
 .../selftests/bpf/exceptions_cleanup.h        |  29 +
 .../bpf/prog_tests/exceptions_cleanup.c       | 128 ++++
 .../selftests/bpf/progs/exceptions_cleanup.c  | 164 +++++
 .../bpf/progs/exceptions_cleanup_fail.c       | 678 ++++++++++++++++++
 4 files changed, 999 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
tools/testing/selftests/bpf/exceptions_cleanup.h new file +29 -0
new file mode 100644
index 000000000000..630d2e207119
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef __EXCEPTIONS_CLEANUP_H__
+#define __EXCEPTIONS_CLEANUP_H__
+
+#define THROW_COOKIE		0x100
+
+/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */
+#define RAN_FOO3_PREEMPT	0x1
+#define RAN_FOO2_RCU		0x2
+#define RAN_FOO1V_PREEMPT	0x4
+#define RAN_FOO2_DROP		0x8
+#define RAN_BUMP		0x10
+
+#define CLEANUP_REC(begin, end, landing_pad)			\
+	".pushsection .bpf_cleanup,\"a\",@progbits;"		\
+	".long " begin ";"					\
+	".long " end ";"					\
+	".long " landing_pad ";"				\
+	".popsection;"
+
+/* Set a bit in @pads_ran. */
+#define PAD_RAN(bit)						\
+	"r1 = %[pads_ran] ll;"					\
+	"r2 = *(u64 *)(r1 + 0);"				\
+	"r2 |= " bit ";"					\
+	"*(u64 *)(r1 + 0) = r2;"
+
+#endif /* __EXCEPTIONS_CLEANUP_H__ */
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +128 -0

128 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup.c new file +164 -0

164 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +678 -0

678 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#44 lore raw

Re: [PATCH bpf-next v5 19/21] selftests/bpf: Add an end-to-end .bpf_cleanup exception test

Does the comment point in the right direction? It says "the one below" but the single supplied record 'rec' is declared roughly twenty lines above this comment, not below it. Should this read…
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +0 -0
new file mode 100644
index 000000000000..d9823023ec04

[ ... ]

Quoted 15 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +14 -0
@@ -0,0 +1,128 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include "exceptions_cleanup.h"
+#include "exceptions_cleanup.skel.h"
+#include "exceptions_cleanup_fail.skel.h"
+
+/* foo3 threw: every frame that has a pad ran it. */
+#define PADS_FOO3_THREW \
+	(RAN_FOO3_PREEMPT | RAN_FOO2_RCU | RAN_FOO1V_PREEMPT | RAN_FOO2_DROP)
+
+/* foo2 threw after foo3 returned normally: foo3's pad must not run. */
+#define PADS_FOO2_THREW \
+	(RAN_FOO2_RCU | RAN_FOO1V_PREEMPT | RAN_FOO2_DROP)

[ ... ]

Quoted 29 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +29 -0
+/*
+ * A table with more records than the program has instructions: no valid one
+ * can look like that, and it is refused before the kernel allocates for it.
+ */
+static void test_cleanup_info_cnt(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_MOV64_IMM(BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	struct bpf_cleanup_info rec = {
+		.begin_off = 0,
+		.end_off = 1,
+		.landing_pad_off = 1,
+	};
+	char log[512] = {};
+
+	LIBBPF_OPTS(bpf_prog_load_opts, opts,
+		    .log_buf = log,
+		    .log_size = sizeof(log),
+		    .log_level = 1,
+		    .cleanup_info = &rec,
+		    .cleanup_info_cnt = 1 << 20,
+		    .cleanup_info_rec_size = sizeof(rec));
+	int fd;
+
+	fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL",
+			   insns, ARRAY_SIZE(insns), &opts);
+	if (!ASSERT_LT(fd, 0, "load")) {

tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c line 77

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +10 -0
+		close(fd);
+		return;
+	}
+	/*
+	 * Turned away on the count, rather than on whatever the records past
+	 * the one below happen to hold.
+	 */
+	ASSERT_HAS_SUBSTR(log, "cleanup info has 1048576 records for 2 instructions",
+			  "log");
+}

Does the comment point in the right direction? It says "the one below" but the single supplied record 'rec' is declared roughly twenty lines above this comment, not below it. Should this read "above" instead of "below", or name 'rec' directly?

The substance is correct - check_cleanup_info() rejects on the count before reading past the first record - only the direction word seems inverted.

Quoted 18 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +18 -0
+
+void test_exceptions_cleanup(void)
+{
+	char log[8192] = {};
+
+	LIBBPF_OPTS(bpf_object_open_opts, opts,
+		    .kernel_log_buf = log,
+		    .kernel_log_size = sizeof(log));
+	struct exceptions_cleanup *skel;
+	int err;
+
+	skel = exceptions_cleanup__open_opts(&opts);
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	err = exceptions_cleanup__load(skel);
+	if (err) {
+		if (err == -EOPNOTSUPP &&

tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c line 105

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +10 -0
+		    strstr(log, "exception cleanup needs a JIT that can dispatch landing pads")) {
+			printf("%s:SKIP:JIT cannot dispatch exception cleanup landing pads\n",
+			       __func__);
+			test__skip();
+		} else if (!ASSERT_OK(err, "load")) {
+			fprintf(stderr, "%s", log);
+		}
+		exceptions_cleanup__destroy(skel);
+		return;
+	}

Is the cleanup_info_cnt subtest reachable on architectures whose JIT cannot dispatch landing pads?

The skip path here returns immediately after destroying the skeleton, which means the code after this block never runs, including:

    if (test__start_subtest("cleanup_info_cnt"))
        test_cleanup_info_cnt();

bpf_jit_supports_cleanup_pads() is only implemented on x86 and arm64; everywhere else the __weak stub returns false. When that happens exceptions_cleanup__load() returns -EOPNOTSUPP, the test calls test__skip() and returns early, skipping everything below.

test_cleanup_info_cnt() only exercises the UAPI count check in check_cleanup_info(), which runs from bpf_check_btf_info() before bpf_prepare_cleanup_exceptions() and therefore never reaches the JIT gate. It loads its own 2-instruction BPF_PROG_TYPE_SOCKET_FILTER program and would pass on any architecture.

Could test_cleanup_info_cnt() run before the skeleton load, or could the skip path fall through to it, so the UAPI validation stays covered on non-x86/arm64 builds?

Quoted 13 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +13 -0
+
+	/* No throw: foo3 returns 1 ^ 1 == 0, foo2 adds one, no pad runs. */
+	if (test__start_subtest("no_throw"))
+		run(skel, 1, 1, 0);
+
+	/* foo3 throws; every pad runs and the cookie is delivered at entry. */
+	if (test__start_subtest("throw_from_foo3"))
+		run(skel, 101, THROW_COOKIE, PADS_FOO3_THREW);
+
+	/*
+	 * foo3 returns 2 ^ 1 == 3, so foo2 throws from its own second region;
+	 * foo3's frame is long gone, so its pad must not run.
+	 */

tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c line 128

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +10 -0
+	if (test__start_subtest("throw_from_foo2"))
+		run(skel, 2, THROW_COOKIE, PADS_FOO2_THREW);
+
+	exceptions_cleanup__destroy(skel);
+
+	RUN_TESTS(exceptions_cleanup_fail);
+
+	if (test__start_subtest("cleanup_info_cnt"))
+		test_cleanup_info_cnt();
+}

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35821138073

author reviewer
#45 lore raw

PATCH v5 20/21 selftests/bpf: Cover the exception cleanup shapes the chain does not reach

In reply to Yonghong Song

The end-to-end test walks one call chain with a pad in most of its frames. This adds the shapes that chain does not reach, numbered in the new file in the order they are listed here:

- what a cleanup table leaves dead
- a callee called from both a covered and an uncovered site
- a pad that reads its frame's callee-saved registers
- a tail-call-reachable callee
- a pad in the main program's own frame
- a tail call that is taken
- an extension standing in for a covered call
- a throwing subprogram named by a BPF_PSEUDO_FUNC
- a record covering bpf_throw() itself
- a pad that calls a subprogram an extension can replace
- a covered bpf_throw() the sweep leaves last
- a region ending on a 16-byte instruction
- a pad that reloads from and writes to its own frame
- a pad two frames up
- a record covering only a nounwind call
- a pad terminated by _Unwind_Resume rather than bpf_unwind_resume
- a pad that calls a subprogram which tail calls
- the tail-call target carrying a table of its own
- a pad whose first instruction is a nop
- a pad that indexes its frame by a register the frame set before the throwing call
- the same over a throwing global subprogram
- a pad that indexes its frame by r0, which no instruction wrote

One more shape needs an object of its own: exceptions_cleanup_ext_table.c is an extension carrying a cleanup table of its own, attached over the target the extension shape above already provides. Shapes 7 and 10 bring small extension programs of their own.

---
 .../selftests/bpf/exceptions_cleanup.h        |   24 +
 .../bpf/prog_tests/exceptions_cleanup.c       |  368 ++++++
 .../bpf/progs/exceptions_cleanup_ext_table.c  |   48 +
 .../bpf/progs/exceptions_cleanup_freplace.c   |   17 +
 .../progs/exceptions_cleanup_pad_freplace.c   |   17 +
 .../bpf/progs/exceptions_cleanup_shapes.c     | 1049 +++++++++++++++++
 6 files changed, 1523 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
tools/testing/selftests/bpf/exceptions_cleanup.h +24 -0
index 630d2e207119..d6f63f147c4a 100644
@@ -4,6 +4,7 @@
 #define __EXCEPTIONS_CLEANUP_H__
 
 #define THROW_COOKIE		0x100
+#define INNER_COOKIE		0x200
 
 /* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */
 #define RAN_FOO3_PREEMPT	0x1
@@ -12,6 +13,29 @@
 #define RAN_FOO2_DROP		0x8
 #define RAN_BUMP		0x10
 
+/* progs/exceptions_cleanup_shapes.c: one bit per shape, numbered its own way. */
+#define RAN_SWEEP		0x1
+#define RAN_SHARED		0x2
+#define RAN_REGS		0x4
+#define RAN_TAIL_CALL		0x8
+#define RAN_MAIN_PAD		0x10
+#define RAN_TC_TAKEN		0x20
+#define RAN_FREPLACE		0x40
+#define RAN_ADDR_TAKEN		0x80
+#define RAN_NO_SUBPROG		0x100
+#define RAN_PAD_CALLS		0x200
+#define RAN_PAD_FIRST		0x400
+#define RAN_WIDE_REC		0x800
+#define RAN_PAD_STACK		0x1000
+#define RAN_DEEP_PAD		0x2000
+#define RAN_NOUNWIND_REC	0x4000
+#define RAN_RESUME_ALIAS	0x8000
+#define RAN_PAD_TAIL_CALL	0x10000
+#define RAN_NOP_PAD		0x20000
+#define RAN_VAR_STACK		0x40000
+#define RAN_GLOBAL_PAD		0x80000
+#define RAN_PAD_R0		0x100000
+
 #define CLEANUP_REC(begin, end, landing_pad)			\
 	".pushsection .bpf_cleanup,\"a\",@progbits;"		\
 	".long " begin ";"					\
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +368 -0

386 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c new file +48 -0

48 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c new file +17 -0
new file mode 100644
index 000000000000..afb358fd3d40
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "exceptions_cleanup.h"
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */
+int btf_int_anchor;
+
+SEC("freplace/fr_callee")
+__u64 new_fr_callee(__u64 x)
+{
+	bpf_throw(THROW_COOKIE);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c new file +17 -0
new file mode 100644
index 000000000000..eabac6baabb7
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "exceptions_cleanup.h"
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */
+int btf_int_anchor;
+
+SEC("freplace/pad_callee")
+__u64 new_pad_callee(__u64 x)
+{
+	bpf_throw(INNER_COOKIE);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +1049 -0

1,049 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#46 lore raw

Re: [PATCH bpf-next v5 20/21] selftests/bpf: Cover the exception cleanup shapes the chain does not reach

This isn't a bug, but test_pad_calls_freplace() repeats the body of run_shape(): it declares its own ctx and LIBBPF_OPTS(bpf_test_run_opts, topts, ...), sets input and pads_ran, calls…
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +0 -0
index d9823023ec04..dbdb78b791e0 100644

[ ... ]

Quoted 87 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +87 -0
+static void run_shape(struct exceptions_cleanup_shapes *skel, struct bpf_program *prog,
+		      __u64 input, __u32 retval, __u64 pads)
+{
+	__u64 ctx = 0;
+	int err;
+
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		    .ctx_in = &ctx,
+		    .ctx_size_in = sizeof(ctx),
+	);
+
+	skel->bss->input = input;
+	skel->bss->pads_ran = 0;
+
+	err = bpf_prog_test_run_opts(bpf_program__fd(prog), &topts);
+	if (!ASSERT_OK(err, "run"))
+		return;
+	ASSERT_EQ(topts.retval, retval, "retval");
+	ASSERT_EQ(skel->bss->pads_ran, pads, "pads_ran");
+}
+
+static void test_freplace(struct exceptions_cleanup_shapes *skel)
+{
+	struct exceptions_cleanup_freplace *fr;
+	struct bpf_link *link;
+	int tgt_fd;
+
+	tgt_fd = bpf_program__fd(skel->progs.entry_freplace);
+
+	fr = exceptions_cleanup_freplace__open();
+	if (!ASSERT_OK_PTR(fr, "freplace open"))
+		return;
+
+	if (!ASSERT_OK(bpf_program__set_attach_target(fr->progs.new_fr_callee,
+						      tgt_fd, "fr_callee"),
+		       "set_attach_target"))
+		goto out;
+	if (!ASSERT_OK(exceptions_cleanup_freplace__load(fr), "freplace load"))
+		goto out;
+
+	link = bpf_program__attach_freplace(fr->progs.new_fr_callee, tgt_fd,
+					    "fr_callee");
+	if (!ASSERT_OK_PTR(link, "attach_freplace"))
+		goto out;
+
+	run_shape(skel, skel->progs.entry_freplace, 101, THROW_COOKIE, 0);
+	bpf_link__destroy(link);
+out:
+	exceptions_cleanup_freplace__destroy(fr);
+}
+
+static void test_pad_calls_freplace(struct exceptions_cleanup_shapes *skel)
+{
+	struct exceptions_cleanup_pad_freplace *fr;
+	struct bpf_link *link;
+	__u64 ctx = 0;
+	int tgt_fd, err;
+
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		    .ctx_in = &ctx,
+		    .ctx_size_in = sizeof(ctx),
+	);
+
+	tgt_fd = bpf_program__fd(skel->progs.entry_pad_calls);
+
+	fr = exceptions_cleanup_pad_freplace__open();
+	if (!ASSERT_OK_PTR(fr, "pad freplace open"))
+		return;
+
+	if (!ASSERT_OK(bpf_program__set_attach_target(fr->progs.new_pad_callee,
+						      tgt_fd, "pad_callee"),
+		       "set_attach_target"))
+		goto out;
+	if (!ASSERT_OK(exceptions_cleanup_pad_freplace__load(fr), "pad freplace load"))
+		goto out;
+
+	link = bpf_program__attach_freplace(fr->progs.new_pad_callee, tgt_fd,
+					    "pad_callee");
+	if (!ASSERT_OK_PTR(link, "attach_freplace"))
+		goto out;
+
+	skel->bss->input = 101;
+	skel->bss->pads_ran = 0;
+	skel->bss->pad_runs = 0;
+
+	err = bpf_prog_test_run_opts(tgt_fd, &topts);
+	if (!ASSERT_OK(err, "run"))

tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 139

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +10 -0
+		goto out_link;
+
+	ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+	ASSERT_EQ(skel->bss->pads_ran, RAN_PAD_CALLS, "pads_ran");
+	ASSERT_EQ(topts.retval, THROW_COOKIE, "retval");
+out_link:
+	bpf_link__destroy(link);
+out:
+	exceptions_cleanup_pad_freplace__destroy(fr);
+}

This isn't a bug, but test_pad_calls_freplace() repeats the body of run_shape(): it declares its own ctx and LIBBPF_OPTS(bpf_test_run_opts, topts, ...), sets input and pads_ran, calls bpf_prog_test_run_opts(), and asserts retval and pads_ran.

test_ext_table() gets its extra ext_pad_ran assertion by wrapping run_shape(); could test_pad_calls_freplace() use run_shape() the same way and keep only the pad_runs reset and assertion around it, instead of open-coding the run?

[ ... ]

Quoted 37 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c new file +17 -0
new file mode 100644
index 000000000000..afb358fd3d40
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "exceptions_cleanup.h"
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */
+int btf_int_anchor;
+
+SEC("freplace/fr_callee")
+__u64 new_fr_callee(__u64 x)
+{
+	bpf_throw(THROW_COOKIE);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c new file +7 -0
new file mode 100644
index 000000000000..eabac6baabb7
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "exceptions_cleanup.h"
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */

tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c line 17

Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c new file +10 -0
+int btf_int_anchor;
+
+SEC("freplace/pad_callee")
+__u64 new_pad_callee(__u64 x)
+{
+	bpf_throw(INNER_COOKIE);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";

This isn't a bug, but these two files are nearly identical, differing only in the SEC() target (pad_callee vs fr_callee), the program name (new_pad_callee vs new_fr_callee) and the cookie (INNER_COOKIE vs THROW_COOKIE).

Would the SEC("?freplace") + bpf_program__set_autoload() idiom that exceptions_ext.c uses let both replacements (and one copy of the btf_int_anchor note) live in a single object?

[ ... ]

Quoted 5 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +0 -0
new file mode 100644
index 000000000000..bea2c8e95db3

[ ... ]

Quoted 11 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +11 -0
+static __used __naked __noinline __u64 regs_frame(void)
+{
+	asm volatile (
+	LOAD_MAGIC_REGS
+	"call bpf_preempt_disable;"
+"1:"	"call regs_thrower;"		/* cleanup region */
+"2:"
+	"call bpf_preempt_enable;"
+	"r0 = 0;"
+	"exit;"
+"3:"					/* landing pad */

tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 195

Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +10 -0
+	"call bpf_preempt_enable;"
+	CHECK_MAGIC_REGS("%[ran]")
+	"call bpf_unwind_resume;"
+	"exit;"
+	CLEANUP_REC("1b", "2b", "3b")
+	:
+	: [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_REGS),
+	  __imm_addr(magic), __imm_addr(pads_ran)
+	: __clobber_all);
+}

This isn't a bug, but regs_frame()'s asm operand list declares [cookie]"i"(THROW_COOKIE), yet the template never references %[cookie]. The cookie is supplied by regs_thrower() one frame down, which has its own [cookie] operand.

Is the [cookie] operand in regs_frame() still needed now that regs_thrower() supplies the cookie, or is it a leftover from the split?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35821138073

maintainer
#47 lore raw

Re: [PATCH bpf-next v5 20/21] selftests/bpf: Cover the exception cleanup shapes the chain does not reach

In reply to Yonghong Song

On Tue, 2026-09-22 at 22:00 -0700, Yonghong Song wrote:

...

tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 175

Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +2 -0
+static void test_shapes(void)
+{

Maybe consider adding an annotation __retbss(var, value)? Similar to __retval(), but checking the value of a variable in .bss after the test execution?
Similar to how veristat does it, it can:
- look for a map with a name ".bss" and enumerate
- look for a btf_is_datasec() with a name ".bss"
- iterate over variables in the section and look
for one with a name specified in the annotation
- read the value if it is e.g. u32
(we can limit support by u32 or u64 at the moment).

This would remove the necessity for test_shapes(), as generic RUN_TESTS() would suffice.

But also, Yonghong, this series adds 2.5K lines of tests, are you sure all these tests are necessary?
Is there a more compact way to encode these?

Quoted 229 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +229 -0
+	struct exceptions_cleanup_shapes *skel;
+
+	skel = exceptions_cleanup_shapes__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "shapes open_and_load"))
+		return;
+
+	/* The frame loads at all only if everything unreachable in it went. */
+	if (test__start_subtest("sweep_no_throw"))
+		run_shape(skel, skel->progs.entry_sweep, 1, 0, 0);
+	if (test__start_subtest("sweep_throw"))
+		run_shape(skel, skel->progs.entry_sweep, 101, THROW_COOKIE, RAN_SWEEP);
+
+	/* The covered call unwinds to the pad; the uncovered one walks past. */
+	if (test__start_subtest("shared_callee_no_throw")) {
+		skel->bss->outer_input = 0;
+		run_shape(skel, skel->progs.entry_shared, 1, 2, 0);
+	}
+	if (test__start_subtest("shared_callee_throw")) {
+		skel->bss->outer_input = 0;
+		run_shape(skel, skel->progs.entry_shared, 101, THROW_COOKIE, RAN_SHARED);
+	}
+	if (test__start_subtest("shared_callee_uncovered_throw")) {
+		skel->bss->outer_input = 101;
+		run_shape(skel, skel->progs.entry_shared, 1, THROW_COOKIE, 0);
+		skel->bss->outer_input = 0;
+	}
+
+	/* The pad only sets its bit if it got the frame's own r6-r9 back. */
+	if (test__start_subtest("pad_sees_callee_saved"))
+		run_shape(skel, skel->progs.entry_regs, 101, THROW_COOKIE, RAN_REGS);
+
+	/* Same check, with a tail-call-reachable callee: its spill moves. */
+	if (test__start_subtest("tail_call_no_throw"))
+		run_shape(skel, skel->progs.entry_tail_call, 1, 0, 0);
+	if (test__start_subtest("tail_call_throw"))
+		run_shape(skel, skel->progs.entry_tail_call, 101, THROW_COOKIE,
+			  RAN_TAIL_CALL);
+
+	/* A region around a nounwind call: no pad dispatched, still loads. */
+	if (test__start_subtest("nounwind_region"))
+		run_shape(skel, skel->progs.entry_nounwind_rec, 1, 0, 0);
+
+	/* A pad in the main program's own frame, not in a subprogram. */
+	if (test__start_subtest("main_program_pad"))
+		run_shape(skel, skel->progs.entry_main_pad, 101, THROW_COOKIE,
+			  RAN_MAIN_PAD);
+
+	/*
+	 * The same call site either way: the subprogram's throw unwinds into
+	 * this frame and runs its pad, an extension's stops at its own boundary.
+	 */
+	if (test__start_subtest("freplace_subprog_throws"))
+		run_shape(skel, skel->progs.entry_freplace, 7, THROW_COOKIE,
+			  RAN_FREPLACE);
+	if (test__start_subtest("freplace_extension_throws"))
+		test_freplace(skel);
+
+	/*
+	 * A tail call that is taken: the walk ends at the target, so the cookie
+	 * comes back from there and this frame's pad does not run -- though it
+	 * is reachable, so a walk past the boundary would find it.
+	 */
+	if (test__start_subtest("tail_call_taken")) {
+		int key = 0, prog_fd = bpf_program__fd(skel->progs.tc_target);
+
+		if (ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.taken_table),
+						  &key, &prog_fd, BPF_ANY),
+			      "populate taken_table"))
+			run_shape(skel, skel->progs.entry_tail_taken, 101,
+				  THROW_COOKIE, 0);
+	}
+
+	/*
+	 * A throwing subprog named by a BPF_PSEUDO_FUNC and handed to a
+	 * bpf_loop() the verifier never reaches: the callback check has to fire
+	 * on the helper call, not on the ld_imm64.
+	 */
+	if (test__start_subtest("addr_taken_no_throw"))
+		run_shape(skel, skel->progs.entry_addr_taken, 1, 2, 0);
+	if (test__start_subtest("addr_taken_throw"))
+		run_shape(skel, skel->progs.entry_addr_taken, 101, THROW_COOKIE,
+			  RAN_ADDR_TAKEN);
+
+	/*
+	 * A record covering bpf_throw() itself rather than a call to a frame
+	 * that throws: raised, caught up with and delivered in one frame.
+	 */
+	if (test__start_subtest("no_subprog_no_throw"))
+		run_shape(skel, skel->progs.entry_no_subprog, 1, 0, 0);
+	if (test__start_subtest("no_subprog_throw"))
+		run_shape(skel, skel->progs.entry_no_subprog, 101, THROW_COOKIE,
+			  RAN_NO_SUBPROG);
+
+	/*
+	 * A pad that calls a subprogram; with a throwing extension in its place,
+	 * the nested exception has to stop there, not restart this pad.
+	 */
+	if (test__start_subtest("pad_calls_subprog")) {
+		skel->bss->pad_runs = 0;
+		run_shape(skel, skel->progs.entry_pad_calls, 101, THROW_COOKIE,
+			  RAN_PAD_CALLS);
+		ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+	}
+	if (test__start_subtest("pad_calls_throwing_extension"))
+		test_pad_calls_freplace(skel);
+
+	/*
+	 * A covered throw the sweep leaves last, where the default exception
+	 * callback is patched in; the pad's bit needs r6-r9 still spilled.
+	 */
+	if (test__start_subtest("pad_before_throw"))
+		run_shape(skel, skel->progs.entry_pad_first, 101, THROW_COOKIE,
+			  RAN_PAD_FIRST);
+
+	/*
+	 * A region whose last instruction is a 16-byte one, so that end - 1
+	 * names the half of it that is not an instruction.
+	 */
+	if (test__start_subtest("region_ends_on_ldimm64"))
+		run_shape(skel, skel->progs.entry_wide_rec, 101, THROW_COOKIE,
+			  RAN_WIDE_REC);
+
+	/*
+	 * A pad that reloads from and writes to its own frame's stack, which a
+	 * JIT addressing the frame through the stack pointer gets wrong.
+	 */
+	if (test__start_subtest("pad_uses_own_frame"))
+		run_shape(skel, skel->progs.entry_pad_stack, 101, THROW_COOKIE,
+			  RAN_PAD_STACK);
+
+	/* The same, with an uncovered frame between the throw and the pad. */
+	if (test__start_subtest("pad_two_frames_up"))
+		run_shape(skel, skel->progs.entry_deep_pad, 101, THROW_COOKIE,
+			  RAN_DEEP_PAD);
+
+	/* An extension program with a cleanup table of its own. */
+	if (test__start_subtest("extension_carries_table"))
+		test_ext_table(skel);
+
+	/*
+	 * A pad terminated by _Unwind_Resume, which libbpf maps onto the kfunc;
+	 * every other program here calls bpf_unwind_resume directly.
+	 */
+	if (test__start_subtest("resume_alias"))
+		run_shape(skel, skel->progs.entry_resume_alias, 101,
+			  THROW_COOKIE, RAN_RESUME_ALIAS);
+
+	/*
+	 * A pad that calls a subprogram which tail calls, array empty and then
+	 * populated: the tail call releases only the callee's own prologue.
+	 */
+	if (test__start_subtest("pad_callee_tail_call")) {
+		int key = 0, prog_fd = bpf_program__fd(skel->progs.pad_tc_target);
+
+		skel->bss->pad_tc_target_ran = 0;
+		skel->bss->pad_runs = 0;
+		run_shape(skel, skel->progs.entry_pad_tail_call, 101,
+			  THROW_COOKIE, RAN_PAD_TAIL_CALL);
+		ASSERT_EQ(skel->bss->pad_tc_target_ran, 0, "target not run");
+		ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+
+		if (ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.pad_tc_table),
+						  &key, &prog_fd, BPF_ANY),
+			      "populate pad_tc_table")) {
+			skel->bss->pad_runs = 0;
+			run_shape(skel, skel->progs.entry_pad_tail_call, 101,
+				  THROW_COOKIE, RAN_PAD_TAIL_CALL);
+			ASSERT_EQ(skel->bss->pad_tc_target_ran, 1, "target ran");
+			ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+		}
+	}
+
+	/*
+	 * The same, into a target that carries a table and throws: that target
+	 * is a boundary, so the outer pad runs once, not twice.
+	 */
+	if (test__start_subtest("pad_callee_tail_call_throws")) {
+		int key = 0, prog_fd = bpf_program__fd(skel->progs.pad_tc_throw_target);
+
+		if (ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.pad_tc_table),
+						  &key, &prog_fd, BPF_ANY),
+			      "populate pad_tc_table")) {
+			skel->bss->pad_runs = 0;
+			skel->bss->tc_target_pad_runs = 0;
+			run_shape(skel, skel->progs.entry_pad_tail_call, 101,
+				  THROW_COOKIE, RAN_PAD_TAIL_CALL);
+			/* The target cleaned up after itself, once. */
+			ASSERT_EQ(skel->bss->tc_target_pad_runs, 1,
+				  "tc_target_pad_runs");
+			/* And the outer pad was not started over. */
+			ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+		}
+	}
+
+	/*
+	 * A pad whose first instruction opt_remove_nops() deletes: the record
+	 * has to follow the pad rather than be dropped with the nop.
+	 */
+	if (test__start_subtest("nop_at_pad_head"))
+		run_shape(skel, skel->progs.entry_nop_pad, 101, THROW_COOKIE,
+			  RAN_NOP_PAD);
+
+	/*
+	 * A pad that indexes its frame's stack by a register the frame set
+	 * before the throwing call, marked precise back across the unwind edge.
+	 */
+	if (test__start_subtest("pad_var_stack_offset"))
+		run_shape(skel, skel->progs.entry_var_stack, 101, THROW_COOKIE,
+			  RAN_VAR_STACK);
+
+	/*
+	 * The same, where the throw is in a global subprogram, which the
+	 * verifier walks without a frame of its own.
+	 */
+	if (test__start_subtest("pad_over_global_subprog"))
+		run_shape(skel, skel->progs.entry_global_pad, 101, THROW_COOKIE,
+			  RAN_GLOBAL_PAD);
+
+	/*
+	 * A pad that indexes its frame by r0, which no insn in the frame
+	 * wrote: the precision request for it must not cross the unwind edge.
+	 */
+	if (test__start_subtest("pad_r0_precision"))
+		run_shape(skel, skel->progs.entry_pad_r0, 101, THROW_COOKIE,
+			  RAN_PAD_R0);
+
+	exceptions_cleanup_shapes__destroy(skel);
+}
+

...

author reviewer
#48 lore raw

PATCH v5 21/21 selftests/bpf: Load an exception cleanup program from a light skeleton

In reply to Yonghong Song

For a light skeleton, libbpf hands the records to bpf_gen__prog_load(), which writes them into a blob and emits a loader program that issues BPF_PROG_LOAD from inside the kernel. Nothing about that is shared with the ordinary path: the attr is built field by field, and the kfunc names are resolved by the loader program when it runs.

exceptions_cleanup_light.c is the smallest program that can tell whether a table survives that trip: one record covering the bpf_throw() call itself, one pad that sets a bit and resumes. If the table arrives, the pad runs and the cookie comes back; if it does not, the program does not load at all.

---
 tools/testing/selftests/bpf/Makefile.skel     |  2 +-
 .../selftests/bpf/exceptions_cleanup.h        |  3 ++
 .../bpf/prog_tests/exceptions_cleanup.c       | 28 +++++++++++++
 .../bpf/progs/exceptions_cleanup_light.c      | 39 +++++++++++++++++++
 4 files changed, 71 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c
tools/testing/selftests/bpf/Makefile.skel +1 -1
index 580d1d82c186..6d7518fa4018 100644
@@ -33,7 +33,7 @@ LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h		\
 LSKELS := fexit_sleep.c trace_printk.c trace_vprintk.c map_ptr_kern.c 	\
 	core_kern.c core_kern_overflow.c test_ringbuf.c			\
 	test_ringbuf_n.c test_ringbuf_map_key.c test_ringbuf_write.c    \
-	test_ringbuf_overwrite.c
+	test_ringbuf_overwrite.c exceptions_cleanup_light.c
 
 LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
 
tools/testing/selftests/bpf/exceptions_cleanup.h +3 -0
index d6f63f147c4a..68f2d61a694b 100644
@@ -36,6 +36,9 @@
 #define RAN_GLOBAL_PAD		0x80000
 #define RAN_PAD_R0		0x100000
 
+/* progs/exceptions_cleanup_light.c: the one pad it has. */
+#define RAN_LIGHT		0x1
+
 #define CLEANUP_REC(begin, end, landing_pad)			\
 	".pushsection .bpf_cleanup,\"a\",@progbits;"		\
 	".long " begin ";"					\
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +28 -0
index dbdb78b791e0..323ee8d72c91 100644
@@ -8,6 +8,7 @@
 #include "exceptions_cleanup_freplace.skel.h"
 #include "exceptions_cleanup_pad_freplace.skel.h"
 #include "exceptions_cleanup_ext_table.skel.h"
+#include "exceptions_cleanup_light.lskel.h"
 
 /* foo3 threw: every frame that has a pad ran it. */
 #define PADS_FOO3_THREW \
@@ -171,6 +172,30 @@ static void test_ext_table(struct exceptions_cleanup_shapes *skel)
 	exceptions_cleanup_ext_table__destroy(fr);
 }
 
+static void test_light_skeleton(void)
+{
+	struct exceptions_cleanup_light_lskel *skel;
+	__u64 ctx = 0;
+	int err;
+
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		    .ctx_in = &ctx,
+		    .ctx_size_in = sizeof(ctx),
+	);
+
+	skel = exceptions_cleanup_light_lskel__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "light open_and_load"))
+		return;
+
+	err = bpf_prog_test_run_opts(skel->progs.entry_light.prog_fd, &topts);
+	if (!ASSERT_OK(err, "run"))
+		goto out;
+	ASSERT_EQ(topts.retval, THROW_COOKIE, "retval");
+	ASSERT_EQ(skel->bss->pads_ran, RAN_LIGHT, "pads_ran");
+out:
+	exceptions_cleanup_light_lskel__destroy(skel);
+}
+
 static void test_shapes(void)
 {
 	struct exceptions_cleanup_shapes *skel;
@@ -487,6 +512,9 @@ void test_exceptions_cleanup(void)
 
 	exceptions_cleanup__destroy(skel);
 
+	if (test__start_subtest("light_skeleton"))
+		test_light_skeleton();
+
 	test_shapes();
 
 	RUN_TESTS(exceptions_cleanup_fail);
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +39 -0
new file mode 100644
index 000000000000..de7ae478d67f
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "exceptions_cleanup.h"
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+	bpf_throw(0);
+	bpf_preempt_disable();
+	bpf_preempt_enable();
+	bpf_unwind_resume();
+}
+
+__u64 pads_ran = 0;
+
+SEC("syscall")
+__naked int entry_light(void)
+{
+	asm volatile (
+	"call bpf_preempt_disable;"
+	"r1 = %[cookie];"
+"1:"	"call bpf_throw;"		/* cleanup region */
+"2:"
+	"exit;"
+"3:"					/* landing pad */
+	"call bpf_preempt_enable;"
+	PAD_RAN("%[ran]")
+	"call bpf_unwind_resume;"
+	"exit;"
+	CLEANUP_REC("1b", "2b", "3b")
+	:
+	: [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_LIGHT),
+	  __imm_addr(pads_ran)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
Signature · 1 line
2.53.0-Meta