Skip to contentSkip to conversation

bpf: Run exception cleanup landing pads when bpf_throw() unwinds

PATCH Superseded Yonghong Song 4 participants 40 messages Sep 17 – Sep 19

Subsystems (9)
x86 Kselftest BPF arm64 +5 more subsystems

v1 of 12 Superseded Latest is v12 →

lore
Share & tools
lore

Yonghong Song

author reviewer
#0 lore raw

cover bpf: Run exception cleanup landing pads when bpf_throw() unwinds

bpf_throw() walks the BPF call stack to the exception boundary and discards every frame in between. A frame that owns something -- an RCU read lock, a preemption-disabled section, a referenced kptr -- never gets to give it back, so the verifier refuses to let such a frame throw at all. That is the whole reason a Rust program cannot use bpf_throw() as its panic path today: Rust's Drop glue *is* that give-back, and there is nowhere to run it.

LLVM 23 added the compiler half ([1]). A Rust function that owns a value across a call that can unwind

    fn foo() {
        let _guard = RcuReadGuard::new();  /* bpf_rcu_read_lock()   */
        may_throw();                       /* extern "C-unwind"     */
    }                                      /* Drop: rcu_read_unlock */

lowers to an invoke with a cleanup landing pad holding the Drop call, and the BPF backend writes one record per invoke region into a .bpf_cleanup
section: a flat table of 12-byte (begin, end, landing_pad) triples, each field a byte offset into the code section. The rule is "if a frame unwinds with its return address in [begin, end), run landing_pad before discarding it". A pad ends with a call to _Unwind_Resume(), which the kernel provides as the bpf_unwind_resume() kfunc.

This series is the kernel half: take that table at BPF_PROG_LOAD, teach the verifier that a covered call can also go to its landing pad, and have bpf_throw() run the pads as it walks.

C has no unwinding, so the selftests spell out by hand what a frontend emits -- a call site bracketed by two labels, a landing pad, and a record tying them together. The frame above, written that way:

"call bpf_rcu_read_lock;"
"1:" "call foo3;" /* cleanup region */
"2:"
... normal path, ends in bpf_rcu_read_unlock ...
"6:" /* landing pad */
"call bpf_rcu_read_unlock;"
"call bpf_unwind_resume;" CLEANUP_REC("1b", "2b", "6b")

Today that program does not load: the throw inside foo3 is reported as "bpf_throw cannot be used inside bpf_rcu_read_lock-ed region", because as far as the verifier is concerned nothing will ever unlock. With the series the verifier walks the unwind the same way the run time will -- into the pad, which unlocks, then on to the next frame -- and the program both loads and releases the lock when it throws.

Design
======

A pad is run, not lowered. bpf_throw() already walks the frames with arch_bpf_stack_walk(); it now looks each frame's return address up in that (sub)program's table and calls the pad as a subroutine of the walker, with the unwinding frame's frame pointer and its callee-saved registers restored from the spill its callee's prologue left. The pad therefore sees its own frame but runs on the walker's stack, far below it, so nothing it calls can disturb the frame it is cleaning up after. The JIT turns its bpf_unwind_resume() into the way back to the walker.

The verifier walks the same thing, step for step, so the resource rules are unchanged: whatever a pad releases is released in the verifier state too, and check_resource_leak() simply moves from "a throw was seen" to the end of the walk.

1-2 uapi: cleanup_info in BPF_PROG_LOAD, struct bpf_cleanup_info,
and the bpf_unwind_resume() kfunc
3-9 verifier: mark the covered call sites, give them an edge to the
pad, walk the unwind, and refuse the shapes that cannot be dispatched

  10     bpf_throw(): dispatch pads while walking
  11-12  x86-64 and arm64 JITs
  13-17  libbpf: collect .bpf_cleanup, pass it to the kernel, resolve

_Unwind_Resume, carry it through the light skeleton and the static linker
18-20 selftests

Limitations
===========

- Cleanup pads only. A catch pad -- one that ends in a plain exit rather than a resume, which is what Rust's catch_unwind would need -- is
refused: the walker calls a pad as a subroutine and cannot hand a frame back its own execution. LLVM refuses type-specific catches and filters on its side as well.
- A JIT that can dispatch pads is required: x86-64 (with CONFIG_UNWINDER_ORC, which bpf_throw() already needs there) and arm64. Anywhere else the load fails with -EOPNOTSUPP rather than silently doing nothing.
- No offloaded programs, no private stack, and no combining a table with an exception callback.
- In a pad body: no tail call, no indirect jump, and no on-stack call arguments -- all of them read or write a stack the pad does not own.
- The Rust toolchain does not properly support BPF exception handling yet. The tables the selftests use are hand-written inline asm, which the assembler turns into the same relocations the BPF AsmPrinter emits, so libbpf and the kernel see an object indistinguishable from a compiler-generated one.

[1] https://github.com/llvm/llvm-project/pull/192164
llvm commit 9d51c891b719 ("[BPF] Add exception handling support with .bpf_cleanup section")

Yonghong Song (20):
bpf: Accept the compiler's exception cleanup table at program load
bpf: Add the bpf_unwind_resume() kfunc
bpf: Add lookups for exception cleanup resumes and landing pads
bpf: Mark the call sites an exception cleanup table covers
bpf: Make exception landing pads reachable in the CFG
bpf: Explore the landing pads no call site reaches
bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch
bpf: Walk the exception unwind in the verifier
bpf: Refuse a private stack for a program with an exception cleanup
table
bpf: Dispatch exception cleanup pads from bpf_throw()
bpf, x86: Dispatch exception cleanup pads at run time
bpf, arm64: Dispatch exception cleanup pads at run time
libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
libbpf: Add cleanup_info to bpf_prog_load_opts
libbpf: Collect .bpf_cleanup records and pass them to the kernel
libbpf: Carry the exception cleanup table through the light skeleton
libbpf: Let the static linker carry .bpf_cleanup relocations selftests/bpf: Add an end-to-end .bpf_cleanup exception test selftests/bpf: Cover the exception cleanup shapes the chain does not
reach
selftests/bpf: Load an exception cleanup program from a light skeleton

 arch/arm64/net/Makefile                       |   2 +-
 arch/arm64/net/bpf_cleanup_pad.S              |  95 ++
 arch/arm64/net/bpf_jit_comp.c                 |  97 +-
 arch/x86/net/Makefile                         |   2 +-
 arch/x86/net/bpf_cleanup_pad.S                |  74 ++
 arch/x86/net/bpf_jit_comp.c                   |  78 +-
 include/linux/bpf.h                           |  75 ++
 include/linux/bpf_cleanup_abi.h               |  16 +
 include/linux/bpf_verifier.h                  |  11 +
 include/linux/filter.h                        |   2 +
 include/uapi/linux/bpf.h                      |   9 +
 kernel/bpf/Makefile                           |   2 +-
 kernel/bpf/cfg.c                              |  64 +-
 kernel/bpf/check_btf.c                        | 147 +++
 kernel/bpf/core.c                             |  35 +-
 kernel/bpf/exception.c                        | 654 +++++++++++++
 kernel/bpf/exception.h                        |  22 +
 kernel/bpf/fixups.c                           | 143 +++
 kernel/bpf/helpers.c                          |  46 +
 kernel/bpf/liveness.c                         |  20 +
 kernel/bpf/states.c                           |   3 +
 kernel/bpf/syscall.c                          |   2 +-
 kernel/bpf/verifier.c                         | 125 ++-
 tools/include/uapi/linux/bpf.h                |   9 +
 tools/lib/bpf/bpf.c                           |   6 +-
 tools/lib/bpf/bpf.h                           |   7 +-
 tools/lib/bpf/gen_loader.c                    |  27 +-
 tools/lib/bpf/libbpf.c                        | 305 ++++++-
 tools/lib/bpf/libbpf_internal.h               |  10 +
 tools/lib/bpf/linker.c                        |  19 +-
 tools/testing/selftests/bpf/Makefile          |   2 +-
 .../selftests/bpf/exceptions_cleanup.h        |  52 ++
 .../bpf/prog_tests/exceptions_cleanup.c       | 421 +++++++++
 .../selftests/bpf/progs/exceptions_cleanup.c  | 152 +++
 .../bpf/progs/exceptions_cleanup_ext_table.c  |  48 +
 .../bpf/progs/exceptions_cleanup_fail.c       | 600 ++++++++++++
 .../bpf/progs/exceptions_cleanup_freplace.c   |  17 +
 .../bpf/progs/exceptions_cleanup_light.c      |  39 +
 .../progs/exceptions_cleanup_pad_freplace.c   |  17 +
 .../bpf/progs/exceptions_cleanup_shapes.c     | 863 ++++++++++++++++++
 40 files changed, 4254 insertions(+), 64 deletions(-)
 create mode 100644 arch/arm64/net/bpf_cleanup_pad.S
 create mode 100644 arch/x86/net/bpf_cleanup_pad.S
 create mode 100644 include/linux/bpf_cleanup_abi.h
 create mode 100644 kernel/bpf/exception.c
 create mode 100644 kernel/bpf/exception.h
 create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
Signature · 1 line
2.53.0-Meta
author reviewer
#1 lore raw

PATCH 1/20 bpf: Accept the compiler's exception cleanup table at program load

LLVM 23 added exception handling support for BPF with the .bpf_cleanup section ([1]). Rust code compiled with panic=unwind runs cleanup code (Drop glue) when bpf_throw() fires, and the LLVM BPF backend emits that section from the landing pads the frontend produced. Plain C cannot generate .bpf_cleanup unless inline asm is used. The Rust compiler does not *properly* support BPF exception handling yet, but the kernel can support the table today, and inline assembly is enough to test it.

Add the UAPI to carry the .bpf_cleanup table into the kernel. BPF_PROG_LOAD grows cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, and struct bpf_cleanup_info describes one record as a triple of instruction indices:
the half-open call-site range [begin_off, end_off) and the landing_pad_off the frame resumes at. check_cleanup_info() validates the table a program is loaded with, so the rest of the kernel can rely on it.

[1] https://github.com/llvm/llvm-project/pull/192164

---
 include/linux/bpf_verifier.h   |   2 +
 include/uapi/linux/bpf.h       |   9 ++
 kernel/bpf/check_btf.c         | 147 +++++++++++++++++++++++++++++++++
 kernel/bpf/syscall.c           |   2 +-
 kernel/bpf/verifier.c          |   1 +
 tools/include/uapi/linux/bpf.h |   9 ++
 6 files changed, 169 insertions(+), 1 deletion(-)
include/linux/bpf_verifier.h +2 -0
index cf85141ea167..c08505b9ba82 100644
@@ -987,6 +987,8 @@ struct bpf_verifier_env {
 	struct arg_track **callsite_at_stack;
 	u32 pass_cnt; /* number of times do_check() was called */
 	u32 subprog_cnt;
+	struct bpf_cleanup_info *cleanup_info;
+	u32 cleanup_info_cnt;
 	/* number of instructions analyzed by the verifier */
 	u32 prev_insn_processed, insn_processed;
 	/* number of jmps, calls, exits analyzed so far */
include/uapi/linux/bpf.h +9 -0
index 732b35cc08d1..f7dc121be094 100644
@@ -1669,6 +1669,9 @@ union bpf_attr {
 		 * verification.
 		 */
 		__s32		keyring_id;
+		__aligned_u64	cleanup_info;	/* exception cleanup table */
+		__u32		cleanup_info_rec_size; /* userspace bpf_cleanup_info size */
+		__u32		cleanup_info_cnt; /* number of bpf_cleanup_info records */
 	};
 
 	struct { /* anonymous struct used by BPF_OBJ_* commands */
@@ -7588,6 +7591,12 @@ struct bpf_line_info {
 	__u32	line_col;
 };
 
+struct bpf_cleanup_info {
+	__u32	begin_off;
+	__u32	end_off;
+	__u32	landing_pad_off;
+};
+
 struct bpf_spin_lock {
 	__u32	val;
 };
kernel/bpf/check_btf.c +147 -0

159 lines not on this page · raw message

kernel/bpf/syscall.c +1 -1
index def57bddb092..ac7091469766 100644
@@ -2912,7 +2912,7 @@ int __init __used bpf_multi_func(void) { return 0; }
 BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
 
 /* last field in 'union bpf_attr' used by this command */
-#define BPF_PROG_LOAD_LAST_FIELD keyring_id
+#define BPF_PROG_LOAD_LAST_FIELD cleanup_info_cnt
 
 static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
 {
kernel/bpf/verifier.c +1 -0
index 6c6b8d8520cd..c65ff2e326bf 100644
@@ -21845,6 +21845,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	kvfree(env->succ);
 	kvfree(env->gotox_tmp_buf);
 	bpf_diag_free(env);
+	kvfree(env->cleanup_info);
 	kvfree(env);
 	return ret;
 }
tools/include/uapi/linux/bpf.h +9 -0
index 732b35cc08d1..f7dc121be094 100644
@@ -1669,6 +1669,9 @@ union bpf_attr {
 		 * verification.
 		 */
 		__s32		keyring_id;
+		__aligned_u64	cleanup_info;	/* exception cleanup table */
+		__u32		cleanup_info_rec_size; /* userspace bpf_cleanup_info size */
+		__u32		cleanup_info_cnt; /* number of bpf_cleanup_info records */
 	};
 
 	struct { /* anonymous struct used by BPF_OBJ_* commands */
@@ -7588,6 +7591,12 @@ struct bpf_line_info {
 	__u32	line_col;
 };
 
+struct bpf_cleanup_info {
+	__u32	begin_off;
+	__u32	end_off;
+	__u32	landing_pad_off;
+};
+
 struct bpf_spin_lock {
 	__u32	val;
 };
Signature · 1 line
2.53.0-Meta
author reviewer
#2 lore raw

PATCH 2/20 bpf: Add the bpf_unwind_resume() kfunc

In reply to Yonghong Song

A compiler-emitted cleanup landing pad ends with a call to _Unwind_Resume(), which carries the unwind on once that frame's cleanups have run. The kernel provides the same terminator as a kfunc, named bpf_unwind_resume() to keep the 'bpf_' prefix kfunc convention; a later libbpf patch resolves the compiler's name to it.

The body never runs. A JIT emits the way back out of a landing pad in its place.

---
 kernel/bpf/helpers.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
kernel/bpf/helpers.c +12 -0
index 051b6654e57c..31d84de4d318 100644
@@ -3407,6 +3407,17 @@ __bpf_kfunc void bpf_throw(u64 cookie)
 	WARN(1, "A call to BPF exception callback should never return\n");
 }
 
+/*
+ * Terminator of a compiler-emitted cleanup landing pad. The compiler names
+ * this _Unwind_Resume, the base unwind ABI's entry point for carrying an
+ * unwind on once a frame's cleanups have run. To match kernel kfunc
+ * convention, the kernel calls it bpf_unwind_resume and libbpf maps the
+ * compiler's name onto it.
+ */
+__bpf_kfunc void bpf_unwind_resume(void)
+{
+}
+
 __bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, unsigned int flags)
 {
 	struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
@@ -4853,6 +4864,7 @@ BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_throw)
+BTF_ID_FLAGS(func, bpf_unwind_resume)
 #ifdef CONFIG_BPF_EVENTS
 BTF_ID_FLAGS(func, bpf_send_signal_task)
 #endif
Signature · 1 line
2.53.0-Meta
author reviewer
#3 lore raw

PATCH 3/20 bpf: Add lookups for exception cleanup resumes and landing pads

In reply to Yonghong Song

Add two new files, exception.h and exception.c, to host the exception handling code. Only a few helpers so far: recognising a call to bpf_unwind_resume(), and asking which landing pad, if any, a call site unwinds to.

The pad of a call site is kept in insn_aux_data, so the two places that move instructions around -- bpf_patch_insn_data() and verifier_remove_insns() -- learn to keep it in step. Nothing sets the mark yet; the next patch does.

---
 include/linux/bpf_verifier.h |  6 ++++++
 kernel/bpf/Makefile          |  2 +-
 kernel/bpf/exception.c       | 36 ++++++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h       | 12 ++++++++++++
 kernel/bpf/fixups.c          | 17 +++++++++++++++++
 5 files changed, 72 insertions(+), 1 deletion(-)
 create mode 100644 kernel/bpf/exception.c
 create mode 100644 kernel/bpf/exception.h
include/linux/bpf_verifier.h +6 -0
index c08505b9ba82..f9bccd3e0f4d 100644
@@ -681,6 +681,11 @@ struct bpf_insn_aux_data {
 	bool needs_zext; /* alu op needs to clear upper bits */
 	bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
 	bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
+	/*
+	 * 1 + the instruction index of the exception cleanup landing pad this
+	 * call site unwinds to, or 0 for none.
+	 */
+	u32 cleanup_pad;
 	bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
 	u8 alu_state; /* used in combination with alu_limit */
 	/* true if STX or LDX instruction is a part of a spill/fill
@@ -1518,6 +1523,7 @@ u32 btf_func_arg_align(const struct btf *btf, const struct btf_type *t);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn);
 int bpf_compute_const_regs(struct bpf_verifier_env *env);
 int bpf_prune_dead_branches(struct bpf_verifier_env *env);
 int bpf_check_cfg(struct bpf_verifier_env *env);
kernel/bpf/Makefile +1 -1
index 9a92c348bbda..af9bc60428ad 100644
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o exception.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
kernel/bpf/exception.c new file +36 -0
new file mode 100644
index 000000000000..3895b9453639
@@ -0,0 +1,36 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
+#include <linux/slab.h>
+#include "exception.h"
+
+#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
+
+enum exc_kfunc {
+	EXC_KF_bpf_unwind_resume,
+};
+
+BTF_ID_LIST(exc_kfunc_list)
+BTF_ID(func, bpf_unwind_resume)
+
+static bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf)
+{
+	return bpf_pseudo_kfunc_call(insn) && insn->off == 0 &&
+	       insn->imm == exc_kfunc_list[kf];
+}
+
+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn)
+{
+	return insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume);
+}
+
+int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx)
+{
+	u32 pad = env->insn_aux_data[idx].cleanup_pad;
+
+	return pad ? (int)pad - 1 : -1;
+}
kernel/bpf/exception.h new file +12 -0
new file mode 100644
index 000000000000..0f2b9624a2ce
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef _LINUX_BPF_EXCEPTION_H
+#define _LINUX_BPF_EXCEPTION_H
+
+#include <linux/types.h>
+
+struct bpf_verifier_env;
+
+int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx);
+
+#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +17 -0
index 2add8001c3ec..82b00fac6bd6 100644
@@ -261,6 +261,11 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 		}
 	}
 
+	if (env->cleanup_info_cnt)
+		for (i = 0; i < prog_len; i++)
+			if (data[i].cleanup_pad > off + 1)
+				data[i].cleanup_pad += cnt - 1;
+
 	/*
 	 * Last slot instruction could be a newly generated
 	 * BPF_ST/BPF_LDX/BPF_STX, systematically mark it for non-stack access
@@ -549,6 +554,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 	struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
 	unsigned int orig_prog_len = env->prog->len;
 	int err;
+	u32 i;
 
 	if (bpf_prog_is_offloaded(env->prog->aux))
 		bpf_prog_offload_remove_insns(env, off, cnt);
@@ -573,6 +579,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 		sizeof(*aux_data) * (orig_prog_len - off - cnt));
 	env->insn_aux_data_len -= cnt;
 
+	if (env->cleanup_info_cnt) {
+		for (i = 0; i < env->insn_aux_data_len; i++) {
+			u32 pad = aux_data[i].cleanup_pad;
+
+			if (pad > off + cnt)
+				aux_data[i].cleanup_pad = pad - cnt;
+			else if (pad > off)
+				aux_data[i].cleanup_pad = 0;
+		}
+	}
+
 	return 0;
 }
 
Signature · 1 line
2.53.0-Meta
author reviewer
#4 lore raw

PATCH 4/20 bpf: Mark the call sites an exception cleanup table covers

In reply to Yonghong Song

Some plumbing work is done before bpf_check_cfg(). More specifically, insn_aux_data records cleanup_throw_site for every bpf_throw(), and cleanup_pad, the landing pad a frame resumes at, for every call within the [begin_off, end_off) range of a cleanup record. Subsequent commits consume both.

bpf_prepare_cleanup_exceptions() runs before bpf_check_cfg(), because what it produces is what the CFG walk consumes. It refuses a table on an offloaded program, on a program whose JIT cannot dispatch landing pads or which the JIT was not asked to compile, and on a program that also installs an exception callback -- two different answers to what runs on the way out. bpf_jit_supports_cleanup_pads() is weak here and says no; the arch patches provide the real ones.

---
 include/linux/bpf_verifier.h |  1 +
 include/linux/filter.h       |  1 +
 kernel/bpf/core.c            |  5 ++++
 kernel/bpf/exception.c       | 55 ++++++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h       |  1 +
 kernel/bpf/verifier.c        |  6 ++++
 6 files changed, 69 insertions(+)
include/linux/bpf_verifier.h +1 -0
index f9bccd3e0f4d..7463e86d15ee 100644
@@ -681,6 +681,7 @@ struct bpf_insn_aux_data {
 	bool needs_zext; /* alu op needs to clear upper bits */
 	bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
 	bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
+	bool cleanup_throw_site; /* call to bpf_throw() */
 	/*
 	 * 1 + the instruction index of the exception cleanup landing pad this
 	 * call site unwinds to, or 0 for none.
include/linux/filter.h +1 -0
index b17222db2efc..9682b98ad890 100644
@@ -1242,6 +1242,7 @@ bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_arena_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
+bool bpf_jit_supports_cleanup_pads(void);
 bool bpf_jit_supports_ptr_xchg(void);
 bool bpf_jit_supports_arena(void);
 bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
kernel/bpf/core.c +5 -0
index 4e208cc94752..f0dd851264f4 100644
@@ -3470,6 +3470,11 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp,
 {
 }
 
+bool __weak bpf_jit_supports_cleanup_pads(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_timed_may_goto(void)
 {
 	return false;
kernel/bpf/exception.c +55 -0

61 lines not on this page · raw message

kernel/bpf/exception.h +1 -0
index 0f2b9624a2ce..f51383fd775c 100644
@@ -7,6 +7,7 @@
 
 struct bpf_verifier_env;
 
+int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env);
 int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx);
 
 #endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/verifier.c +6 -0
index c65ff2e326bf..6496c30a10f7 100644
@@ -37,6 +37,7 @@
 
 #include "diagnostics.h"
 #include "disasm.h"
+#include "exception.h"
 
 static const struct bpf_verifier_ops * const bpf_verifier_ops[] = {
 #define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
@@ -21638,6 +21639,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/* The CFG needs an edge from a call in a cleanup range to its pad. */
+	ret = bpf_prepare_cleanup_exceptions(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	/* Validate instructions and resolve the program's referenced resources. */
 	ret = check_and_resolve_insns(env);
 	if (ret < 0)
Signature · 1 line
2.53.0-Meta
author reviewer
#5 lore raw

PATCH 5/20 bpf: Make exception landing pads reachable in the CFG

In reply to Yonghong Song

Both the CFG walk and liveness are involved. A bpf_throw() or a bpf2bpf call inside the [begin_off, end_off) range of a cleanup record can reach that record's landing pad, so both grow that edge; and a bpf_unwind_resume() reaches nothing after it, so it has no successors.

Liveness needs one more thing. bpf_stack_slot_alive() decides whether an outer frame's stack slot is still read after the call the frame is suspended at by looking at the instruction after the call. A slot whose only remaining reader is the landing pad -- which is every slot a compiler-generated pad reloads, since only four registers survive a call -- is dead by that measure, so clean_verifier_state() poisons it while the callee runs and the pad is then rejected for reading it. Ask about the pad as well when the call site names one.

---
 kernel/bpf/cfg.c      | 47 ++++++++++++++++++++++++++++++++++++++++---
 kernel/bpf/liveness.c | 20 ++++++++++++++++++
 2 files changed, 64 insertions(+), 3 deletions(-)
kernel/bpf/cfg.c +44 -3
index 842c7d1eabcc..9f8b8b54d5ea 100644
@@ -6,6 +6,7 @@
 #include <linux/sort.h>
 
 #include "diagnostics.h"
+#include "exception.h"
 
 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
 
@@ -158,17 +159,57 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
 	return DONE_EXPLORING;
 }
 
+static int visit_cleanup_pad_edge(int t, struct bpf_verifier_env *env)
+{
+	int *insn_stack = env->cfg.insn_stack;
+	int *insn_state = env->cfg.insn_state;
+	int w;
+
+	if (!env->cleanup_info_cnt)
+		return DONE_EXPLORING;
+	w = bpf_cleanup_pad_of_call(env, t);
+	if (w < 0)
+		return DONE_EXPLORING;
+
+	mark_prune_point(env, t);
+	mark_jmp_point(env, w);
+	mark_jump_target(env, w);
+
+	if (insn_state[w])
+		return DONE_EXPLORING;
+	if (env->cfg.cur_stack >= env->prog->len)
+		return -E2BIG;
+	insn_stack[env->cfg.cur_stack++] = w;
+	insn_state[w] |= DISCOVERED;
+	return KEEP_EXPLORING;
+}
+
+static int merge_visit_ret(int a, int b)
+{
+	if (a < 0)
+		return a;
+	if (b < 0)
+		return b;
+	if (a == KEEP_EXPLORING || b == KEEP_EXPLORING)
+		return KEEP_EXPLORING;
+	return DONE_EXPLORING;
+}
+
 static int visit_func_call_insn(int t, struct bpf_insn *insns,
 				struct bpf_verifier_env *env,
 				bool visit_callee)
 {
-	int ret, insn_sz;
+	int ret, insn_sz, pad_ret;
 	int w;
 
+	pad_ret = visit_cleanup_pad_edge(t, env);
+	if (pad_ret < 0)
+		return pad_ret;
+
 	insn_sz = bpf_is_ldimm64(&insns[t]) ? 2 : 1;
 	ret = push_insn(t, t + insn_sz, FALLTHROUGH, env);
 	if (ret)
-		return ret;
+		return merge_visit_ret(pad_ret, ret);
 
 	mark_prune_point(env, t + insn_sz);
 	/* when we exit from subprog, we need to record non-linear history */
@@ -180,7 +221,7 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,
 		merge_callee_effects(env, t, w);
 		ret = push_insn(t, w, BRANCH, env);
 	}
-	return ret;
+	return merge_visit_ret(pad_ret, ret);
 }
 
 struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem)
kernel/bpf/liveness.c +20 -0
index 44ecdc5b4ec2..9cfd05f970bc 100644
@@ -8,6 +8,8 @@
 #include <linux/slab.h>
 #include <linux/sort.h>
 
+#include "exception.h"
+
 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
 
 struct per_frame_masks {
@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 	succ = env->succ;
 	succ->cnt = 0;
 
+	if (unlikely(bpf_is_unwind_resume_kfunc(insn)))
+		return succ;
+
 	opcode_info = &opcode_info_tbl[BPF_CLASS(insn->code) | BPF_OP(insn->code)];
 	insn_sz = bpf_is_ldimm64(insn) ? 2 : 1;
 	if (opcode_info->can_fallthrough)
@@ -264,6 +269,13 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 	if (opcode_info->can_jump)
 		succ->items[succ->cnt++] = idx + bpf_jmp_offset(insn) + 1;
 
+	if (unlikely(env->cleanup_info_cnt)) {
+		int pad = bpf_cleanup_pad_of_call(env, idx);
+
+		if (pad >= 0)
+			succ->items[succ->cnt++] = pad;
+	}
+
 	return succ;
 }
 
@@ -397,6 +409,14 @@ bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp
 		alive = bpf_calls_callback(env, callsite)
 			? is_live_before(instance, callsite, rel, half_spi)
 			: is_live_before(instance, callsite + 1, rel, half_spi);
+
+		/* Control may also go to the landing pad. */
+		if (!alive && unlikely(env->cleanup_info_cnt)) {
+			int pad = bpf_cleanup_pad_of_call(env, callsite);
+
+			if (pad >= 0)
+				alive = is_live_before(instance, pad, rel, half_spi);
+		}
 		if (alive)
 			return true;
 	}
Signature · 1 line
2.53.0-Meta
#6 lore raw

Re: [PATCH bpf-next 05/20] bpf: Make exception landing pads reachable in the CFG

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] bpf: bpf_unwind_resume() retains fallthrough edge in CFG walk --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: bpf_unwind_resume() retains fallthrough edge in CFG walk
--

commit 663c539069f6d8aab713a59c4d2aa7c90391da34
Author: Yonghong Song <[email protected]>

bpf: Make exception landing pads reachable in the CFG

This commit updates the verifier's CFG walk and liveness analysis to account for exception landing pads. It adds edges from calls within a cleanup record's range to that record's landing pad. The commit message also states that bpf_unwind_resume() reaches nothing after it and should have no successors.

Quoted 4 lines
kernel/bpf/cfg.c +0 -0
index 842c7d1eabccc..9f8b8b54d5ea7 100644

[ ... ]

Quoted 1 line
kernel/bpf/cfg.c +0 -0
@@ -158,17 +159,57 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)

[ ... ]

Quoted 4 lines

static int visit_func_call_insn(int t, struct bpf_insn *insns,

kernel/bpf/cfg.c +0 -0
 				struct bpf_verifier_env *env,
 				bool visit_callee)
 {

kernel/bpf/cfg.c static int push_insn(int t, int w, int e, struct bpf_verifier_env *env) · line 210

Quoted 10 lines
kernel/bpf/cfg.c +5 -1
-	int ret, insn_sz;
+	int ret, insn_sz, pad_ret;
 	int w;
 
+	pad_ret = visit_cleanup_pad_edge(t, env);
+	if (pad_ret < 0)
+		return pad_ret;
+
 	insn_sz = bpf_is_ldimm64(&insns[t]) ? 2 : 1;
 	ret = push_insn(t, t + insn_sz, FALLTHROUGH, env);

[Severity: High]
Does this unconditionally push a fallthrough edge for all call instructions, failing to account for non-returning kfuncs like bpf_unwind_resume()?

If the compiler places a bpf_unwind_resume() call at the very end of the program (the end of the .text section) without any trailing instructions, will the CFG walk attempt to push a fallthrough edge out of bounds and incorrectly reject valid BPF programs with a 'jump out of range' error?

Quoted 5 lines
kernel/bpf/cfg.c +1 -1
 	if (ret)
-		return ret;
+		return merge_visit_ret(pad_ret, ret);
 
 	mark_prune_point(env, t + insn_sz);

[ ... ]

Quoted 4 lines
kernel/bpf/liveness.c +0 -0
index 44ecdc5b4ec2d..9cfd05f970bc9 100644

[ ... ]

kernel/bpf/liveness.c bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) · line 263

Quoted 7 lines
kernel/bpf/liveness.c +3 -0
@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 	succ = env->succ;
 	succ->cnt = 0;
 
+	if (unlikely(bpf_is_unwind_resume_kfunc(insn)))
+		return succ;
+

[Severity: High]
This correctly applies the logic mentioned in the commit message (that bpf_unwind_resume() reaches nothing after it and has no successors) to the liveness algorithm. Should visit_func_call_insn() in kernel/bpf/cfg.c be updated with similar logic to avoid pushing a fallthrough edge?

Quoted 3 lines
kernel/bpf/liveness.c +0 -0
 	opcode_info = &opcode_info_tbl[BPF_CLASS(insn->code) | BPF_OP(insn->code)];
 	insn_sz = bpf_is_ldimm64(insn) ? 2 : 1;
 	if (opcode_info->can_fallthrough)
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
author reviewer
#7 lore raw

Re: [PATCH bpf-next 05/20] bpf: Make exception landing pads reachable in the CFG

Quoted 39 lines · On 9/16/26 11:22 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: bpf_unwind_resume() retains fallthrough edge in CFG walk
--

commit 663c539069f6d8aab713a59c4d2aa7c90391da34
Author: Yonghong Song <[email protected]>

bpf: Make exception landing pads reachable in the CFG

This commit updates the verifier's CFG walk and liveness analysis to account for exception landing pads. It adds edges from calls within a cleanup record's range to that record's landing pad. The commit message also states that bpf_unwind_resume() reaches nothing after it and should have no successors.

Quoted 4 lines
kernel/bpf/cfg.c +0 -0
index 842c7d1eabccc..9f8b8b54d5ea7 100644

[ ... ]

Quoted 1 line
kernel/bpf/cfg.c +0 -0
@@ -158,17 +159,57 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)

[ ... ]

Quoted 4 lines

static int visit_func_call_insn(int t, struct bpf_insn *insns,
struct bpf_verifier_env *env, bool visit_callee)
{

kernel/bpf/cfg.c static int push_insn(int t, int w, int e, struct bpf_verifier_env *env) · line 210

Quoted 10 lines
kernel/bpf/cfg.c +5 -1
-	int ret, insn_sz;
+	int ret, insn_sz, pad_ret;
  	int w;
 
+	pad_ret = visit_cleanup_pad_edge(t, env);
+	if (pad_ret < 0)
+		return pad_ret;
+
  	insn_sz = bpf_is_ldimm64(&insns[t]) ? 2 : 1;
  	ret = push_insn(t, t + insn_sz, FALLTHROUGH, env);

[Severity: High]
Does this unconditionally push a fallthrough edge for all call instructions, failing to account for non-returning kfuncs like bpf_unwind_resume()?

Quoted 4 lines

If the compiler places a bpf_unwind_resume() call at the very end of the program (the end of the .text section) without any trailing instructions, will the CFG walk attempt to push a fallthrough edge out of bounds and incorrectly reject valid BPF programs with a 'jump out of range' error?

This won't happen. See verifier.c like below:

                        if (code != (BPF_JMP | BPF_EXIT) &&
                            code != (BPF_JMP32 | BPF_JA) &&
                            code != (BPF_JMP | BPF_JA)) {
                                verbose(env, "last insn is not an exit or jmp\n");
				...
			}
Quoted 16 lines
Quoted 6 lines

if (ret)

kernel/bpf/cfg.c +1 -1
-		return ret;
+		return merge_visit_ret(pad_ret, ret);
 
  	mark_prune_point(env, t + insn_sz);

[ ... ]

Quoted 4 lines
kernel/bpf/liveness.c +0 -0
index 44ecdc5b4ec2d..9cfd05f970bc9 100644

[ ... ]

Quoted 4 lines
kernel/bpf/liveness.c +0 -0
@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
  	succ = env->succ;
  	succ->cnt = 0;
 
Quoted 8 lines

kernel/bpf/liveness.c bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) · line 263

Quoted 3 lines
kernel/bpf/liveness.c +3 -0
+	if (unlikely(bpf_is_unwind_resume_kfunc(insn)))
+		return succ;
+

[Severity: High]
This correctly applies the logic mentioned in the commit message (that bpf_unwind_resume() reaches nothing after it and has no successors) to the liveness algorithm. Should visit_func_call_insn() in kernel/bpf/cfg.c be updated with similar logic to avoid pushing a fallthrough edge?

The same as above, bpf_unwind_resume() cannot be the last insn in the prog.

Quoted 4 lines
Quoted 4 lines
kernel/bpf/liveness.c +0 -0
  	opcode_info = &opcode_info_tbl[BPF_CLASS(insn->code) | BPF_OP(insn->code)];
  	insn_sz = bpf_is_ldimm64(insn) ? 2 : 1;
  	if (opcode_info->can_fallthrough)
author reviewer
#8 lore raw

PATCH 6/20 bpf: Explore the landing pads no call site reaches

In reply to Yonghong Song

A cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving bpf_check_cfg() to refuse the program over code its own frontend had no way not to emit:

0: call bpf_preempt_disable
1: call bpf_preempt_enable record = { begin = 1, end = 2, pad = 4 }
2: r0 = 0
3: exit
4: r1 = pads_ran ll landing pad
6: r2 = *(u64 *)(r1 + 0)
7: r2 |= RAN_NOUNWIND_REC
8: *(u64 *)(r1 + 0) = r2
9: call bpf_unwind_resume
10: exit

The range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it. cleanup_mark_call_sites() marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.

Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other: do_check() never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.

---
 kernel/bpf/cfg.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
kernel/bpf/cfg.c +17 -0
index 9f8b8b54d5ea..5a2b48b6a9e1 100644
@@ -633,6 +633,7 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
 	int insn_cnt = env->prog->len;
 	int *insn_stack, *insn_state;
 	int ex_insn_beg, i, ret = 0;
+	u32 pad_idx = 0;
 
 	insn_state = env->cfg.insn_state = kvzalloc_objs(int, insn_cnt,
 							 GFP_KERNEL_ACCOUNT);
@@ -688,6 +689,22 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
 		goto walk_cfg;
 	}
 
+	/*
+	 * A landing pad no call site was marked with -- a record whose range
+	 * holds no call an exception can unwind out of -- is reached by
+	 * nothing. Walk it from here, and let the dead code sweep remove it.
+	 */
+	while (pad_idx < env->cleanup_info_cnt) {
+		u32 pad = env->cleanup_info[pad_idx++].landing_pad_off;
+
+		if (insn_state[pad] != EXPLORED) {
+			insn_state[pad] = DISCOVERED;
+			insn_stack[0] = pad;
+			env->cfg.cur_stack = 1;
+			goto walk_cfg;
+		}
+	}
+
 	for (i = 0; i < insn_cnt; i++) {
 		struct bpf_insn *insn = &env->prog->insnsi[i];
 
Signature · 1 line
2.53.0-Meta
author reviewer
#9 lore raw

PATCH 7/20 bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch

In reply to Yonghong Song

The table is on the instructions and the landing pads are in the control flow graph. What is left before bpf_throw() can be taught to dispatch them is to work out what that walk will need, and refuse the shapes it could not handle.

bpf_check_cleanup_exceptions() runs after bpf_check_cfg(). Everything it needs is control flow, so it reads what that walk has already worked out rather than working it out again:

subprog_info.might_throw which subprograms an exception may leave,
closed over the call graph by merge_callee_effects() as the walk pops each callee
cleanup_reachability() what each instruction can reach -- a resume, a
plain exit, a throw, an indirect jump
cleanup_mark_pad_bodies() which instructions only ever run with an
exception already in flight

What it refuses:

- a pad that reaches both a resume and a plain exit, or neither: nothing says whether it is a cleanup pad or a catch pad
- a catch pad, which ends in a plain exit: the walker calls a pad as a subroutine and cannot hand a frame back its own execution
- a throw in a pad, or a call from a pad to a subprogram that can throw:
a second unwind over frames the first is still discarding
- a bpf_unwind_resume() outside a pad body
- a subprogram that may unwind used as a helper callback: the helper's own kernel frame would end the walk before it found a boundary
- a tail call, an indirect jump, or an outgoing on-stack call argument in a pad body, all of which touch a stack the pad does not own

---
 include/linux/bpf_verifier.h |   1 +
 kernel/bpf/exception.c       | 397 +++++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h       |   2 +
 kernel/bpf/fixups.c          |   1 +
 kernel/bpf/verifier.c        |   8 +
 5 files changed, 409 insertions(+)
include/linux/bpf_verifier.h +1 -0
index 7463e86d15ee..fe8b26351a15 100644
@@ -682,6 +682,7 @@ struct bpf_insn_aux_data {
 	bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
 	bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
 	bool cleanup_throw_site; /* call to bpf_throw() */
+	bool in_cleanup_pad; /* only runs with an exception in flight */
 	/*
 	 * 1 + the instruction index of the exception cleanup landing pad this
 	 * call site unwinds to, or 0 for none.
kernel/bpf/exception.c +397 -0

421 lines not on this page · raw message

kernel/bpf/exception.h +2 -0
index f51383fd775c..7313dd2b65a1 100644
@@ -8,6 +8,8 @@
 struct bpf_verifier_env;
 
 int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env);
+int bpf_check_cleanup_exceptions(struct bpf_verifier_env *env);
+int bpf_cleanup_check_callback(struct bpf_verifier_env *env, int subprog);
 int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx);
 
 #endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +1 -0
index 82b00fac6bd6..f120ae66dbbc 100644
@@ -252,6 +252,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 		/* Expand insni[off]'s seen count to the patched range. */
 		data[i].seen = old_seen;
 		data[i].zext_dst = bpf_insn_def32(new_prog, insn + i) >= 0;
+		data[i].in_cleanup_pad = data[off + cnt - 1].in_cleanup_pad;
 		if (!memcmp(insn + i, original_insn, sizeof(struct bpf_insn))) {
 			data[i].non_stack_access =
 				data[off + cnt - 1].non_stack_access;
kernel/bpf/verifier.c +8 -0
index 6496c30a10f7..9cbdb8339701 100644
@@ -10485,6 +10485,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
 	 * callbacks
 	 */
 	env->subprog_info[subprog].is_cb = true;
+	err = bpf_cleanup_check_callback(env, subprog);
+	if (err)
+		return err;
+
 	if (bpf_pseudo_kfunc_call(insn) &&
 	    !is_callback_calling_kfunc(insn->imm)) {
 		verifier_bug(env, "kfunc %s#%d not marked as callback-calling",
@@ -21664,6 +21668,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	ret = bpf_check_cleanup_exceptions(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	ret = bpf_compute_postorder(env);
 	if (ret < 0)
 		goto skip_full_check;
Signature · 1 line
2.53.0-Meta
#10 lore raw

Re: [PATCH bpf-next 07/20] bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch

Quoted 4 lines · Yonghong Song
kernel/bpf/exception.c +0 -0
index dfe9c2a9ce7d..b2bf831242b8 100644

[...]

Quoted 5 lines
kernel/bpf/exception.c +5 -0
+static enum cleanup_insn_kind cleanup_classify(struct bpf_verifier_env *env, u32 i,
+					       int *next, int *target)
+{
+	struct bpf_insn *insn = &env->prog->insnsi[i];
+	u8 class = BPF_CLASS(insn->code);

kernel/bpf/exception.c static bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf) · line 122

Quoted 10 lines
kernel/bpf/exception.c +10 -0
+
+	*next = i + 1;
+	*target = -1;
+
+	if (insn->code == (BPF_LD | BPF_IMM | BPF_DW)) {
+		*next = i + 2;
+		return CLEANUP_INSN_PLAIN;
+	}
+	if (class != BPF_JMP && class != BPF_JMP32)
+		return CLEANUP_INSN_PLAIN;

robot voice:

LD_ABS/LD_IND is not plain. bpf_check_cfg() treats it as an abnormal return and gen_ld_abs() later patches in a real "r0 = 0; exit" for the failed load. With this classification a pad in a tc prog that keeps skb in r6 and does an ld_abs passes t
he "reaches no exit" check,
cleanup_check_pad_insn() doesn't look at it, and check_ld_abs() only wants no locks/refs held at that point. At run time a failed load runs the subprog epilogue on the walker's stack: callee regs get popped from arch_bpf_run_cleanup_pad()'s frame, leave/ret with rbp == the unwinding frame goes straight back into the BPF caller with garbage in rbx/r13-r15, and the rest of bpf_throw() is abandoned. Treat it like BPF_EXIT here or refuse it in a pad body like tail call.

author reviewer
#11 lore raw

Re: [PATCH bpf-next 07/20] bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch

Quoted 24 lines · Alexei Starovoitov
Quoted 4 lines · Yonghong Song
kernel/bpf/exception.c +0 -0
index dfe9c2a9ce7d..b2bf831242b8 100644

[...]

Quoted 5 lines
kernel/bpf/exception.c +5 -0
+static enum cleanup_insn_kind cleanup_classify(struct bpf_verifier_env *env, u32 i,
+					       int *next, int *target)
+{
+	struct bpf_insn *insn = &env->prog->insnsi[i];
+	u8 class = BPF_CLASS(insn->code);

kernel/bpf/exception.c static bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf) · line 122

Quoted 10 lines
kernel/bpf/exception.c +10 -0
+
+	*next = i + 1;
+	*target = -1;
+
+	if (insn->code == (BPF_LD | BPF_IMM | BPF_DW)) {
+		*next = i + 2;
+		return CLEANUP_INSN_PLAIN;
+	}
+	if (class != BPF_JMP && class != BPF_JMP32)
+		return CLEANUP_INSN_PLAIN;

robot voice:

Quoted 12 lines

LD_ABS/LD_IND is not plain. bpf_check_cfg() treats it as an abnormal return and gen_ld_abs() later patches in a real "r0 = 0; exit" for the failed load. With this classification a pad in a tc prog that keeps skb in r6 and does an ld_abs passes t
he "reaches no exit" check,
cleanup_check_pad_insn() doesn't look at it, and check_ld_abs() only wants no locks/refs held at that point. At run time a failed load runs the subprog epilogue on the walker's stack: callee regs get popped from arch_bpf_run_cleanup_pad()'s frame, leave/ret with rbp == the unwinding frame goes straight back into the BPF caller with garbage in rbx/r13-r15, and the rest of bpf_throw() is abandoned. Treat it like BPF_EXIT here or refuse it in a pad body like tail call.

Yes, this is an issue. For insns in landing pad, we should not allow LD_ABS/LD_IND since they will have a hidden return. So the above suggestion to refuse it in a pad body is correct.

author reviewer
#12 lore raw

PATCH 8/20 bpf: Walk the exception unwind in the verifier

In reply to Yonghong Song

bpf_throw() is about to stop discarding the frames it unwinds and start running their landing pads instead. Have the verifier walk the same thing, step for step:

at a throw, with the frame chain in env->cur_state->frame[]
if a record covers the call control is at
run that record's landing pad in this frame on reaching its resume, pop the frame and carry on
else
pop the frame and carry on
at the boundary, deliver

Doing it this way is what keeps the resource rules unchanged. Whatever a pad releases is released in the verifier state too, so by the time the walk reaches the boundary the state says exactly what the program will really hold there -- and check_resource_leak(), which used to fire the moment a throw was seen, simply moves to the end of the walk. A frame that no record covers contributes nothing, so anything it held is still held when the walk ends, and that is what gets reported.

---
 include/linux/bpf_cleanup_abi.h |  16 +++++
 include/linux/bpf_verifier.h    |   1 +
 kernel/bpf/states.c             |   3 +
 kernel/bpf/verifier.c           | 102 +++++++++++++++++++++++++-------
 4 files changed, 99 insertions(+), 23 deletions(-)
 create mode 100644 include/linux/bpf_cleanup_abi.h
include/linux/bpf_cleanup_abi.h new file +16 -0
new file mode 100644
index 000000000000..b6c1d589abda
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef _LINUX_BPF_CLEANUP_ABI_H
+#define _LINUX_BPF_CLEANUP_ABI_H
+
+/*
+ * Value arch_bpf_run_cleanup_pad() leaves in r0 on the way into a landing pad.
+ * It has to be a constant the verifier knows: LLVM names r0 as both the
+ * exception pointer and the exception selector register, so every pad reads it
+ * before anything else and is free to store what it read. Kept on its own
+ * because the verifier and the arch dispatchers, which are assembly, have to
+ * agree on it.
+ */
+#define BPF_PAD_ENTRY_R0	1
+
+#endif /* _LINUX_BPF_CLEANUP_ABI_H */
include/linux/bpf_verifier.h +1 -0
index fe8b26351a15..09fb89fda40a 100644
@@ -509,6 +509,7 @@ struct bpf_verifier_state {
 
 	bool speculative;
 	bool in_sleepable;
+	bool unwinding;
 
 	/* first and last insn idx of this verifier state */
 	u32 first_insn_idx;
kernel/bpf/states.c +3 -0
index 66fb11b6c6a7..be0f529f7ecc 100644
@@ -996,6 +996,9 @@ static bool states_equal(struct bpf_verifier_env *env,
 	if (old->in_sleepable != cur->in_sleepable)
 		return false;
 
+	if (old->unwinding != cur->unwinding)
+		return false;
+
 	if (!refsafe(old, cur, &env->idmap_scratch))
 		return false;
 
kernel/bpf/verifier.c +79 -23

160 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#13 lore raw

Re: [PATCH bpf-next 08/20] bpf: Walk the exception unwind in the verifier

Quoted 4 lines · Yonghong Song
kernel/bpf/verifier.c +0 -0
index 9cbdb8339701..a3b34ded1392 100644

[...]

Quoted 8 lines
kernel/bpf/verifier.c +8 -0
+static int unwind_step(struct bpf_verifier_env *env, u32 callsite, int *insn_idx)
+{
+	struct bpf_verifier_state *state = env->cur_state;
+
+	state->unwinding = true;
+	for (;;) {
+		int pad = bpf_cleanup_pad_of_call(env, callsite);
+

kernel/bpf/verifier.c enum { · line 18491

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+		if (pad >= 0) {
+			unwind_enter_pad(env);
+			*insn_idx = pad;
+			return INSN_IDX_UPDATED;
+		}
+		if (!state->curframe)
+			return unwind_finish(env);
+		callsite = unwind_pop_frame(env);
+	}
+}

robot voice:

mark_chain_precision() doesn't know about this edge. The jmp history gets (bpf_throw in frame N+k) -> (pad in frame N), or (resume in N+1) -> (pad in N), with no BPF_EXIT in between, and backtrack_insn() only switches frames on E
XIT and pseudo calls. So bt->frame stays at N
while it walks the callee's insns backwards. If the callee wrote its own r6 the request for the caller's r6 is cleared there, the caller's def of r6 is never marked precise, and a later state with a different r6 is pruned at the call site checkpoint even though the pad does r10 + r6 with it. If the callee didn't touch r6 the walk reaches the static call insn with r6 still set and hits verifier_bug("static subprog unexpected regs"). For a throwing global subprog it's verifier_bug_if(idx + 1 != subseq_idx) right away. The unwind transition needs its own jmp history flag and backtrack_insn() has to bt_subprog_enter() once per popped frame, like it does for BPF_EXIT.
Pls add a test that does a variable offset stack access in a pad with the offset coming from r6-r9 set before the throwing call.

kernel/bpf/verifier.c enum {

Quoted 3 lines
kernel/bpf/verifier.c +3 -0
+static int process_cleanup_resume(struct bpf_verifier_env *env, int *insn_idx)
+{
+	struct bpf_verifier_state *state = env->cur_

state;

kernel/bpf/verifier.c enum { · line 18507

Quoted 12 lines
kernel/bpf/verifier.c +12 -0
+
+	/* A pad entered by ordinary control flow. */
+	if (!state->unwinding) {
+		verbose(env,
+			"bpf_unwind_resume() at insn %d reached without an exception in flight\n",
+			*insn_idx);
+		return -EINVAL;
+	}
+	if (!state->curframe)
+		return unwind_finish(env);
+	return unwind_step(env, unwind_pop_frame(env), insn_idx);
+}

unwinding is one bit for the whole state, so once a throw happened a bpf_unwind_resume() is accepted in any frame, not only in the frame whose pad the walker dispatched. The in_pad rule from patch 7 doesn't close it: take subprog S that never throws, has its own record, and whose pad is just "call bpf_unwind_resume" reachable by a plain branch from S's entry. F's pad does "call S" and S branches into its pad. Here the verifier pops S, finds no pad for the "call S" site, pops F and finishes, so whatever follows "call S" in F's pad is never walked in this state. On x86 S's resume is a bare ret, so at run time it

returns into F's pad right after "call S" and keeps executing it with S's leftover registers. On arm64 br x23 ends F's pad early instead. Remember the frame unwind_step() entered the pad in and reject bpf_unwind_resume() when curframe doesn't match.

pw-bot: cr

author reviewer
#14 lore raw

Re: [PATCH bpf-next 08/20] bpf: Walk the exception unwind in the verifier

Quoted 73 lines · Alexei Starovoitov
Quoted 4 lines · Yonghong Song
kernel/bpf/verifier.c +0 -0
index 9cbdb8339701..a3b34ded1392 100644

[...]

Quoted 8 lines
kernel/bpf/verifier.c +8 -0
+static int unwind_step(struct bpf_verifier_env *env, u32 callsite, int *insn_idx)
+{
+	struct bpf_verifier_state *state = env->cur_state;
+
+	state->unwinding = true;
+	for (;;) {
+		int pad = bpf_cleanup_pad_of_call(env, callsite);
+

kernel/bpf/verifier.c enum { · line 18491

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+		if (pad >= 0) {
+			unwind_enter_pad(env);
+			*insn_idx = pad;
+			return INSN_IDX_UPDATED;
+		}
+		if (!state->curframe)
+			return unwind_finish(env);
+		callsite = unwind_pop_frame(env);
+	}
+}

robot voice:

mark_chain_precision() doesn't know about this edge. The jmp history gets (bpf_throw in frame N+k) -> (pad in frame N), or (resume in N+1) -> (pad in N), with no BPF_EXIT in between, and backtrack_insn() only switches frames on E
XIT and pseudo calls. So bt->frame stays at N
while it walks the callee's insns backwards. If the callee wrote its own r6 the request for the caller's r6 is cleared there, the caller's def of r6 is never marked precise, and a later state with a different r6 is pruned at the call site checkpoint even though the pad does r10 + r6 with it. If the callee didn't touch r6 the walk reaches the static call insn with r6 still set and hits verifier_bug("static subprog unexpected regs"). For a throwing global subprog it's verifier_bug_if(idx + 1 != subseq_idx) right away. The unwind transition needs its own jmp history flag and backtrack_insn() has to bt_subprog_enter() once per popped frame, like it does for BPF_EXIT.
Pls add a test that does a variable offset stack access in a pad with the offset coming from r6-r9 set before the throwing call.

kernel/bpf/verifier.c enum {

Quoted 3 lines
kernel/bpf/verifier.c +3 -0
+static int process_cleanup_resume(struct bpf_verifier_env *env, int *insn_idx)
+{
+	struct bpf_verifier_state *state = env->cur_

state;

kernel/bpf/verifier.c enum { · line 18507

Quoted 12 lines
kernel/bpf/verifier.c +12 -0
+
+	/* A pad entered by ordinary control flow. */
+	if (!state->unwinding) {
+		verbose(env,
+			"bpf_unwind_resume() at insn %d reached without an exception in flight\n",
+			*insn_idx);
+		return -EINVAL;
+	}
+	if (!state->curframe)
+		return unwind_finish(env);
+	return unwind_step(env, unwind_pop_frame(env), insn_idx);
+}

unwinding is one bit for the whole state, so once a throw happened a bpf_unwind_resume() is accepted in any frame, not only in the frame whose pad the walker dispatched. The in_pad rule from patch 7 doesn't close it: take subprog S that never throws, has its own record, and whose pad is just "call bpf_unwind_resume" reachable by a plain branch from S's entry. F's pad does "call S" and S branches into its pad. Here the verifier pops S, finds no pad for the "call S" site, pops F and finishes, so whatever follows "call S" in F's pad is never walked in this state. On x86 S's resume is a bare ret, so at run time it

Quoted 4 lines

returns into F's pad right after "call S" and keeps executing it with S's leftover registers. On arm64 br x23 ends F's pad early instead. Remember the frame unwind_step() entered the pad in and reject bpf_unwind_resume() when curframe doesn't match.

Yes, this is a real issue for backtracking. The problem is stated in the above curframe doesn't match for frames after unwind_step(), mostly due to backtracking from landing pad insn to bpf_throw. Will fix.

Quoted 2 lines

pw-bot: cr

author reviewer
#15 lore raw

PATCH 9/20 bpf: Refuse a private stack for a program with an exception cleanup table

In reply to Yonghong Song

A landing pad is about to start running outside its own frame, on the bpf_throw() walker's stack, with that frame's registers put back from a spill area whose contents arch_bpf_run_cleanup_pad() knows how to read. The x86-64 JIT addresses a private-stack program's frame through a scratch register that it recomputes after each call rather than through rbp, and no spill area holds that register, so a pad there would address its frame through whatever the kernel left behind.

Refuse the combination in check_max_stack_depth(), where the choice is made. Everywhere rather than arch-conditionally: arm64 keeps its private stack pointer in x27, which is in the prologue spill and so survives, but a rule that holds on one arch and not the other is not worth the second code path when nothing is lost but an optimization.

---
 kernel/bpf/verifier.c | 8 ++++++++
 1 file changed, 8 insertions(+)
kernel/bpf/verifier.c +8 -0
index a3b34ded1392..43ecf79baa4a 100644
@@ -5591,6 +5591,14 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
 		}
 	}
 
+	/*
+	 * A pad rebuilds its frame from a spill area, and on x86-64 a private
+	 * stack's frame pointer is in no spill area. Refused on every arch
+	 * rather than just that one.
+	 */
+	if (env->cleanup_info_cnt)
+		priv_stack_mode = NO_PRIV_STACK;
+
 	if (priv_stack_mode == PRIV_STACK_UNKNOWN)
 		priv_stack_mode = bpf_enable_priv_stack(env->prog);
 
Signature · 1 line
2.53.0-Meta
author reviewer
#16 lore raw

PATCH 10/20 bpf: Dispatch exception cleanup pads from bpf_throw()

In reply to Yonghong Song

Stop discarding the frames an exception unwinds through. bpf_throw() already walks the BPF call stack with arch_bpf_stack_walk() to find the exception boundary; have it look each frame's return address up in that (sub)program's cleanup table on the way and run the landing pad a matching record names.

A pad is run as a subroutine of the walker, not jumped to. It executes with the unwinding frame's frame pointer and BPF callee-saved registers, so everything it reads is that frame's, but on the current stack far below it, so nothing it calls can disturb the frame it is cleaning up after. It ends in what the JIT emits for its bpf_unwind_resume(), which hands control back to the walker rather than to the frame's caller. That is exactly the procedure the LLVM commit emitting the section describes.

Restoring the frame's r6-r9 is what makes this work, and it is only possible because the callee about to be discarded spilled them in its own prologue. bpf_cleanup_force_spill() tells a JIT to make that spill unconditional and of a known shape for every subprogram of a program carrying a cleanup table, and aux->exc->spill_off records where it starts, so the walker needs no per-frame metadata. The frame that called bpf_throw() has no callee to have spilled anything and never runs its own epilogue, so the JIT spills that frame's registers at the throw site instead, in the area aux->exc->throw_spill_off names.

The main program needs the table handed to it rather than built for it. jit_subprogs() compiles it as func[0], but the ksym covering that image is the one bpf_prog_load() registers for the outer bpf_prog, and that is what the walker finds -- so without the handover a landing pad in the main program's own frame is never dispatched, silently, since the exception still reaches the boundary and the cookie still comes back.

All of it hangs off bpf_prog_aux by a single pointer. bpf_prog_aux is on every BPF program and this is a niche feature, so struct bpf_exception_info exists only for a program that carries a table, and aux->exc being set is what says the program carries one at all.

---
 include/linux/bpf.h    |  75 +++++++++++++++++++
 include/linux/filter.h |   1 +
 kernel/bpf/core.c      |  30 +++++++-
 kernel/bpf/exception.c | 166 +++++++++++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h |   7 ++
 kernel/bpf/fixups.c    | 125 +++++++++++++++++++++++++++++++
 kernel/bpf/helpers.c   |  34 +++++++++
 7 files changed, 435 insertions(+), 3 deletions(-)
include/linux/bpf.h +75 -0
index 2a5fa346aada..83f2b0d7e596 100644
@@ -1770,6 +1770,80 @@ enum bpf_sig_keyring {
 	BPF_SIG_KEYRING_BPF,
 };
 
+/*
+ * One cleanup region of a JITed (sub)program: @pad is the landing pad to run
+ * for a return address in (begin, end], the native code of its call sites.
+ */
+struct bpf_cleanup_range {
+	u64 begin;
+	u64 end;
+	u64 pad;
+};
+
+struct bpf_exception_info {
+	struct bpf_cleanup_info *info;
+	struct bpf_cleanup_range *ranges;
+	/* Landing pad instruction indices, sorted and deduplicated. */
+	u32 *pad_at;
+	/* bpf_throw() call instruction indices, sorted. */
+	u32 *throw_at;
+	/* One bit per instruction that only runs while unwinding. */
+	unsigned long *pad_body;
+	u32 nr_info;
+	u32 nr_ranges;
+	u32 nr_pad_at;
+	u32 nr_throw_at;
+	u32 nr_pad_body;
+	/* Offset from a frame's FP to the caller's spilled r6-r9. */
+	s32 spill_off;
+	/* Likewise, to the registers a frame spills before calling bpf_throw(). */
+	s32 throw_spill_off;
+};
+
+#ifdef CONFIG_BPF_SYSCALL
+bool bpf_cleanup_force_spill(const struct bpf_prog *prog);
+bool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx);
+bool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx);
+bool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx);
+int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env, struct bpf_prog *prog);
+void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image);
+void bpf_cleanup_free_info(struct bpf_prog_aux *aux);
+#else
+static inline bool bpf_cleanup_force_spill(const struct bpf_prog *prog)
+{
+	return false;
+}
+
+static inline bool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx)
+{
+	return false;
+}
+
+static inline bool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx)
+{
+	return false;
+}
+
+static inline bool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx)
+{
+	return false;
+}
+
+static inline int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env,
+					       struct bpf_prog *prog)
+{
+	return 0;
+}
+
+static inline void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image)
+{
+}
+
+static inline void bpf_cleanup_free_info(struct bpf_prog_aux *aux)
+{
+}
+#endif
+
 struct bpf_prog_aux {
 	atomic64_t refcnt;
 	u32 used_map_cnt;
@@ -1850,6 +1924,7 @@ struct bpf_prog_aux {
 	char name[BPF_OBJ_NAME_LEN];
 	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
 	u16 stack_arg_sp_adjust;
+	struct bpf_exception_info *exc;
 #ifdef CONFIG_SECURITY
 	void *security;
 #endif
include/linux/filter.h +1 -0
index 9682b98ad890..287cd9b59aa9 100644
@@ -1243,6 +1243,7 @@ bool bpf_jit_supports_arena_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
 bool bpf_jit_supports_cleanup_pads(void);
+void arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base);
 bool bpf_jit_supports_ptr_xchg(void);
 bool bpf_jit_supports_arena(void);
 bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
kernel/bpf/core.c +27 -3
index f0dd851264f4..bd2919063cec 100644
@@ -292,6 +292,7 @@ void __bpf_prog_free(struct bpf_prog *fp)
 		mutex_destroy(&fp->aux->dst_mutex);
 		mutex_destroy(&fp->aux->st_ops_assoc_mutex);
 		kfree(fp->aux->poke_tab);
+		bpf_cleanup_free_info(fp->aux);
 		kfree(fp->aux);
 	}
 	free_percpu(fp->stats);
@@ -2625,13 +2626,21 @@ static bool bpf_prog_select_interpreter(struct bpf_prog *fp)
 	return select_interpreter;
 }
 
-static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
+static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog,
+					     int *err)
 {
 #ifdef CONFIG_BPF_JIT
 	struct bpf_prog *orig_prog;
+	int ret;
 
-	if (!bpf_prog_need_blind(prog))
+	if (!bpf_prog_need_blind(prog)) {
+		ret = bpf_cleanup_attach_main_prog(env, prog);
+		if (ret) {
+			*err = ret;
+			return prog;
+		}
 		return bpf_int_jit_compile(env, prog);
+	}
 
 	orig_prog = prog;
 	prog = bpf_jit_blind_constants(env, prog);
@@ -2642,6 +2651,13 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
 	if (IS_ERR(prog))
 		goto out_restore;
 
+	ret = bpf_cleanup_attach_main_prog(env, prog);
+	if (ret) {
+		*err = ret;
+		bpf_jit_prog_release_other(orig_prog, prog);
+		goto out_restore;
+	}
+
 	prog = bpf_int_jit_compile(env, prog);
 	if (prog->jited) {
 		bpf_jit_prog_release_other(prog, orig_prog);
@@ -2681,8 +2697,10 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
 		if (*err)
 			return fp;
 
-		fp = bpf_prog_jit_compile(env, fp);
+		fp = bpf_prog_jit_compile(env, fp, err);
 		bpf_prog_jit_attempt_done(fp);
+		if (*err)
+			return fp;
 		if (!fp->jited && jit_needed) {
 			*err = -ENOTSUPP;
 			return fp;
@@ -3475,6 +3493,12 @@ bool __weak bpf_jit_supports_cleanup_pads(void)
 	return false;
 }
 
+/* Call @pad with the frame pointer @frame_fp and r6-r9 spilled at @spill_base. */
+void __weak arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base)
+{
+	WARN_ON_ONCE(1);
+}
+
 bool __weak bpf_jit_supports_timed_may_goto(void)
 {
 	return false;
kernel/bpf/exception.c +166 -0

174 lines not on this page · raw message

kernel/bpf/exception.h +7 -0
index 7313dd2b65a1..c0e68ce227c8 100644
@@ -5,11 +5,18 @@
 
 #include <linux/types.h>
 
+struct bpf_cleanup_info;
+struct bpf_cleanup_range;
+struct bpf_prog;
+struct bpf_prog_aux;
 struct bpf_verifier_env;
 
 int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env);
 int bpf_check_cleanup_exceptions(struct bpf_verifier_env *env);
 int bpf_cleanup_check_callback(struct bpf_verifier_env *env, int subprog);
 int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx);
+int bpf_cleanup_alloc_info(struct bpf_prog_aux *aux);
+int bpf_cleanup_attach_info(struct bpf_prog_aux *aux, struct bpf_cleanup_info *recs, u32 cnt);
+const struct bpf_cleanup_range *bpf_cleanup_pad_for_ip(const struct bpf_prog *prog, u64 ip);
 
 #endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +125 -0

166 lines not on this page · raw message

kernel/bpf/helpers.c +34 -0
index 31d84de4d318..ffef72804fc9 100644
@@ -31,6 +31,7 @@
 #include <linux/buildid.h>
 
 #include "../../lib/kstrtox.h"
+#include "exception.h"
 
 /* If kernel subsystem is allowing eBPF programs to call this function,
  * inside its own verifier_ops->get_func_proto() callback it should return
@@ -3360,8 +3361,36 @@ struct bpf_throw_ctx {
 	u64 sp;
 	u64 bp;
 	int cnt;
+	const struct bpf_prog *callee;
+	u64 callee_fp;
 };
 
+static void bpf_run_cleanup_pad(struct bpf_throw_ctx *ctx, const struct bpf_prog *prog,
+				u64 ip, u64 fp)
+{
+	const struct bpf_exception_info *exc = prog->aux->exc;
+	const struct bpf_cleanup_range *rec;
+	u64 spill_base;
+
+	if (!exc || !exc->nr_ranges)
+		return;
+	rec = bpf_cleanup_pad_for_ip(prog, ip);
+	if (!rec)
+		return;
+
+	/*
+	 * The callee is always another subprogram of this program -- the walk
+	 * ends at any frame that is not one -- so its prologue spilled these
+	 * registers and its exc is there to say where.
+	 */
+	if (ctx->callee)
+		spill_base = ctx->callee_fp + ctx->callee->aux->exc->spill_off;
+	else
+		spill_base = fp + exc->throw_spill_off;
+
+	arch_bpf_run_cleanup_pad(rec->pad, fp, spill_base);
+}
+
 static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
 {
 	struct bpf_throw_ctx *ctx = cookie;
@@ -3378,6 +3407,11 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
 	if (!prog)
 		return !ctx->cnt;
 	ctx->cnt++;
+
+	bpf_run_cleanup_pad(ctx, prog, ip, bp);
+	ctx->callee = prog;
+	ctx->callee_fp = bp;
+
 	if (bpf_is_subprog(prog))
 		return true;
 	ctx->aux = prog->aux;
Signature · 1 line
2.53.0-Meta
author reviewer
#17 lore raw

PATCH 11/20 bpf, x86: Dispatch exception cleanup pads at run time

In reply to Yonghong Song

Provide the arch half for x86-64: force the full callee-saved spill for a program carrying a cleanup table so the walker can find a frame's r6-r9 in its callee's prologue, record where that spill area starts, build the native cleanup table from the JIT's addrs[], emit a bare return for a pad's bpf_unwind_resume(), and hand control to a pad from arch_bpf_run_cleanup_pad().

Support is gated on CONFIG_UNWINDER_ORC, the same requirement arch_bpf_stack_walk() and therefore bpf_throw() already have here.

---
 arch/x86/net/Makefile          |  2 +-
 arch/x86/net/bpf_cleanup_pad.S | 74 ++++++++++++++++++++++++++++++++
 arch/x86/net/bpf_jit_comp.c    | 78 ++++++++++++++++++++++++++++------
 3 files changed, 141 insertions(+), 13 deletions(-)
 create mode 100644 arch/x86/net/bpf_cleanup_pad.S
arch/x86/net/Makefile +1 -1
index dddbefc0f439..9d574d972df3 100644
@@ -6,5 +6,5 @@
 ifeq ($(CONFIG_X86_32),y)
         obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o
 else
-        obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+        obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o
 endif
arch/x86/net/bpf_cleanup_pad.S new file +74 -0

74 lines not on this page · raw message

arch/x86/net/bpf_jit_comp.c +66 -12

170 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#18 lore raw

Re: [PATCH bpf-next 11/20] bpf, x86: Dispatch exception cleanup pads at run time

Quoted 4 lines · Yonghong Song
arch/x86/net/bpf_jit_comp.c +0 -0
index d4a980140b48..9d0dd54773e8 100644

[...]

arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · line 2947

Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +5 -0
+			if (bpf_is_unwind_resume_kfunc(insn)) {
+				emit_return(&prog, image + addrs[i - 1] + (prog - temp));
+				break;
+			}
+
 			func = (u8 *) __bpf_call_base + imm32;

bpf_is_unwind_resume_kfunc() matches imm == btf_id && off == 0, but by now bpf_fixup_kfunc_call() has rewritten every other kfunc's imm to an offset from __bpf_call_base. A vmlinux kfunc that happens to sit btf_id bytes after __bpf_call_base gets JITed as a bare ret in the middle of a normal prog. Looks incorrect.

author reviewer
#19 lore raw

Re: [PATCH bpf-next 11/20] bpf, x86: Dispatch exception cleanup pads at run time

Quoted 7 lines · Alexei Starovoitov
Quoted 4 lines · Yonghong Song
arch/x86/net/bpf_jit_comp.c +0 -0
index d4a980140b48..9d0dd54773e8 100644

[...]

Quoted 11 lines

arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · line 2947

Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +5 -0
+			if (bpf_is_unwind_resume_kfunc(insn)) {
+				emit_return(&prog, image + addrs[i - 1] + (prog - temp));
+				break;
+			}
+
  			func = (u8 *) __bpf_call_base + imm32;

bpf_is_unwind_resume_kfunc() matches imm == btf_id && off == 0, but by now bpf_fixup_kfunc_call() has rewritten every other kfunc's imm to an offset from __bpf_call_base. A vmlinux kfunc that happens to sit btf_id bytes after __bpf_call_base gets JITed as a bare ret in the middle of a normal prog. Looks incorrect.

Right, bpf_is_unwind_resume_kfunc() is not correct as it compares 'imm == bpf_id' while imm has been rewritten.
Will fix.

author reviewer
#20 lore raw

PATCH 12/20 bpf, arm64: Dispatch exception cleanup pads at run time

In reply to Yonghong Song

The same arch half for arm64: force the full callee-saved spill, record where it starts, build the native table from the JIT's byte offsets, and hand control to a pad from arch_bpf_run_cleanup_pad().

Unlike x86-64 a pad cannot simply return: every call it makes clobbers x30, so no return address survives to its resume. x23 carries it instead. bpf2a64[] maps nothing to x23 or x24 -- they are pushed only to keep the frame shape the exception callback expects -- so nothing else in generated code touches them, and being callee-saved they survive every kfunc the pad calls. The JIT emits "br x23" for a pad's bpf_unwind_resume().

x24 is the other one, and it is what makes a pad able to touch its own frame at all. Generated code addresses the BPF frame through the stack pointer -- the frame sits directly on top of it, which turns every offset positive and each access into one instruction -- and in a pad the stack pointer is the walker's. So the JIT has a pad recompute the equivalent of its frame's stack pointer from BPF r10 on entry, into x24, and addresses the frame off x24 for every instruction the previous patches marked as running only while unwinding. One instruction per pad, and the accesses keep the shape and the immediate range they have everywhere else.

---
 arch/arm64/net/Makefile          |  2 +-
 arch/arm64/net/bpf_cleanup_pad.S | 95 +++++++++++++++++++++++++++++++
 arch/arm64/net/bpf_jit_comp.c    | 97 ++++++++++++++++++++++++++++++--
 3 files changed, 187 insertions(+), 7 deletions(-)
 create mode 100644 arch/arm64/net/bpf_cleanup_pad.S
arch/arm64/net/Makefile +1 -1
index 3ae382bfca87..ebec2a44a52b 100644
@@ -2,4 +2,4 @@
 #
 # ARM64 networking code
 #
-obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o
arch/arm64/net/bpf_cleanup_pad.S new file +95 -0

95 lines not on this page · raw message

arch/arm64/net/bpf_jit_comp.c +91 -6

196 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
author reviewer
#21 lore raw

PATCH 13/20 libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc

In reply to Yonghong Song

LLVM terminates a cleanup landing pad with a call to _Unwind_Resume: the base unwind ABI's entry point for carrying an unwind on once a frame's cleanups have run. The kernel provides that terminator as a kfunc, but not under that name: claiming _Unwind_Resume in the kernel's own symbol table, for a function whose body never runs, would be needlessly confusing, so it is called bpf_unwind_resume.

---
 tools/lib/bpf/libbpf.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.c +14 -3
index 613afae26519..ea1c09fa3793 100644
@@ -8346,6 +8346,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s
 	}
 }
 
+/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the
+ * base unwind ABI's entry point for carrying an unwind on once a frame's
+ * cleanups have run. The kernel knows it as bpf_unwind_resume.
+ */
+static const char *kern_extern_name(const char *name)
+{
+	return strcmp(name, "_Unwind_Resume") ? name : "bpf_unwind_resume";
+}
+
 static int bpf_program_record_relos(struct bpf_program *prog)
 {
 	struct bpf_object *obj = prog->obj;
@@ -8807,17 +8816,19 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,
 	struct module_btf *mod_btf = NULL;
 	const struct btf_type *kern_func;
 	struct btf *kern_btf = NULL;
+	const char *kern_name;
 	int ret;
 
 	local_func_proto_id = ext->ksym.type_id;
 
-	kfunc_id = find_ksym_btf_id(obj, ext->essent_name ?: ext->name, BTF_KIND_FUNC, &kern_btf,
-				    &mod_btf);
+	kern_name = kern_extern_name(ext->essent_name ?: ext->name);
+
+	kfunc_id = find_ksym_btf_id(obj, kern_name, BTF_KIND_FUNC, &kern_btf, &mod_btf);
 	if (kfunc_id < 0) {
 		if (kfunc_id == -ESRCH && ext->is_weak)
 			return 0;
 		pr_warn("extern (func ksym) '%s': not found in kernel or module BTFs\n",
-			ext->name);
+			strcmp(kern_name, "bpf_unwind_resume") ? ext->name : kern_name);
 		return kfunc_id;
 	}
 
Signature · 1 line
2.53.0-Meta
#22 lore raw

Re: [PATCH bpf-next 13/20] libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] bpf: multi-line comment violates BPF subsystem style guide --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: multi-line comment violates BPF subsystem style guide
--

commit 082b791c15a0d9341a2e4f4d61c9b916d48b95fc
Author: Yonghong Song <[email protected]>

libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc

This commit maps LLVM's _Unwind_Resume function calls in cleanup landing pads to the kernel's corresponding bpf_unwind_resume kfunc. This ensures proper symbol resolution without polluting the kernel's symbol table.

tools/lib/bpf/libbpf.c static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s · line 8352

Quoted 12 lines
tools/lib/bpf/libbpf.c +4 -0
index 613afae265199..ea1c09fa37939 100644
@@ -8346,6 +8346,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s
 	}
 }
 
+/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the
+ * base unwind ABI's entry point for carrying an unwind on once a frame's
+ * cleanups have run. The kernel knows it as bpf_unwind_resume.
+ */

[Severity: Low]
This isn't a bug, but does this multi-line comment format violate the BPF subsystem style guide? The guide specifies that multi-line comments must have the opening /* on its own line, rather than starting the text on the same line as the opening characters.

Quoted 4 lines
tools/lib/bpf/libbpf.c +4 -0
+static const char *kern_extern_name(const char *name)
+{
+	return strcmp(name, "_Unwind_Resume") ? name : "bpf_unwind_resume";
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
#23 lore raw

Re: [PATCH bpf-next 13/20] libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc

In reply to Yonghong Song
Is the kern_extern_name() mapping applied on the light skeleton load path?
Quoted 10 lines
lib/bpf/libbpf.c +2 -0
index 613afae265199..ea1c09fa37939 100644
@@ -8346,6 +8346,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s
 	}
 }
 
+/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the
+ * base unwind ABI's entry point for carrying an unwind on once a frame's

lib/bpf/libbpf.c static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s · line 8360

Quoted 10 lines
lib/bpf/libbpf.c +7 -0
+ * cleanups have run. The kernel knows it as bpf_unwind_resume.
+ */
+static const char *kern_extern_name(const char *name)
+{
+	return strcmp(name, "_Unwind_Resume") ? name : "bpf_unwind_resume";
+}
+
 static int bpf_program_record_relos(struct bpf_program *prog)
 {
 	struct bpf_object *obj = prog->obj;

Is the kern_extern_name() mapping applied on the light skeleton load path?

Looking at bpf_program_record_relos() below kern_extern_name(), the RELO_EXTERN_CALL case records ext->name directly without calling kern_extern_name():

static int bpf_program_record_relos(struct bpf_program *prog)

{
    ...
    case RELO_EXTERN_CALL:
        bpf_gen__record_extern(obj->gen_loader, ext->name,
                               ext->is_weak, false, false, BTF_KIND_FUNC,
                               relo->insn_idx);
        break;
    ...
}

This means the generated loader program would ask the kernel for BTF_KIND_FUNC '_Unwind_Resume', which doesn't exist in vmlinux BTF. The kernel only knows bpf_unwind_resume per BTF_ID_FLAGS(func, bpf_unwind_resume) in kernel/bpf/helpers.c.

For comparison, the direct BPF_PROG_LOAD path does use the mapping:

bpf_object__resolve_ksym_func_btf_id() calls kern_extern_name() at libbpf.c:8824, but that resolution is deliberately skipped when obj->gen_loader is set:

bpf_object__resolve_ksyms_btf_id() {
    ...
    if (obj->gen_loader) {
        ext->is_set = true;
        ext->ksym.kernel_btf_obj_fd = 0;
        ext->ksym.kernel_btf_id = 0;
        continue;
    }
    ...
}

On the light skeleton path, bpf_btf_find_by_name_kind() would return -ESRCH for '_Unwind_Resume', causing emit_check_err() to jump to cleanup and fail the load. If the extern were weak, insn->imm/off would be stored as 0, which the verifier would reject as an invalid kfunc call.

The test added later in the series (selftests/bpf/progs/exceptions_cleanup_light.c) writes 'call bpf_unwind_resume' directly in inline asm, so it uses the kernel name and never exercises the mapping.

Quoted 12 lines
lib/bpf/libbpf.c +2 -2
@@ -8807,17 +8816,19 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,
 	struct module_btf *mod_btf = NULL;
 	const struct btf_type *kern_func;
 	struct btf *kern_btf = NULL;
+	const char *kern_name;
 	int ret;
 
 	local_func_proto_id = ext->ksym.type_id;
 
-	kfunc_id = find_ksym_btf_id(obj, ext->essent_name ?: ext->name, BTF_KIND_FUNC, &kern_btf,
-				    &mod_btf);
+	kern_name = kern_extern_name(ext->essent_name ?: ext->name);

lib/bpf/libbpf.c static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj, · line 8833

Quoted 10 lines
lib/bpf/libbpf.c +3 -1
+
+	kfunc_id = find_ksym_btf_id(obj, kern_name, BTF_KIND_FUNC, &kern_btf, &mod_btf);
 	if (kfunc_id < 0) {
 		if (kfunc_id == -ESRCH && ext->is_weak)
 			return 0;
 		pr_warn("extern (func ksym) '%s': not found in kernel or module BTFs\n",
-			ext->name);
+			strcmp(kern_name, "bpf_unwind_resume") ? ext->name : kern_name);
 		return kfunc_id;
 	}

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35188619605

author reviewer
#24 lore raw

Re: [PATCH bpf-next 13/20] libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc

Quoted 65 lines · On 9/17/26 12:00 AM, [email protected] wrote:
Quoted 10 lines
lib/bpf/libbpf.c +2 -0
index 613afae265199..ea1c09fa37939 100644
@@ -8346,6 +8346,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s
  	}
  }
 
+/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the
+ * base unwind ABI's entry point for carrying an unwind on once a frame's

lib/bpf/libbpf.c static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s · line 8360

Quoted 10 lines
lib/bpf/libbpf.c +7 -0
+ * cleanups have run. The kernel knows it as bpf_unwind_resume.
+ */
+static const char *kern_extern_name(const char *name)
+{
+	return strcmp(name, "_Unwind_Resume") ? name : "bpf_unwind_resume";
+}
+
  static int bpf_program_record_relos(struct bpf_program *prog)
  {
  	struct bpf_object *obj = prog->obj;

Is the kern_extern_name() mapping applied on the light skeleton load path?

Looking at bpf_program_record_relos() below kern_extern_name(), the RELO_EXTERN_CALL case records ext->name directly without calling kern_extern_name():

static int bpf_program_record_relos(struct bpf_program *prog)

{
     ...
     case RELO_EXTERN_CALL:
         bpf_gen__record_extern(obj->gen_loader, ext->name,
                                ext->is_weak, false, false, BTF_KIND_FUNC,
                                relo->insn_idx);
         break;
     ...
}

This means the generated loader program would ask the kernel for BTF_KIND_FUNC '_Unwind_Resume', which doesn't exist in vmlinux BTF. The kernel only knows bpf_unwind_resume per BTF_ID_FLAGS(func, bpf_unwind_resume) in kernel/bpf/helpers.c.

For comparison, the direct BPF_PROG_LOAD path does use the mapping:

bpf_object__resolve_ksym_func_btf_id() calls kern_extern_name() at libbpf.c:8824, but that resolution is deliberately skipped when obj->gen_loader is set:

bpf_object__resolve_ksyms_btf_id() {
     ...
     if (obj->gen_loader) {
         ext->is_set = true;
         ext->ksym.kernel_btf_obj_fd = 0;
         ext->ksym.kernel_btf_id = 0;
         continue;
     }
     ...
}

On the light skeleton path, bpf_btf_find_by_name_kind() would return -ESRCH for '_Unwind_Resume', causing emit_check_err() to jump to cleanup and fail the load. If the extern were weak, insn->imm/off would be stored as 0, which the verifier would reject as an invalid kfunc call.

Quoted 4 lines

The test added later in the series (selftests/bpf/progs/exceptions_cleanup_light.c) writes 'call bpf_unwind_resume' directly in inline asm, so it uses the kernel name and never exercises the mapping.

Indeed, later patch in selftests/bpf/progs/exceptions_cleanup_light.c fixed the issue with bpf_undiwn_resume properly. So this is not really a bug. But we can do better than this. We can change from ext->name to bpf_unwind_resume in file selftests/bpf/progs/exceptions_cleanup_light.c for this patch. This will make things easier to understand.

Quoted 29 lines
Quoted 13 lines
lib/bpf/libbpf.c +2 -2
@@ -8807,17 +8816,19 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,
  	struct module_btf *mod_btf = NULL;
  	const struct btf_type *kern_func;
  	struct btf *kern_btf = NULL;
+	const char *kern_name;
  	int ret;
 
  	local_func_proto_id = ext->ksym.type_id;
 
-	kfunc_id = find_ksym_btf_id(obj, ext->essent_name ?: ext->name, BTF_KIND_FUNC, &kern_btf,
-				    &mod_btf);
+	kern_name = kern_extern_name(ext->essent_name ?: ext->name);

lib/bpf/libbpf.c static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj, · line 8833

Quoted 10 lines
lib/bpf/libbpf.c +3 -1
+
+	kfunc_id = find_ksym_btf_id(obj, kern_name, BTF_KIND_FUNC, &kern_btf, &mod_btf);
  	if (kfunc_id < 0) {
  		if (kfunc_id == -ESRCH && ext->is_weak)
  			return 0;
  		pr_warn("extern (func ksym) '%s': not found in kernel or module BTFs\n",
-			ext->name);
+			strcmp(kern_name, "bpf_unwind_resume") ? ext->name : kern_name);
  		return kfunc_id;
  	}

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35188619605

author reviewer
#25 lore raw

PATCH 14/20 libbpf: Add cleanup_info to bpf_prog_load_opts

In reply to Yonghong Song

Let a caller hand the kernel an exception cleanup table. Extend struct bpf_prog_load_opts with cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, pass them through to BPF_PROG_LOAD, and grow the attr size bpf_prog_load() computes to cover the new fields.

---
 tools/lib/bpf/bpf.c | 6 +++++-
 tools/lib/bpf/bpf.h | 7 ++++++-
 2 files changed, 11 insertions(+), 2 deletions(-)
tools/lib/bpf/bpf.c +5 -1
index 96819c082c77..bcf490570961 100644
@@ -295,7 +295,7 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
 		  const struct bpf_insn *insns, size_t insn_cnt,
 		  struct bpf_prog_load_opts *opts)
 {
-	const size_t attr_sz = offsetofend(union bpf_attr, keyring_id);
+	const size_t attr_sz = offsetofend(union bpf_attr, cleanup_info_cnt);
 	void *finfo = NULL, *linfo = NULL;
 	const char *func_info, *line_info;
 	__u32 log_size, log_level, attach_prog_fd, attach_btf_obj_fd;
@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
 	attr.fd_array = ptr_to_u64(OPTS_GET(opts, fd_array, NULL));
 	attr.fd_array_cnt = OPTS_GET(opts, fd_array_cnt, 0);
 
+	attr.cleanup_info = ptr_to_u64(OPTS_GET(opts, cleanup_info, NULL));
+	attr.cleanup_info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+	attr.cleanup_info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);
+
 	if (log_level) {
 		attr.log_buf = ptr_to_u64(log_buf);
 		attr.log_size = log_size;
tools/lib/bpf/bpf.h +6 -1
index 7534a593edae..6f62a99e1e3e 100644
@@ -128,9 +128,14 @@ struct bpf_prog_load_opts {
 
 	/* if set, provides the length of fd_array */
 	__u32 fd_array_cnt;
+
+	/* exception cleanup table, from the .bpf_cleanup section */
+	const void *cleanup_info;
+	__u32 cleanup_info_cnt;
+	__u32 cleanup_info_rec_size;
 	size_t :0;
 };
-#define bpf_prog_load_opts__last_field fd_array_cnt
+#define bpf_prog_load_opts__last_field cleanup_info_rec_size
 
 LIBBPF_API int bpf_prog_load(enum bpf_prog_type prog_type,
 			     const char *prog_name, const char *license,
Signature · 1 line
2.53.0-Meta
author reviewer
#26 lore raw

PATCH 15/20 libbpf: Collect .bpf_cleanup records and pass them to the kernel

In reply to Yonghong Song

Parse the compiler-emitted .bpf_cleanup section and hand the resulting table to BPF_PROG_LOAD.

Each record is three 4-byte fields, and each field is a byte offset into some code section named by a matching .rel.bpf_cleanup relocation. bpf_object__init_cleanup_info() resolves both halves once at open time and keeps (section, instruction index) pairs; it rejects a record whose field has no relocation, whose relocation is not one of the two 32-bit types that spell a data reference to a code section -- R_BPF_64_NODYLD32 from LLVM, R_BPF_64_ABS32 from GNU as -- or whose offset is not instruction aligned.

The records are sorted by begin_off once the offsets are final: the kernel wants the table sorted with disjoint ranges so that it can find the record covering a call site with a binary search, and records arrive in .bpf_cleanup order, which says nothing about where the subprograms they describe were appended. Overlapping ranges are reported here, where the program name and both regions are still at hand.

bpf_object_load_prog() then passes the per-program table through the bpf_prog_load() options added in the previous patch, with the record size carried on the program the way func_info and line_info carry theirs rather than taken from a sizeof() at the call site. bpf_program__clone() carries it too. That is a second load path -- the one veristat uses -- and without the table the kernel sees landing pads nothing reaches and refuses the program with "unreachable insn".

---
 tools/lib/bpf/libbpf.c          | 284 ++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |   3 +
 2 files changed, 287 insertions(+)
tools/lib/bpf/libbpf.c +284 -0
index ea1c09fa3793..d7cb93a70f58 100644
@@ -514,6 +514,11 @@ struct bpf_program {
 	void *line_info;
 	__u32 line_info_rec_size;
 	__u32 line_info_cnt;
+
+	struct bpf_cleanup_info *cleanup_info;
+	__u32 cleanup_info_rec_size;
+	__u32 cleanup_info_cnt;
+
 	__u32 prog_flags;
 	__u8  hash[SHA256_DIGEST_LENGTH];
 
@@ -549,6 +554,7 @@ struct bpf_struct_ops {
 #define STRUCT_OPS_SEC ".struct_ops"
 #define STRUCT_OPS_LINK_SEC ".struct_ops.link"
 #define ARENA_SEC ".addr_space.1"
+#define CLEANUP_SEC ".bpf_cleanup"
 
 enum libbpf_map_type {
 	LIBBPF_MAP_UNSPEC,
@@ -677,6 +683,25 @@ struct elf_sec_desc {
 	Elf_Data *data;
 };
 
+#define CLEANUP_REC_FIELDS	(sizeof(struct bpf_cleanup_info) / sizeof(__u32))
+
+/* Index of each field of struct bpf_cleanup_info, read as an array of __u32. */
+enum {
+	CLEANUP_REC_BEGIN,
+	CLEANUP_REC_END,
+	CLEANUP_REC_PAD,
+};
+
+/* One (begin, end, landing_pad) triple from .bpf_cleanup, with each field
+ * resolved from its relocation to an ELF section plus a section-relative
+ * instruction index. The mapping to final program instruction indices can only
+ * happen after subprogram placement, which differs per main program.
+ */
+struct cleanup_raw_rec {
+	int sec_idx[CLEANUP_REC_FIELDS];
+	size_t insn_idx[CLEANUP_REC_FIELDS];
+};
+
 struct elf_state {
 	int fd;
 	const void *obj_buf;
@@ -696,6 +721,8 @@ struct elf_state {
 	bool has_st_ops;
 	int arena_data_shndx;
 	int jumptables_data_shndx;
+	Elf_Data *cleanup_data;
+	int cleanup_shndx;
 };
 
 struct usdt_manager;
@@ -771,6 +798,9 @@ struct bpf_object {
 	void *jumptables_data;
 	size_t jumptables_data_sz;
 
+	struct cleanup_raw_rec *cleanup_recs;
+	size_t cleanup_rec_cnt;
+
 	struct {
 		struct bpf_program *prog;
 		unsigned int sym_off;
@@ -817,7 +847,10 @@ static void bpf_program__exit(struct bpf_program *prog)
 	zfree(&prog->sec_name);
 	zfree(&prog->insns);
 	zfree(&prog->reloc_desc);
+	zfree(&prog->cleanup_info);
 
+	prog->cleanup_info_rec_size = 0;
+	prog->cleanup_info_cnt = 0;
 	prog->nr_reloc = 0;
 	prog->insns_cnt = 0;
 	prog->sec_idx = -1;
@@ -1554,6 +1587,7 @@ static struct bpf_object *bpf_object__new(const char *path,
 	obj->efile.obj_buf = obj_buf;
 	obj->efile.obj_buf_sz = obj_buf_sz;
 	obj->efile.btf_maps_shndx = -1;
+	obj->efile.cleanup_shndx = -1;
 	obj->kconfig_map_idx = -1;
 	obj->arena_map_idx = -1;
 
@@ -4040,6 +4074,9 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 				sec_desc->shdr = sh;
 				sec_desc->data = data;
 				obj->efile.has_st_ops = true;
+			} else if (strcmp(name, CLEANUP_SEC) == 0) {
+				obj->efile.cleanup_data = data;
+				obj->efile.cleanup_shndx = idx;
 			} else if (strcmp(name, ARENA_SEC) == 0) {
 				obj->efile.arena_data = data;
 				obj->efile.arena_data_shndx = idx;
@@ -4067,6 +4104,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 			    strcmp(name, ".rel" STRUCT_OPS_LINK_SEC) &&
 			    strcmp(name, ".rel?" STRUCT_OPS_SEC) &&
 			    strcmp(name, ".rel?" STRUCT_OPS_LINK_SEC) &&
+			    strcmp(name, ".rel" CLEANUP_SEC) &&
 			    strcmp(name, ".rel" MAPS_ELF_SEC)) {
 				pr_info("elf: skipping relo section(%d) %s for section(%d) %s\n",
 					idx, name, targ_sec_idx,
@@ -4847,6 +4885,214 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,
 	return NULL;
 }
 
+static int bpf_object__init_cleanup_info(struct bpf_object *obj)
+{
+	Elf_Data *data = obj->efile.cleanup_data;
+	Elf_Data *relo = NULL;
+	size_t i, nrels, nslots, nrecs;
+	struct cleanup_raw_rec *recs;
+	int *slot_sec, ret = 0;
+	size_t *slot_val;
+	const __u32 *vals;
+	bool native;
+
+	if (!data || obj->efile.cleanup_shndx < 0)
+		return 0;
+
+	native = is_native_endianness(obj);
+
+	for (i = 0; i < obj->efile.sec_cnt; i++) {
+		struct elf_sec_desc *sd = &obj->efile.secs[i];
+
+		if (sd->sec_type == SEC_RELO && sd->shdr &&
+		    sd->shdr->sh_info == (Elf64_Word)obj->efile.cleanup_shndx) {
+			relo = sd->data;
+			break;
+		}
+	}
+	if (!relo) {
+		pr_warn("%s present without relocations\n", CLEANUP_SEC);
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+	if (data->d_size % sizeof(struct bpf_cleanup_info)) {
+		pr_warn("%s size %zu is not a multiple of the record size %zu\n",
+			CLEANUP_SEC, data->d_size, sizeof(struct bpf_cleanup_info));
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+
+	vals = data->d_buf;
+	nslots = data->d_size / sizeof(__u32);
+	nrecs = data->d_size / sizeof(struct bpf_cleanup_info);
+
+	slot_sec = calloc(nslots, sizeof(*slot_sec));
+	slot_val = calloc(nslots, sizeof(*slot_val));
+	recs = calloc(nrecs ?: 1, sizeof(*recs));
+	if (!slot_sec || !slot_val || !recs) {
+		ret = -ENOMEM;
+		goto out;
+	}
+	for (i = 0; i < nslots; i++)
+		slot_sec[i] = -1;
+
+	/* One relocation per 4-byte field, naming the section it points into. */
+	nrels = relo->d_size / sizeof(Elf64_Rel);
+	for (i = 0; i < nrels; i++) {
+		Elf64_Rel *rel = elf_rel_by_idx(relo, i);
+		Elf64_Sym *sym = elf_sym_by_idx(obj, ELF64_R_SYM(rel->r_info));
+		size_t type = ELF64_R_TYPE(rel->r_info);
+		size_t slot = rel->r_offset / sizeof(__u32);
+
+		if (type != R_BPF_64_NODYLD32 && type != R_BPF_64_ABS32) {
+			pr_warn("%s: relocation %zu has unexpected type %zu\n",
+				CLEANUP_SEC, i, type);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		if (!sym || slot >= nslots || rel->r_offset % sizeof(__u32)) {
+			pr_warn("%s: bad relocation %zu\n", CLEANUP_SEC, i);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		slot_sec[slot] = sym->st_shndx;
+		/* The addend lives in the section data, which libelf leaves in
+		 * the object's byte order; a non-section symbol additionally
+		 * contributes its own value.
+		 */
+		slot_val[slot] = (native ? vals[slot] : bswap_32(vals[slot])) +
+				 sym->st_value;
+	}
+
+	for (i = 0; i < nslots; i++) {
+		struct cleanup_raw_rec *rec = &recs[i / CLEANUP_REC_FIELDS];
+		size_t field = i % CLEANUP_REC_FIELDS;
+
+		if (slot_sec[i] < 0) {
+			pr_warn("%s: field %zu has no relocation\n", CLEANUP_SEC, i);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		if (slot_val[i] % BPF_INSN_SZ) {
+			pr_warn("%s: field %zu offset %zu is not instruction aligned\n",
+				CLEANUP_SEC, i, slot_val[i]);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		rec->sec_idx[field] = slot_sec[i];
+		rec->insn_idx[field] = slot_val[i] / BPF_INSN_SZ;
+	}
+
+	obj->cleanup_recs = recs;
+	obj->cleanup_rec_cnt = nrecs;
+	recs = NULL;
+out:
+	free(recs);
+	free(slot_val);
+	free(slot_sec);
+	return ret;
+}
+
+static int cmp_cleanup_info(const void *a, const void *b)
+{
+	const struct bpf_cleanup_info *x = a, *y = b;
+
+	if (x->begin_off == y->begin_off)
+		return 0;
+	return x->begin_off < y->begin_off ? -1 : 1;
+}
+
+static int bpf_prog_collect_cleanup_info(struct bpf_object *obj,
+					 struct bpf_program *prog)
+{
+	size_t i;
+	int j;
+
+	for (i = 0; i < obj->cleanup_rec_cnt; i++) {
+		struct cleanup_raw_rec *raw = &obj->cleanup_recs[i];
+		struct bpf_program *owner = NULL;
+		struct bpf_cleanup_info ci = {};
+		__u32 *fields = (__u32 *)&ci;
+		void *tmp;
+
+		for (j = 0; j < CLEANUP_REC_FIELDS; j++) {
+			size_t idx = raw->insn_idx[j], final;
+			struct bpf_program *p;
+
+			/* The end of a range is exclusive, so it may name the
+			 * instruction just past the last one of a function,
+			 * which belongs to the next function or to nothing at
+			 * all. Ask about the last instruction the range covers,
+			 * the way the kernel does.
+			 */
+			if (j == CLEANUP_REC_END) {
+				if (!idx) {
+					pr_warn("%s: record %zu is an empty range\n",
+						CLEANUP_SEC, i);
+					return -LIBBPF_ERRNO__FORMAT;
+				}
+				idx--;
+			}
+
+			p = find_prog_by_sec_insn(obj, raw->sec_idx[j], idx);
+			if (!p) {
+				pr_warn("%s: record %zu field %d is not inside a function\n",
+					CLEANUP_SEC, i, j);
+				return -LIBBPF_ERRNO__FORMAT;
+			}
+			if (!owner) {
+				owner = p;
+			} else if (owner != p) {
+				pr_warn("%s: record %zu spans functions '%s' and '%s'\n",
+					CLEANUP_SEC, i, owner->name, p->name);
+				return -LIBBPF_ERRNO__FORMAT;
+			}
+
+			if (owner == prog) {
+				final = raw->insn_idx[j] - prog->sec_insn_off;
+			} else if (prog_is_subprog(obj, owner) && owner->sub_insn_off) {
+				/* sub_insn_off is where this subprogram was
+				 * appended to the main program being relocated;
+				 * zero means it is not part of it.
+				 */
+				final = owner->sub_insn_off +
+					raw->insn_idx[j] - owner->sec_insn_off;
+			} else {
+				owner = NULL;
+				break;
+			}
+			fields[j] = final;
+		}
+		if (!owner)
+			continue;
+
+		tmp = libbpf_reallocarray(prog->cleanup_info, prog->cleanup_info_cnt + 1,
+					  sizeof(*prog->cleanup_info));
+		if (!tmp)
+			return -ENOMEM;
+		prog->cleanup_info = tmp;
+		prog->cleanup_info_rec_size = sizeof(struct bpf_cleanup_info);
+		prog->cleanup_info[prog->cleanup_info_cnt++] = ci;
+
+		pr_debug("prog '%s': cleanup region [%u,%u) -> landing pad %u\n",
+			 prog->name, ci.begin_off, ci.end_off, ci.landing_pad_off);
+	}
+
+	qsort(prog->cleanup_info, prog->cleanup_info_cnt,
+	      sizeof(*prog->cleanup_info), cmp_cleanup_info);
+	for (i = 1; i < prog->cleanup_info_cnt; i++) {
+		struct bpf_cleanup_info *prev = &prog->cleanup_info[i - 1];
+		struct bpf_cleanup_info *cur = &prog->cleanup_info[i];
+
+		if (cur->begin_off < prev->end_off) {
+			pr_warn("prog '%s': overlapping cleanup regions [%u,%u) and [%u,%u)\n",
+				prog->name, prev->begin_off, prev->end_off,
+				cur->begin_off, cur->end_off);
+			return -LIBBPF_ERRNO__FORMAT;
+		}
+	}
+
+	return 0;
+}
+
 static int
 bpf_object__collect_prog_relos(struct bpf_object *obj, Elf64_Shdr *shdr, Elf_Data *data)
 {
@@ -7556,6 +7802,13 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_pat
 					return err;
 			}
 		}
+
+		err = bpf_prog_collect_cleanup_info(obj, prog);
+		if (err) {
+			pr_warn("prog '%s': failed to collect cleanup info: %s\n",
+				prog->name, errstr(err));
+			return err;
+		}
 	}
 	for (i = 0; i < obj->nr_programs; i++) {
 		prog = &obj->programs[i];
@@ -7746,6 +7999,9 @@ static int bpf_object__collect_relos(struct bpf_object *obj)
 			return -LIBBPF_ERRNO__INTERNAL;
 		}
 
+		if (idx == obj->efile.cleanup_shndx)
+			continue;
+
 		if (obj->efile.secs[idx].sec_type == SEC_ST_OPS)
 			err = bpf_object__collect_st_ops_relos(obj, shdr, data);
 		else if (idx == obj->efile.btf_maps_shndx)
@@ -8018,6 +8274,11 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
 		load_attr.line_info_rec_size = prog->line_info_rec_size;
 		load_attr.line_info_cnt = prog->line_info_cnt;
 	}
+	if (prog->cleanup_info_cnt) {
+		load_attr.cleanup_info = prog->cleanup_info;
+		load_attr.cleanup_info_cnt = prog->cleanup_info_cnt;
+		load_attr.cleanup_info_rec_size = prog->cleanup_info_rec_size;
+	}
 	load_attr.log_level = log_level;
 	load_attr.prog_flags = prog->prog_flags;
 	load_attr.fd_array = obj->fd_array;
@@ -8574,6 +8835,7 @@ static struct bpf_object *bpf_object_open(const char *path, const void *obj_buf,
 	err = err ? : bpf_object__init_maps(obj, opts);
 	err = err ? : bpf_object_init_progs(obj, opts);
 	err = err ? : bpf_object__collect_relos(obj);
+	err = err ? : bpf_object__init_cleanup_info(obj);
 	if (err)
 		goto out;
 
@@ -9687,6 +9949,9 @@ void bpf_object__close(struct bpf_object *obj)
 	zfree(&obj->jumptables_data);
 	obj->jumptables_data_sz = 0;
 
+	zfree(&obj->cleanup_recs);
+	obj->cleanup_rec_cnt = 0;
+
 	for (i = 0; i < obj->jumptable_map_cnt; i++)
 		close(obj->jumptable_maps[i].fd);
 	zfree(&obj->jumptable_maps);
@@ -10079,6 +10344,25 @@ int bpf_program__clone(struct bpf_program *prog, const struct bpf_prog_load_opts
 		attr.line_info_rec_size = info ? info_rec_size : prog->line_info_rec_size;
 	}
 
+	/* exception cleanup table */
+	info = OPTS_GET(opts, cleanup_info, NULL);
+	info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+	info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);
+	if (!!info != !!info_cnt || !!info != !!info_rec_size) {
+		pr_warn("prog '%s': cleanup_info, cleanup_info_cnt, and cleanup_info_rec_size must all be specified or all omitted\n",
+			prog->name);
+		return libbpf_err(-EINVAL);
+	}
+	if (info) {
+		attr.cleanup_info = info;
+		attr.cleanup_info_cnt = info_cnt;
+		attr.cleanup_info_rec_size = info_rec_size;
+	} else if (prog->cleanup_info_cnt) {
+		attr.cleanup_info = prog->cleanup_info;
+		attr.cleanup_info_cnt = prog->cleanup_info_cnt;
+		attr.cleanup_info_rec_size = prog->cleanup_info_rec_size;
+	}
+
 	/* Logging is caller-controlled; no fallback to prog/obj log settings */
 	attr.log_buf = OPTS_GET(opts, log_buf, NULL);
 	attr.log_size = OPTS_GET(opts, log_size, 0);
tools/lib/bpf/libbpf_internal.h +3 -0
index cb4d96233844..3ba6d9090368 100644
@@ -56,6 +56,9 @@
 #ifndef R_BPF_64_ABS32
 #define R_BPF_64_ABS32 3
 #endif
+#ifndef R_BPF_64_NODYLD32
+#define R_BPF_64_NODYLD32 4
+#endif
 #ifndef R_BPF_64_32
 #define R_BPF_64_32 10
 #endif
Signature · 1 line
2.53.0-Meta
#27 lore raw

Re: [PATCH bpf-next 15/20] libbpf: Collect .bpf_cleanup records and pass them to the kernel

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] bpf: undefined behavior when calling qsort with a NULL pointer - [Low] bpf: array bounds…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] bpf: undefined behavior when calling qsort with a NULL pointer
- [Low] bpf: array bounds violation via type-punned pointer arithmetic
--

commit 1e591245d7ec663b872eae44c8fa7f02cfd956a4
Author: Yonghong Song <[email protected]>

libbpf: Collect .bpf_cleanup records and pass them to the kernel

This commit parses the compiler-emitted .bpf_cleanup section and hands the resulting table to BPF_PROG_LOAD so the kernel can resolve exceptions.

Quoted 4 lines
tools/lib/bpf/libbpf.c +0 -0
index ea1c09fa37939..d7cb93a70f58d 100644

[ ... ]

Quoted 1 line
tools/lib/bpf/libbpf.c +0 -0
@@ -4847,6 +4885,214 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,

[ ... ]

Quoted 16 lines
tools/lib/bpf/libbpf.c +16 -0
+static int bpf_prog_collect_cleanup_info(struct bpf_object *obj,
+					 struct bpf_program *prog)
+{
+	size_t i;
+	int j;
+
+	for (i = 0; i < obj->cleanup_rec_cnt; i++) {
+		struct cleanup_raw_rec *raw = &obj->cleanup_recs[i];
+		struct bpf_program *owner = NULL;
+		struct bpf_cleanup_info ci = {};
+		__u32 *fields = (__u32 *)&ci;
+		void *tmp;
+
+		for (j = 0; j < CLEANUP_REC_FIELDS; j++) {
+			size_t idx = raw->insn_idx[j], final;
+			struct bpf_program *p;

[ ... ]

Quoted 4 lines
tools/lib/bpf/libbpf.c +4 -0
+			if (owner == prog) {
+				final = raw->insn_idx[j] - prog->sec_insn_off;
+			} else if (prog_is_subprog(obj, owner) && owner->sub_insn_off) {
+				/* sub_insn_off is where this subprogram was

tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 5062

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+				 * appended to the main program being relocated;
+				 * zero means it is not part of it.
+				 */
+				final = owner->sub_insn_off +
+					raw->insn_idx[j] - owner->sec_insn_off;
+			} else {
+				owner = NULL;
+				break;
+			}
+			fields[j] = final;

[Severity: Low]
Is it safe to cast a pointer to a struct field into an array pointer and access subsequent members via pointer arithmetic in this loop?

The struct bpf_cleanup_info fields are populated by casting a pointer to the struct to a __u32 * array and indexing into it in a loop.

According to C11 6.5.6, pointer arithmetic on a pointer to a scalar (the first struct field) only allows accessing up to one element past it, making fields[1] and fields[2] undefined behavior.

This can trigger compiler warnings (-Warray-bounds) and potentially cause aggressive optimizers to generate incorrect code.

Quoted 8 lines
tools/lib/bpf/libbpf.c +8 -0
+		}
+		if (!owner)
+			continue;
+
+		tmp = libbpf_reallocarray(prog->cleanup_info, prog->cleanup_info_cnt + 1,
+					  sizeof(*prog->cleanup_info));
+		if (!tmp)
+			return -ENOMEM;

tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 5080

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+		prog->cleanup_info = tmp;
+		prog->cleanup_info_rec_size = sizeof(struct bpf_cleanup_info);
+		prog->cleanup_info[prog->cleanup_info_cnt++] = ci;
+
+		pr_debug("prog '%s': cleanup region [%u,%u) -> landing pad %u\n",
+			 prog->name, ci.begin_off, ci.end_off, ci.landing_pad_off);
+	}
+
+	qsort(prog->cleanup_info, prog->cleanup_info_cnt,
+	      sizeof(*prog->cleanup_info), cmp_cleanup_info);

[Severity: Medium]
Can this unconditionally call qsort() with a NULL base pointer?

If obj->cleanup_rec_cnt is 0, or if none of the records match the program, prog->cleanup_info_cnt remains 0 and prog->cleanup_info will be NULL.

Calling qsort() with a NULL pointer invokes undefined behavior under C11 7.1.4, even if the element count is 0. This triggers UBSAN errors (nonnull attribute violation in libc headers) and can fail runtime checks.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
author reviewer
#28 lore raw

Re: [PATCH bpf-next 15/20] libbpf: Collect .bpf_cleanup records and pass them to the kernel

Quoted 61 lines · On 9/16/26 11:12 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] bpf: undefined behavior when calling qsort with a NULL pointer
- [Low] bpf: array bounds violation via type-punned pointer arithmetic
--

commit 1e591245d7ec663b872eae44c8fa7f02cfd956a4
Author: Yonghong Song <[email protected]>

libbpf: Collect .bpf_cleanup records and pass them to the kernel

This commit parses the compiler-emitted .bpf_cleanup section and hands the resulting table to BPF_PROG_LOAD so the kernel can resolve exceptions.

Quoted 4 lines
tools/lib/bpf/libbpf.c +0 -0
index ea1c09fa37939..d7cb93a70f58d 100644

[ ... ]

Quoted 1 line
tools/lib/bpf/libbpf.c +0 -0
@@ -4847,6 +4885,214 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,

[ ... ]

Quoted 16 lines
tools/lib/bpf/libbpf.c +16 -0
+static int bpf_prog_collect_cleanup_info(struct bpf_object *obj,
+					 struct bpf_program *prog)
+{
+	size_t i;
+	int j;
+
+	for (i = 0; i < obj->cleanup_rec_cnt; i++) {
+		struct cleanup_raw_rec *raw = &obj->cleanup_recs[i];
+		struct bpf_program *owner = NULL;
+		struct bpf_cleanup_info ci = {};
+		__u32 *fields = (__u32 *)&ci;
+		void *tmp;
+
+		for (j = 0; j < CLEANUP_REC_FIELDS; j++) {
+			size_t idx = raw->insn_idx[j], final;
+			struct bpf_program *p;

[ ... ]

Quoted 4 lines
tools/lib/bpf/libbpf.c +4 -0
+			if (owner == prog) {
+				final = raw->insn_idx[j] - prog->sec_insn_off;
+			} else if (prog_is_subprog(obj, owner) && owner->sub_insn_off) {
+				/* sub_insn_off is where this subprogram was

tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 5062

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+				 * appended to the main program being relocated;
+				 * zero means it is not part of it.
+				 */
+				final = owner->sub_insn_off +
+					raw->insn_idx[j] - owner->sec_insn_off;
+			} else {
+				owner = NULL;
+				break;
+			}
+			fields[j] = final;

[Severity: Low]
Is it safe to cast a pointer to a struct field into an array pointer and access subsequent members via pointer arithmetic in this loop?

The struct bpf_cleanup_info fields are populated by casting a pointer to the struct to a __u32 * array and indexing into it in a loop.

Quoted 6 lines

According to C11 6.5.6, pointer arithmetic on a pointer to a scalar (the first struct field) only allows accessing up to one element past it, making fields[1] and fields[2] undefined behavior.

This can trigger compiler warnings (-Warray-bounds) and potentially cause aggressive optimizers to generate incorrect code.

Yes, the code is correct but it is awkward and my trigger compiler warnings with additional compiler flag. Will make it explicit with bpf_cleanup_info fields.

Quoted 13 lines
Quoted 13 lines
tools/lib/bpf/libbpf.c +12 -0
+		}
+		if (!owner)
+			continue;
+
+		tmp = libbpf_reallocarray(prog->cleanup_info, prog->cleanup_info_cnt + 1,
+					  sizeof(*prog->cleanup_info));
+		if (!tmp)
+			return -ENOMEM;
+		prog->cleanup_info = tmp;
+		prog->cleanup_info_rec_size = sizeof(struct bpf_cleanup_info);
+		prog->cleanup_info[prog->cleanup_info_cnt++] = ci;
+
Quoted 8 lines

tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 5080

Quoted 6 lines
tools/lib/bpf/libbpf.c +6 -0
+		pr_debug("prog '%s': cleanup region [%u,%u) -> landing pad %u\n",
+			 prog->name, ci.begin_off, ci.end_off, ci.landing_pad_off);
+	}
+
+	qsort(prog->cleanup_info, prog->cleanup_info_cnt,
+	      sizeof(*prog->cleanup_info), cmp_cleanup_info);

[Severity: Medium]
Can this unconditionally call qsort() with a NULL base pointer?

You are right. It is possible prog->cleanup_info_cnt is 0 and in such cases, cleanup_info will be NULL. Will add a condition like 'prog->cleanup_info_cnt > 0'.

Quoted 8 lines

If obj->cleanup_rec_cnt is 0, or if none of the records match the program, prog->cleanup_info_cnt remains 0 and prog->cleanup_info will be NULL.

Calling qsort() with a NULL pointer invokes undefined behavior under C11 7.1.4, even if the element count is 0. This triggers UBSAN errors (nonnull attribute violation in libc headers) and can fail runtime checks.

author reviewer
#29 lore raw

PATCH 16/20 libbpf: Carry the exception cleanup table through the light skeleton

In reply to Yonghong Song

A light skeleton does not call bpf_prog_load(). bpf_gen__prog_load() builds its own union bpf_attr field by field, and the loader program it emits is what issues BPF_PROG_LOAD when the skeleton runs -- so a program loaded this way reached the kernel without the table the previous patch collected for it. The load then fails, because the landing pads are code nothing reaches and the verifier says so. It says "unreachable insn", which names neither the skeleton nor the table.

Carry it the way func_info and line_info are carried: the records go into the loader's blob of bytes, the count and record size into the attr, and a relocation stores the blob's address into attr.cleanup_info once that address is known. The attr grows to its new last field, cleanup_info_cnt. Records are 4-byte fields like the other info blobs, so a cross-endian build has to swap them too.

The kfunc names take the same detour. The light skeleton writes the name into the blob for the loader program to resolve when it runs, rather than resolving it against the kernel's BTF while libbpf runs, so the name recorded there has to go through kern_extern_name() as well.

---
 tools/lib/bpf/gen_loader.c      | 27 +++++++++++++++++++++++----
 tools/lib/bpf/libbpf.c          |  4 ++--
 tools/lib/bpf/libbpf_internal.h |  7 +++++++
 3 files changed, 32 insertions(+), 6 deletions(-)
tools/lib/bpf/gen_loader.c +23 -4
index af3a04f161ac..e9bb282a557a 100644
@@ -981,13 +981,15 @@ static void cleanup_relos(struct bpf_gen *gen, int insns)
 	cleanup_core_relo(gen);
 }
 
-/* Convert func, line, and core relo info blobs to target endianness */
+/* Convert func, line, core relo and cleanup info blobs to target endianness */
 static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
-			    int core_relos, struct bpf_prog_load_opts *load_attr)
+			    int core_relos, int cleanup_info,
+			    struct bpf_prog_load_opts *load_attr)
 {
 	struct bpf_func_info *fi = gen->data_start + func_info;
 	struct bpf_line_info *li = gen->data_start + line_info;
 	struct bpf_core_relo *cr = gen->data_start + core_relos;
+	struct bpf_cleanup_info *ci = gen->data_start + cleanup_info;
 	int i;
 
 	for (i = 0; i < load_attr->func_info_cnt; i++)
@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
 
 	for (i = 0; i < gen->core_relo_cnt; i++)
 		bpf_core_relo_bswap(cr++);
+
+	for (i = 0; i < load_attr->cleanup_info_cnt; i++)
+		bpf_cleanup_info_bswap(ci++);
 }
 
 void bpf_gen__prog_load(struct bpf_gen *gen,
@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 			       load_attr->line_info_rec_size;
 	int core_relo_tot_sz = gen->core_relo_cnt *
 			       sizeof(struct bpf_core_relo);
+	int cleanup_info_tot_sz = load_attr->cleanup_info_cnt *
+				  load_attr->cleanup_info_rec_size;
 	int prog_load_attr, license_off, insns_off, func_info, line_info, core_relos;
-	int attr_size = offsetofend(union bpf_attr, core_relo_rec_size);
+	int attr_size = offsetofend(union bpf_attr, cleanup_info_cnt);
+	int cleanup_info;
 	union bpf_attr attr;
 
 	memset(&attr, 0, attr_size);
@@ -1061,9 +1069,17 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 		 core_relos, gen->core_relo_cnt,
 		 sizeof(struct bpf_core_relo));
 
+	attr.cleanup_info_rec_size = tgt_endian(load_attr->cleanup_info_rec_size);
+	attr.cleanup_info_cnt = tgt_endian(load_attr->cleanup_info_cnt);
+	cleanup_info = add_data(gen, load_attr->cleanup_info, cleanup_info_tot_sz);
+	pr_debug("gen: prog_load: cleanup_info: off %d cnt %u rec size %u\n",
+		 cleanup_info, load_attr->cleanup_info_cnt,
+		 load_attr->cleanup_info_rec_size);
+
 	/* convert all info blobs to target endianness */
 	if (gen->swapped_endian && !gen->error)
-		info_blob_bswap(gen, func_info, line_info, core_relos, load_attr);
+		info_blob_bswap(gen, func_info, line_info, core_relos, cleanup_info,
+				load_attr);
 
 	libbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name));
 	prog_load_attr = add_data(gen, &attr, attr_size);
@@ -1085,6 +1101,9 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 	/* populate union bpf_attr with a pointer to core_relos */
 	emit_rel_store(gen, attr_field(prog_load_attr, core_relos), core_relos);
 
+	/* populate union bpf_attr with a pointer to cleanup_info */
+	emit_rel_store(gen, attr_field(prog_load_attr, cleanup_info), cleanup_info);
+
 	/* populate union bpf_attr fd_array with a pointer to data where map_fds are saved */
 	emit_rel_store(gen, attr_field(prog_load_attr, fd_array), gen->fd_array);
 
tools/lib/bpf/libbpf.c +2 -2
index d7cb93a70f58..ebb21c9cdfe0 100644
@@ -8632,12 +8632,12 @@ static int bpf_program_record_relos(struct bpf_program *prog)
 				continue;
 			kind = btf_is_var(btf__type_by_id(obj->btf, ext->btf_id)) ?
 				BTF_KIND_VAR : BTF_KIND_FUNC;
-			bpf_gen__record_extern(obj->gen_loader, ext->name,
+			bpf_gen__record_extern(obj->gen_loader, kern_extern_name(ext->name),
 					       ext->is_weak, !ext->ksym.type_id,
 					       true, kind, relo->insn_idx);
 			break;
 		case RELO_EXTERN_CALL:
-			bpf_gen__record_extern(obj->gen_loader, ext->name,
+			bpf_gen__record_extern(obj->gen_loader, kern_extern_name(ext->name),
 					       ext->is_weak, false, false, BTF_KIND_FUNC,
 					       relo->insn_idx);
 			break;
tools/lib/bpf/libbpf_internal.h +7 -0
index 3ba6d9090368..78519f24fb40 100644
@@ -572,6 +572,13 @@ static inline void bpf_core_relo_bswap(struct bpf_core_relo *i)
 	i->kind = bswap_32(i->kind);
 }
 
+static inline void bpf_cleanup_info_bswap(struct bpf_cleanup_info *i)
+{
+	i->begin_off = bswap_32(i->begin_off);
+	i->end_off = bswap_32(i->end_off);
+	i->landing_pad_off = bswap_32(i->landing_pad_off);
+}
+
 enum btf_field_iter_kind {
 	BTF_FIELD_ITER_IDS,
 	BTF_FIELD_ITER_STRS,
Signature · 1 line
2.53.0-Meta
#30 lore raw

Re: [PATCH bpf-next 16/20] libbpf: Carry the exception cleanup table through the light skeleton

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] libbpf: unconditional cleanup_info emission breaks backward compatibility on older kernels…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] libbpf: unconditional cleanup_info emission breaks backward compatibility on older kernels
- [Medium] libbpf: integer overflow in cleanup_info size calculation leads to OOB read/write --

commit 076a2794e54df3a939c1a2aebf4467a8a0af5e41
Author: Yonghong Song <[email protected]>

libbpf: Carry the exception cleanup table through the light skeleton

This commit adds support for carrying the exception cleanup table through the light skeleton during BPF program loads. It expands the bpf_attr union to include cleanup_info fields and handles endianness byte-swapping for the records.

Quoted 4 lines
tools/lib/bpf/gen_loader.c +0 -0
index af3a04f161ac1..e9bb282a557a3 100644

[ ... ]

tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1020

Quoted 6 lines
tools/lib/bpf/gen_loader.c +2 -0
@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 			       load_attr->line_info_rec_size;
 	int core_relo_tot_sz = gen->core_relo_cnt *
 			       sizeof(struct bpf_core_relo);
+	int cleanup_info_tot_sz = load_attr->cleanup_info_cnt *
+				  load_attr->cleanup_info_rec_size;

[Severity: Medium]
Is there a risk of integer overflow here when parsing untrusted ELF files?

If load_attr->cleanup_info_cnt is large enough, the multiplication with load_attr->cleanup_info_rec_size could overflow the 32-bit signed integer cleanup_info_tot_sz, resulting in a truncated heap allocation.

Later in info_blob_bswap(), the loop executes load_attr->cleanup_info_cnt times:

info_blob_bswap() {
    ...
    for (i = 0; i < load_attr->cleanup_info_cnt; i++)
        bpf_cleanup_info_bswap(ci++);
}

Could this lead to an out-of-bounds memory read and write over the truncated buffer, potentially causing memory corruption in userspace tooling like bpftool?

[ ... ]

tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1105

Quoted 6 lines
tools/lib/bpf/gen_loader.c +2 -0
@@ -1085,6 +1101,9 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 	/* populate union bpf_attr with a pointer to core_relos */
 	emit_rel_store(gen, attr_field(prog_load_attr, core_relos), core_relos);
 
+	/* populate union bpf_attr with a pointer to cleanup_info */
+	emit_rel_store(gen, attr_field(prog_load_attr, cleanup_info), cleanup_info);

[Severity: High]
Will this unconditional emission cause light skeletons to fail to load on older kernels?

By unconditionally writing a non-zero pointer (the offset in the data blob) into the tail of the BPF attribute structure, older target kernels that do not support the cleanup_info field might reject the BPF_PROG_LOAD syscall.

Because sys_bpf verifies that all fields beyond its known sizeof(union bpf_attr) are zero using bpf_check_uarg_tail_zero(), it seems this could fail with -E2BIG. Does this break backward compatibility for all light skeletons running on older kernels, even those that do not use exceptions?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
author reviewer
#31 lore raw

Re: [PATCH bpf-next 16/20] libbpf: Carry the exception cleanup table through the light skeleton

Quoted 29 lines · On 9/16/26 11:18 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] libbpf: unconditional cleanup_info emission breaks backward compatibility on older kernels
- [Medium] libbpf: integer overflow in cleanup_info size calculation leads to OOB read/write --

commit 076a2794e54df3a939c1a2aebf4467a8a0af5e41
Author: Yonghong Song <[email protected]>

libbpf: Carry the exception cleanup table through the light skeleton

This commit adds support for carrying the exception cleanup table through the light skeleton during BPF program loads. It expands the bpf_attr union to include cleanup_info fields and handles endianness byte-swapping for the records.

Quoted 4 lines
tools/lib/bpf/gen_loader.c +0 -0
index af3a04f161ac1..e9bb282a557a3 100644

[ ... ]

tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1020

Quoted 6 lines
tools/lib/bpf/gen_loader.c +2 -0
@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
  			       load_attr->line_info_rec_size;
  	int core_relo_tot_sz = gen->core_relo_cnt *
  			       sizeof(struct bpf_core_relo);
+	int cleanup_info_tot_sz = load_attr->cleanup_info_cnt *
+				  load_attr->cleanup_info_rec_size;

[Severity: Medium]
Is there a risk of integer overflow here when parsing untrusted ELF files?

Quoted 3 lines

If load_attr->cleanup_info_cnt is large enough, the multiplication with load_attr->cleanup_info_rec_size could overflow the 32-bit signed integer cleanup_info_tot_sz, resulting in a truncated heap allocation.

We should be okay here. In add_data, we will see:

static int add_data(struct bpf_gen *gen, const void *data, __u32 size)

{
        __u64 zero = 0;
        __u32 size8;
        void *prev;

        if (size > INT32_MAX) {
                gen->error = -ERANGE;
                return 0;
        }
        size8 = roundup(size, 8);
	...
}

So size guarding above prevents integer overflow. add_data() also applies to func_info_tot_sz, line_info_tot_sz, core_relo_tot_sz and here cleanup_info_tot_sz.

Quoted 19 lines

Later in info_blob_bswap(), the loop executes load_attr->cleanup_info_cnt times:

info_blob_bswap() {
     ...
     for (i = 0; i < load_attr->cleanup_info_cnt; i++)
         bpf_cleanup_info_bswap(ci++);
}

Could this lead to an out-of-bounds memory read and write over the truncated buffer, potentially causing memory corruption in userspace tooling like bpftool?

[ ... ]

Quoted 4 lines
tools/lib/bpf/gen_loader.c +0 -0
@@ -1085,6 +1101,9 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
  	/* populate union bpf_attr with a pointer to core_relos */
  	emit_rel_store(gen, attr_field(prog_load_attr, core_relos), core_relos);
 
Quoted 5 lines

tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1105

Quoted 2 lines
tools/lib/bpf/gen_loader.c +2 -0
+	/* populate union bpf_attr with a pointer to cleanup_info */
+	emit_rel_store(gen, attr_field(prog_load_attr, cleanup_info), cleanup_info);

[Severity: High]
Will this unconditional emission cause light skeletons to fail to load on older kernels?

You are right. We cannot unconditional emit. We can only do emit if cleanup_info_cnt > 0.

Quoted 10 lines

By unconditionally writing a non-zero pointer (the offset in the data blob) into the tail of the BPF attribute structure, older target kernels that do not support the cleanup_info field might reject the BPF_PROG_LOAD syscall.

Because sys_bpf verifies that all fields beyond its known sizeof(union bpf_attr) are zero using bpf_check_uarg_tail_zero(), it seems this could fail with -E2BIG. Does this break backward compatibility for all light skeletons running on older kernels, even those that do not use exceptions?

author reviewer
#32 lore raw

PATCH 17/20 libbpf: Let the static linker carry .bpf_cleanup relocations

In reply to Yonghong Song

An object that carries a compiler-emitted exception cleanup table cannot be linked today. The table's fields are byte offsets into a code section, materialised by a 32-bit relocation against that section's symbol with the offset itself as the implicit addend, and the linker rejects both halves of
that: the relocation type is not in the list it accepts, and a relocation against an STT_SECTION symbol from a non-executable section is an outright error.

Both spellings of that relocation have to be taken. LLVM emits R_BPF_64_NODYLD32 for a .long against a section symbol; GNU as emits R_BPF_64_ABS32, which is what bpf_reloc_type_lookup() maps BFD_RELOC_32 to. They describe the same value, and the selftests are built with both compilers.

---
 tools/lib/bpf/linker.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
tools/lib/bpf/linker.c +18 -1
index 78f92c39290a..e5c06023cb5b 100644
@@ -1036,7 +1036,8 @@ static int linker_sanity_check_elf_relos(struct src_obj *obj, struct src_sec *se
 		size_t sym_type = ELF64_R_TYPE(relo->r_info);
 
 		if (sym_type != R_BPF_64_64 && sym_type != R_BPF_64_32 &&
-		    sym_type != R_BPF_64_ABS64 && sym_type != R_BPF_64_ABS32) {
+		    sym_type != R_BPF_64_ABS64 && sym_type != R_BPF_64_ABS32 &&
+		    sym_type != R_BPF_64_NODYLD32) {
 			pr_warn("ELF relo #%d in section #%zu has unexpected type %zu in %s\n",
 				i, sec->sec_idx, sym_type, obj->filename);
 			return -EINVAL;
@@ -2274,6 +2275,22 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
 						insn->imm += sec->dst_off / sizeof(struct bpf_insn);
 					else
 						insn->imm += sec->dst_off;
+				} else if (sym_type == R_BPF_64_NODYLD32 ||
+					   sym_type == R_BPF_64_ABS32) {
+					__u32 *val;
+
+					/* Two spellings of the one thing: LLVM
+					 * emits NODYLD32 for a .long against a
+					 * section symbol, GNU as emits ABS32
+					 * (bpf_reloc_type_lookup() maps
+					 * BFD_RELOC_32 to it), and the value
+					 * they describe is the same.
+					 */
+					val = dst_linked_sec->raw_data + dst_rel->r_offset;
+					if (linker->swapped_endian)
+						*val = bswap_32(bswap_32(*val) + sec->dst_off);
+					else
+						*val += sec->dst_off;
 				} else {
 					pr_warn("relocation against STT_SECTION in non-exec section is not supported!\n");
 					return -EINVAL;
Signature · 1 line
2.53.0-Meta
#33 lore raw

Re: [PATCH bpf-next 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] libbpf: unaligned memory access in linker for 32-bit relocations --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] libbpf: unaligned memory access in linker for 32-bit relocations --

commit 408d36d153c1766697801912b5eb3e8087995ef9
Author: Yonghong Song <[email protected]>

libbpf: Let the static linker carry .bpf_cleanup relocations

This commit allows the BPF static linker to process 32-bit relocations against non-executable sections by handling R_BPF_64_NODYLD32 and R_BPF_64_ABS32 relocation types. It updates the 32-bit values directly in the raw data of the relocated section.

Quoted 4 lines
tools/lib/bpf/linker.c +0 -0
index 78f92c39290af..e5c06023cb5b1 100644

[ ... ]

Quoted 10 lines
tools/lib/bpf/linker.c +6 -0
@@ -2274,6 +2275,22 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
 						insn->imm += sec->dst_off / sizeof(struct bpf_insn);
 					else
 						insn->imm += sec->dst_off;
+				} else if (sym_type == R_BPF_64_NODYLD32 ||
+					   sym_type == R_BPF_64_ABS32) {
+					__u32 *val;
+
+					/* Two spellings of the one thing: LLVM
+					 * emits NODYLD32 for a .long against a

tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2293

Quoted 10 lines
tools/lib/bpf/linker.c +10 -0
+					 * section symbol, GNU as emits ABS32
+					 * (bpf_reloc_type_lookup() maps
+					 * BFD_RELOC_32 to it), and the value
+					 * they describe is the same.
+					 */
+					val = dst_linked_sec->raw_data + dst_rel->r_offset;
+					if (linker->swapped_endian)
+						*val = bswap_32(bswap_32(*val) + sec->dst_off);
+					else
+						*val += sec->dst_off;

[Severity: Medium]
Can this lead to an unaligned memory access crash?

The static linker processes these 32-bit relocations against non-executable sections by casting the relocated offset to a 32-bit integer pointer and directly dereferencing it.

If a user supplies a malformed BPF object file with an unaligned relocation offset, this could result in undefined behavior and a crash (SIGBUS) on host architectures with strict alignment requirements.

Should linker_sanity_check_elf_relos() be updated to ensure the relocation offset is a multiple of 4 for non-executable sections, similar to how it checks alignment for SHF_EXECINSTR sections?

tools/lib/bpf/linker.c:linker_sanity_check_elf_relos() {
		if (link_sec->shdr->sh_flags & SHF_EXECINSTR) {
			if (relo->r_offset % sizeof(struct bpf_insn) != 0) {
				pr_warn("ELF relo #%d in section #%zu points to missing symbol #%zu in %s\n",
					i, sec->sec_idx, sym_idx, obj->filename);
				return -EINVAL;
			}
		}
}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
author reviewer
#34 lore raw

Re: [PATCH bpf-next 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations

Quoted 43 lines · On 9/16/26 11:16 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] libbpf: unaligned memory access in linker for 32-bit relocations --

commit 408d36d153c1766697801912b5eb3e8087995ef9
Author: Yonghong Song <[email protected]>

libbpf: Let the static linker carry .bpf_cleanup relocations

This commit allows the BPF static linker to process 32-bit relocations against non-executable sections by handling R_BPF_64_NODYLD32 and R_BPF_64_ABS32 relocation types. It updates the 32-bit values directly in the raw data of the relocated section.

Quoted 4 lines
tools/lib/bpf/linker.c +0 -0
index 78f92c39290af..e5c06023cb5b1 100644

[ ... ]

Quoted 10 lines
tools/lib/bpf/linker.c +6 -0
@@ -2274,6 +2275,22 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
  						insn->imm += sec->dst_off / sizeof(struct bpf_insn);
  					else
  						insn->imm += sec->dst_off;
+				} else if (sym_type == R_BPF_64_NODYLD32 ||
+					   sym_type == R_BPF_64_ABS32) {
+					__u32 *val;
+
+					/* Two spellings of the one thing: LLVM
+					 * emits NODYLD32 for a .long against a

tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2293

Quoted 10 lines
tools/lib/bpf/linker.c +10 -0
+					 * section symbol, GNU as emits ABS32
+					 * (bpf_reloc_type_lookup() maps
+					 * BFD_RELOC_32 to it), and the value
+					 * they describe is the same.
+					 */
+					val = dst_linked_sec->raw_data + dst_rel->r_offset;
+					if (linker->swapped_endian)
+						*val = bswap_32(bswap_32(*val) + sec->dst_off);
+					else
+						*val += sec->dst_off;

[Severity: Medium]
Can this lead to an unaligned memory access crash?

Quoted 7 lines

The static linker processes these 32-bit relocations against non-executable sections by casting the relocated offset to a 32-bit integer pointer and directly dereferencing it.

If a user supplies a malformed BPF object file with an unaligned relocation offset, this could result in undefined behavior and a crash (SIGBUS) on host architectures with strict alignment requirements.

This only happen for a malformed BPF object file and we don't care if the BPF object file is indeed malformed.

Quoted 15 lines

Should linker_sanity_check_elf_relos() be updated to ensure the relocation offset is a multiple of 4 for non-executable sections, similar to how it checks alignment for SHF_EXECINSTR sections?

tools/lib/bpf/linker.c:linker_sanity_check_elf_relos() {
		if (link_sec->shdr->sh_flags & SHF_EXECINSTR) {
			if (relo->r_offset % sizeof(struct bpf_insn) != 0) {
				pr_warn("ELF relo #%d in section #%zu points to missing symbol #%zu in %s\n",
					i, sec->sec_idx, sym_idx, obj->filename);
				return -EINVAL;
			}
		}
}
author reviewer
#35 lore raw

PATCH 18/20 selftests/bpf: Add an end-to-end .bpf_cleanup exception test

In reply to Yonghong Song

C has no unwinding, so nothing here comes out of the frontend: the frames that own a resource are written as __naked inline assembly, which spells out by hand exactly what a frontend emits -- a call site bracketed by two labels, a landing pad unreachable in the compiler's CFG, and a .bpf_cleanup record tying them together. The assembler turns ".long <text label>" into the same R_BPF_64_NODYLD32 relocation the BPF AsmPrinter emits, so libbpf and the kernel see an object indistinguishable from a compiler-generated one.

Call chain: entry -> foo1 -> foo1v -> foo2 -> foo3. foo3 holds a non-preemptible section and throws inside it; foo2 holds an RCU read lock and has two call sites sharing one pad, one of them its own throw; foo1v is a void frame whose pad ends in a jump to a resume block placed after an unrelated block that ends in a plain exit; foo1 owns nothing and gets no record; entry is the boundary.

There are also some shapes the kernel refuses -- the ones with no correct answer, and the ones a pad running on the bpf_throw() walker's stack cannot express -- plus the return-value rejection that delivering an exception at the boundary of a program type which constrains its return value produces. The test skips rather than fails where the JIT cannot dispatch a landing pad at all.

---
 .../selftests/bpf/exceptions_cleanup.h        |  29 +
 .../bpf/prog_tests/exceptions_cleanup.c       |  79 +++
 .../selftests/bpf/progs/exceptions_cleanup.c  | 152 +++++
 .../bpf/progs/exceptions_cleanup_fail.c       | 600 ++++++++++++++++++
 4 files changed, 860 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
tools/testing/selftests/bpf/exceptions_cleanup.h new file +29 -0
new file mode 100644
index 000000000000..630d2e207119
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef __EXCEPTIONS_CLEANUP_H__
+#define __EXCEPTIONS_CLEANUP_H__
+
+#define THROW_COOKIE		0x100
+
+/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */
+#define RAN_FOO3_PREEMPT	0x1
+#define RAN_FOO2_RCU		0x2
+#define RAN_FOO1V_PREEMPT	0x4
+#define RAN_FOO2_DROP		0x8
+#define RAN_BUMP		0x10
+
+#define CLEANUP_REC(begin, end, landing_pad)			\
+	".pushsection .bpf_cleanup,\"a\",@progbits;"		\
+	".long " begin ";"					\
+	".long " end ";"					\
+	".long " landing_pad ";"				\
+	".popsection;"
+
+/* Set a bit in @pads_ran. */
+#define PAD_RAN(bit)						\
+	"r1 = %[pads_ran] ll;"					\
+	"r2 = *(u64 *)(r1 + 0);"				\
+	"r2 |= " bit ";"					\
+	"*(u64 *)(r1 + 0) = r2;"
+
+#endif /* __EXCEPTIONS_CLEANUP_H__ */
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +79 -0

79 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup.c new file +152 -0

152 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +600 -0

600 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#36 lore raw

Re: [PATCH bpf-next 18/20] selftests/bpf: Add an end-to-end .bpf_cleanup exception test

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] bpf: multi-line comments do not follow subsystem style --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: multi-line comments do not follow subsystem style
--

commit 5918cbd599a9ba7808857475f6ca49093862b54a
Author: Yonghong Song <[email protected]>

selftests/bpf: Add an end-to-end .bpf_cleanup exception test

This commit adds an end-to-end test for .bpf_cleanup exception handling, using inline assembly to construct landing pads and cleanup records. It validates both successful exception propagation scenarios and various edge cases that the kernel is expected to refuse.

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +0 -0
index d1e45b765af26..0000000000000 100644

[ ... ]

tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c line 72

Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +7 -0
+	/* foo3 throws; every pad runs and the cookie is delivered at entry. */
+	if (test__start_subtest("throw_from_foo3"))
+		run(skel, 101, THROW_COOKIE, PADS_FOO3_THREW);
+
+	/* foo3 returns 2 ^ 1 == 3, so foo2 throws from its own second region;
+	 * foo3's frame is long gone, so its pad must not run.
+	 */

[Severity: Low]
This isn't a bug, but does this multi-line comment follow the BPF subsystem style? The guidelines prefer placing the opening /* on its own line for multi-line comments.

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +4 -0
+	if (test__start_subtest("throw_from_foo2"))
+		run(skel, 2, THROW_COOKIE, PADS_FOO2_THREW);
+
+	exceptions_cleanup__destroy(skel);

[ ... ]

Quoted 4 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c +0 -0
index ce2dac306a84c..0000000000000 100644

[ ... ]

tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c line 586

Quoted 9 lines

+"3:" /* landing pad */

tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c +8 -0
+	"r1 = jt0_%= ll;"
+	"r1 += 8;"
+	"r2 = *(u64 *)(r1 + 0);"
+	/* gotox r2. Spelled as a raw insn on purpose: the "gotox" mnemonic
+	 * only reached the LLVM assembler in llvm 22, and BPF_RAW_INSN()
+	 * needs <linux/bpf.h>, which this file cannot have -- vmlinux.h
+	 * already defines the uapi enums.
+	 */

[Severity: Low]
This is also not a bug, but does this comment formatting comply with the subsystem guidelines? Like the previous instance, the preferred BPF style places the opening /* on a separate line.

Quoted 4 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c +4 -0
+	".8byte 0x20d;"
+"l0_%=:"
+	"call bpf_unwind_resume;"
+	"exit;"
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
author reviewer
#37 lore raw

PATCH 19/20 selftests/bpf: Cover the exception cleanup shapes the chain does not reach

In reply to Yonghong Song

The end-to-end test walks one call chain with a pad in most of its frames. This adds the shapes that chain does not reach: a callee called from both a covered and an uncovered site, a pad that reads its frame's callee-saved registers, a tail-call-reachable callee, a tail call that is taken, an extension standing in for a covered call, a pad in the main program's own frame, a record covering bpf_throw() itself, a record covering only a nounwind call, a throwing subprogram named by a BPF_PSEUDO_FUNC, a region ending on a 16-byte instruction, a pad that reloads from and writes to its own frame, a pad two frames up, a pad that calls a subprogram which tail calls, an extension carrying a table of its own, and a pad terminated by _Unwind_Resume rather than bpf_unwind_resume.

---
 .../selftests/bpf/exceptions_cleanup.h        |  20 +
 .../bpf/prog_tests/exceptions_cleanup.c       | 314 +++++++
 .../bpf/progs/exceptions_cleanup_ext_table.c  |  48 +
 .../bpf/progs/exceptions_cleanup_freplace.c   |  17 +
 .../progs/exceptions_cleanup_pad_freplace.c   |  17 +
 .../bpf/progs/exceptions_cleanup_shapes.c     | 863 ++++++++++++++++++
 6 files changed, 1279 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
tools/testing/selftests/bpf/exceptions_cleanup.h +20 -0
index 630d2e207119..5896cf83d15e 100644
@@ -4,6 +4,7 @@
 #define __EXCEPTIONS_CLEANUP_H__
 
 #define THROW_COOKIE		0x100
+#define INNER_COOKIE		0x200
 
 /* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */
 #define RAN_FOO3_PREEMPT	0x1
@@ -12,6 +13,25 @@
 #define RAN_FOO2_DROP		0x8
 #define RAN_BUMP		0x10
 
+/* progs/exceptions_cleanup_shapes.c: one bit per shape, numbered its own way. */
+#define RAN_SWEEP		0x1
+#define RAN_SHARED		0x2
+#define RAN_REGS		0x4
+#define RAN_TAIL_CALL		0x8
+#define RAN_MAIN_PAD		0x10
+#define RAN_TC_TAKEN		0x20
+#define RAN_FREPLACE		0x40
+#define RAN_ADDR_TAKEN		0x80
+#define RAN_NO_SUBPROG		0x100
+#define RAN_PAD_CALLS		0x200
+#define RAN_PAD_FIRST		0x400
+#define RAN_WIDE_REC		0x800
+#define RAN_PAD_STACK		0x1000
+#define RAN_DEEP_PAD		0x2000
+#define RAN_NOUNWIND_REC	0x4000
+#define RAN_RESUME_ALIAS	0x8000
+#define RAN_PAD_TAIL_CALL	0x10000
+
 #define CLEANUP_REC(begin, end, landing_pad)			\
 	".pushsection .bpf_cleanup,\"a\",@progbits;"		\
 	".long " begin ";"					\
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +314 -0

331 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c new file +48 -0

48 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c new file +17 -0
new file mode 100644
index 000000000000..afb358fd3d40
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "exceptions_cleanup.h"
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */
+int btf_int_anchor;
+
+SEC("freplace/fr_callee")
+__u64 new_fr_callee(__u64 x)
+{
+	bpf_throw(THROW_COOKIE);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c new file +17 -0
new file mode 100644
index 000000000000..eabac6baabb7
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "exceptions_cleanup.h"
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */
+int btf_int_anchor;
+
+SEC("freplace/pad_callee")
+__u64 new_pad_callee(__u64 x)
+{
+	bpf_throw(INNER_COOKIE);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +863 -0

863 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#38 lore raw

Re: [PATCH bpf-next 19/20] selftests/bpf: Cover the exception cleanup shapes the chain does not reach

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] bpf: multi-line comments violate BPF subsystem style --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: multi-line comments violate BPF subsystem style
--

commit 621c13df81280f1c0c32e09255ce8c64860bd768
Author: Yonghong Song <[email protected]>

selftests/bpf: Cover the exception cleanup shapes the chain does not reach

The end-to-end test walks one call chain with a pad in most of its frames. This adds the shapes that chain does not reach: a callee called from both a covered and an uncovered site, a pad that reads its frame's callee-saved registers, a tail-call-reachable callee, a tail call that is taken, an extension standing in for a covered call, a pad in the main program's own frame, a record covering bpf_throw() itself, a record covering only a nounwind call, a throwing subprogram named by a BPF_PSEUDO_FUNC, a region ending on a 16-byte instruction, a pad that reloads from and writes to its own frame, a pad two frames up, a pad that calls a subprogram which tail calls, an extension carrying a table of its own, and a pad terminated by _Unwind_Resume rather than bpf_unwind_resume.

Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +0 -0
index d1e45b765af26..ffc0b95191681 100644
@@ -212,6 +212,9 @@ static void test_shapes(void)
 

[ ... ]

tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 215

Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +3 -0
+	/* The same call site either way: the subprogram's throw unwinds into
+	 * this frame and runs its pad, an extension's stops at its own boundary.
+	 */

[Severity: Low]
Does this multi-line comment in test_shapes() violate the BPF subsystem style?

The style guidelines prefer the opening /* on its own line:

	/*
	 * The same call site either way: the subprogram's throw unwinds into
	 * this frame and runs its pad, an extension's stops at its own boundary.
	 */

[ ... ]

tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 224

Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +3 -0
+	/* A tail call that is taken: the walk ends at the target, so the cookie
+	 * comes back from there and this frame's pad does not run.
+	 */

[Severity: Low]
Could this multi-line comment in test_shapes() also be updated to start the text on the line following the opening /* to match the preferred BPF style?

tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c static __used __noinline __u64 tc_thrower(void *ctx) · line 217

Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +3 -0
new file mode 100644
index 0000000000000..f5eb2ff15c896
@@ -214,6 +214,9 @@ static __used __noinline __u64 tc_thrower(void *ctx)
 {
+	/* Never taken; its presence is what makes this frame, whose spill the
+	 * walker reads, tail-call-reachable.
+	 */

[Severity: Low]
Is this multi-line comment in tc_thrower() formatted correctly?

The BPF subsystem style requires the opening /* on its own line for multi-line comments.

[ ... ]

tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c static __used __noinline __u64 tc_taken_callee(void *ctx, __u64 x) · line 314

Quoted 5 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +3 -0
@@ -311,6 +311,9 @@ static __used __noinline __u64 tc_taken_callee(void *ctx, __u64 x)
 {
+	/* Never true at run time; the verifier cannot know that, and its
+	 * unwind out of here is what keeps the caller's pad alive.
+	 */

[Severity: Low]
Should this multi-line comment in tc_taken_callee() also be adjusted so the text starts on the next line after the opening /* marker?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19
author reviewer
#39 lore raw

PATCH 20/20 selftests/bpf: Load an exception cleanup program from a light skeleton

In reply to Yonghong Song

For a light skeleton, libbpf hands the records to bpf_gen__prog_load(), which writes them into a blob and emits a loader program that issues BPF_PROG_LOAD from inside the kernel. Nothing about that is shared with the ordinary path: the attr is built field by field, and the kfunc names are resolved by the loader program when it runs.

exceptions_cleanup_light.c is the smallest program that can tell whether a table survives that trip: one record covering the bpf_throw() call itself, one pad that sets a bit and resumes. If the table arrives, the pad runs and the cookie comes back; if it does not, the program does not load at all.

---
 tools/testing/selftests/bpf/Makefile          |  2 +-
 .../selftests/bpf/exceptions_cleanup.h        |  3 ++
 .../bpf/prog_tests/exceptions_cleanup.c       | 28 +++++++++++++
 .../bpf/progs/exceptions_cleanup_light.c      | 39 +++++++++++++++++++
 4 files changed, 71 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c
tools/testing/selftests/bpf/Makefile +1 -1
index 7ea5ba1df29e..981cc2643049 100644
@@ -523,7 +523,7 @@ LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h		\
 LSKELS := fexit_sleep.c trace_printk.c trace_vprintk.c map_ptr_kern.c 	\
 	core_kern.c core_kern_overflow.c test_ringbuf.c			\
 	test_ringbuf_n.c test_ringbuf_map_key.c test_ringbuf_write.c    \
-	test_ringbuf_overwrite.c
+	test_ringbuf_overwrite.c exceptions_cleanup_light.c
 
 LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
 
tools/testing/selftests/bpf/exceptions_cleanup.h +3 -0
index 5896cf83d15e..0c088d96ca01 100644
@@ -32,6 +32,9 @@
 #define RAN_RESUME_ALIAS	0x8000
 #define RAN_PAD_TAIL_CALL	0x10000
 
+/* progs/exceptions_cleanup_light.c: the one pad it has. */
+#define RAN_LIGHT		0x1
+
 #define CLEANUP_REC(begin, end, landing_pad)			\
 	".pushsection .bpf_cleanup,\"a\",@progbits;"		\
 	".long " begin ";"					\
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +28 -0
index ffc0b9519168..ad0ff949f10d 100644
@@ -8,6 +8,7 @@
 #include "exceptions_cleanup_freplace.skel.h"
 #include "exceptions_cleanup_pad_freplace.skel.h"
 #include "exceptions_cleanup_ext_table.skel.h"
+#include "exceptions_cleanup_light.lskel.h"
 
 /* foo3 threw: every frame that has a pad ran it. */
 #define PADS_FOO3_THREW \
@@ -170,6 +171,30 @@ static void test_ext_table(struct exceptions_cleanup_shapes *skel)
 	exceptions_cleanup_ext_table__destroy(fr);
 }
 
+static void test_light_skeleton(void)
+{
+	struct exceptions_cleanup_light_lskel *skel;
+	__u64 ctx = 0;
+	int err;
+
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		    .ctx_in = &ctx,
+		    .ctx_size_in = sizeof(ctx),
+	);
+
+	skel = exceptions_cleanup_light_lskel__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "light open_and_load"))
+		return;
+
+	err = bpf_prog_test_run_opts(skel->progs.entry_light.prog_fd, &topts);
+	if (!ASSERT_OK(err, "run"))
+		goto out;
+	ASSERT_EQ(topts.retval, THROW_COOKIE, "retval");
+	ASSERT_EQ(skel->bss->pads_ran, RAN_LIGHT, "pads_ran");
+out:
+	exceptions_cleanup_light_lskel__destroy(skel);
+}
+
 static void test_shapes(void)
 {
 	struct exceptions_cleanup_shapes *skel;
@@ -387,6 +412,9 @@ void test_exceptions_cleanup(void)
 
 	exceptions_cleanup__destroy(skel);
 
+	if (test__start_subtest("light_skeleton"))
+		test_light_skeleton();
+
 	test_shapes();
 
 	RUN_TESTS(exceptions_cleanup_fail);
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +39 -0
new file mode 100644
index 000000000000..de7ae478d67f
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "exceptions_cleanup.h"
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+	bpf_throw(0);
+	bpf_preempt_disable();
+	bpf_preempt_enable();
+	bpf_unwind_resume();
+}
+
+__u64 pads_ran = 0;
+
+SEC("syscall")
+__naked int entry_light(void)
+{
+	asm volatile (
+	"call bpf_preempt_disable;"
+	"r1 = %[cookie];"
+"1:"	"call bpf_throw;"		/* cleanup region */
+"2:"
+	"exit;"
+"3:"					/* landing pad */
+	"call bpf_preempt_enable;"
+	PAD_RAN("%[ran]")
+	"call bpf_unwind_resume;"
+	"exit;"
+	CLEANUP_REC("1b", "2b", "3b")
+	:
+	: [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_LIGHT),
+	  __imm_addr(pads_ran)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
Signature · 1 line
2.53.0-Meta