Skip to contentSkip to conversation

bpf: Run exception cleanup landing pads when bpf_throw() unwinds

PATCH Superseded Yonghong Song 4 participants 56 messages Sep 18 – Sep 19

Subsystems (9)
x86 Kselftest BPF arm64 +5 more subsystems

v2 of 12 Superseded Latest is v12 →

lore
Share & tools
lore

Yonghong Song

author reviewer
#0 lore raw

v2 cover bpf: Run exception cleanup landing pads when bpf_throw() unwinds

bpf_throw() walks the BPF call stack to the exception boundary and discards every frame in between. A frame that owns something -- an RCU read lock, a preemption-disabled section, a referenced kptr -- never gets to give it back, so the verifier refuses to let such a frame throw at all. That is the whole reason a Rust program cannot use bpf_throw() as its panic path right now: Rust's Drop glue *is* that give-back, and there is nowhere to run it.

LLVM 23 added the compiler half ([1]). A Rust function that owns a value across a call that can unwind

    fn foo() {
        let _guard = RcuReadGuard::new();  /* bpf_rcu_read_lock()   */
        may_throw();                       /* extern "C-unwind"     */
    }                                      /* Drop: rcu_read_unlock */

lowers to an invoke with a cleanup landing pad holding the Drop call, and the BPF backend writes one record per invoke region into a .bpf_cleanup
section: a flat table of 12-byte (begin, end, landing_pad) triples, each field a byte offset into the code section. The rule is "if a frame unwinds with its return address in [begin, end), run landing_pad before discarding it". A pad ends with a call to _Unwind_Resume(), which the kernel provides as the bpf_unwind_resume() kfunc.

This series is the kernel half: take that table at BPF_PROG_LOAD, teach the verifier that a covered call can also go to its landing pad, and have bpf_throw() run the pads as it walks.

C has no unwinding, so the selftests spell out by hand what a frontend emits -- a call site bracketed by two labels, a landing pad, and a record tying them together. The frame above, written that way:

"call bpf_rcu_read_lock;"
"1:" "call foo3;" /* cleanup region */
"2:"
... normal path, ends in bpf_rcu_read_unlock ...
"6:" /* landing pad */
"call bpf_rcu_read_unlock;"
"call bpf_unwind_resume;" CLEANUP_REC("1b", "2b", "6b")

Right now that program does not load: the throw inside foo3 is reported as "bpf_throw cannot be used inside bpf_rcu_read_lock-ed region", because as far as the verifier is concerned nothing will ever unlock. With the series the verifier walks the unwind the same way the run time will -- into the pad, which unlocks, then on to the next frame -- and the program both loads and releases the lock when it throws.

Design
======

A pad is run, not lowered. bpf_throw() already walks the frames with arch_bpf_stack_walk(); it now looks each frame's return address up in that (sub)program's table and calls the pad as a subroutine of the walker, with the unwinding frame's frame pointer and its callee-saved registers restored from the spill its callee's prologue left. The pad therefore sees its own frame but runs on the walker's stack, far below it, so nothing it calls can disturb the frame it is cleaning up after. The JIT turns its bpf_unwind_resume() into the way back to the walker.

The verifier walks the same thing, step for step, so the resource rules are unchanged: whatever a pad releases is released in the verifier state too, and check_resource_leak() simply moves from "a throw was seen" to the end of the walk.

1-2 uapi: cleanup_info in BPF_PROG_LOAD, struct bpf_cleanup_info,
and the bpf_unwind_resume() kfunc
3-9 verifier: mark the covered call sites, give them an edge to the
pad, walk the unwind, and refuse the shapes that cannot be dispatched

  10     bpf_throw(): dispatch pads while walking
  11-12  x86-64 and arm64 JITs
  13-17  libbpf: collect .bpf_cleanup, pass it to the kernel, resolve

_Unwind_Resume, carry it through the light skeleton and the static linker
18-20 selftests

Limitations
===========

- Cleanup pads only. A catch pad -- one that ends in a plain exit rather than a resume, which is what Rust's catch_unwind would need -- is
refused: the walker calls a pad as a subroutine and cannot hand a frame back its own execution. LLVM refuses type-specific catches and filters on its side as well.
- A JIT that can dispatch pads is required: x86-64 (with CONFIG_UNWINDER_ORC, which bpf_throw() already needs there) and arm64. Anywhere else the load fails with -EOPNOTSUPP rather than silently doing nothing.
- No offloaded programs, no private stack, and no combining a table with an exception callback.
- In a pad body: no tail call, no indirect jump, and no on-stack call arguments -- all of them read or write a stack the pad does not own.
- The Rust toolchain does not properly support BPF exception handling yet. The tables the selftests use are hand-written inline asm, which the assembler turns into the same relocations the BPF AsmPrinter emits, so libbpf and the kernel see an object indistinguishable from a compiler-generated one.

[1] https://github.com/llvm/llvm-project/pull/192164
llvm commit 9d51c891b719 ("[BPF] Add exception handling support with .bpf_cleanup section")

Changelog
=========
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Consolidate all usages of kern_extern_name() in a single patch in libbpf.
- Avoid compiler warning and add proper cleanup_info_cnt guard in libbpf when collecting .bpf_cleanup records.
- Add cleanup_info_cnt condition for emit_rel_store() with cleanup_info.

Yonghong Song (20):
bpf: Accept the compiler's exception cleanup table at program load
bpf: Add the bpf_unwind_resume() kfunc
bpf: Add lookups for exception cleanup resumes and landing pads
bpf: Mark the call sites an exception cleanup table covers
bpf: Make exception landing pads reachable in the CFG
bpf: Explore the landing pads no call site reaches
bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch
bpf: Walk the exception unwind in the verifier
bpf: Refuse a private stack for a program with an exception cleanup
table
bpf: Dispatch exception cleanup pads from bpf_throw()
bpf, x86: Dispatch exception cleanup pads at run time
bpf, arm64: Dispatch exception cleanup pads at run time
libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
libbpf: Add cleanup_info to bpf_prog_load_opts
libbpf: Collect .bpf_cleanup records and pass them to the kernel
libbpf: Carry the exception cleanup table through the light skeleton
libbpf: Let the static linker carry .bpf_cleanup relocations selftests/bpf: Add an end-to-end .bpf_cleanup exception test selftests/bpf: Cover the exception cleanup shapes the chain does not
reach
selftests/bpf: Load an exception cleanup program from a light skeleton

 arch/arm64/net/Makefile                       |   2 +-
 arch/arm64/net/bpf_cleanup_pad.S              |  95 ++
 arch/arm64/net/bpf_jit_comp.c                 |  97 +-
 arch/x86/net/Makefile                         |   2 +-
 arch/x86/net/bpf_cleanup_pad.S                |  74 ++
 arch/x86/net/bpf_jit_comp.c                   |  78 +-
 include/linux/bpf.h                           |  75 ++
 include/linux/bpf_cleanup_abi.h               |  16 +
 include/linux/bpf_verifier.h                  |  11 +
 include/linux/filter.h                        |   2 +
 include/uapi/linux/bpf.h                      |   9 +
 kernel/bpf/Makefile                           |   2 +-
 kernel/bpf/cfg.c                              |  64 +-
 kernel/bpf/check_btf.c                        | 147 +++
 kernel/bpf/core.c                             |  35 +-
 kernel/bpf/exception.c                        | 654 +++++++++++++
 kernel/bpf/exception.h                        |  22 +
 kernel/bpf/fixups.c                           | 143 +++
 kernel/bpf/helpers.c                          |  46 +
 kernel/bpf/liveness.c                         |  20 +
 kernel/bpf/states.c                           |   3 +
 kernel/bpf/syscall.c                          |   2 +-
 kernel/bpf/verifier.c                         | 125 ++-
 tools/include/uapi/linux/bpf.h                |   9 +
 tools/lib/bpf/bpf.c                           |   6 +-
 tools/lib/bpf/bpf.h                           |   7 +-
 tools/lib/bpf/gen_loader.c                    |  29 +-
 tools/lib/bpf/libbpf.c                        | 312 ++++++-
 tools/lib/bpf/libbpf_internal.h               |  10 +
 tools/lib/bpf/linker.c                        |  19 +-
 tools/testing/selftests/bpf/Makefile          |   2 +-
 .../selftests/bpf/exceptions_cleanup.h        |  52 ++
 .../bpf/prog_tests/exceptions_cleanup.c       | 421 +++++++++
 .../selftests/bpf/progs/exceptions_cleanup.c  | 152 +++
 .../bpf/progs/exceptions_cleanup_ext_table.c  |  48 +
 .../bpf/progs/exceptions_cleanup_fail.c       | 600 ++++++++++++
 .../bpf/progs/exceptions_cleanup_freplace.c   |  17 +
 .../bpf/progs/exceptions_cleanup_light.c      |  39 +
 .../progs/exceptions_cleanup_pad_freplace.c   |  17 +
 .../bpf/progs/exceptions_cleanup_shapes.c     | 863 ++++++++++++++++++
 40 files changed, 4263 insertions(+), 64 deletions(-)
 create mode 100644 arch/arm64/net/bpf_cleanup_pad.S
 create mode 100644 arch/x86/net/bpf_cleanup_pad.S
 create mode 100644 include/linux/bpf_cleanup_abi.h
 create mode 100644 kernel/bpf/exception.c
 create mode 100644 kernel/bpf/exception.h
 create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
Signature · 1 line
2.53.0-Meta
author reviewer
#1 lore raw

PATCH v2 1/20 bpf: Accept the compiler's exception cleanup table at program load

LLVM 23 added exception handling support for BPF with the .bpf_cleanup section ([1]). Rust code compiled with panic=unwind runs cleanup code (Drop glue) when bpf_throw() fires, and the LLVM BPF backend emits that section from the landing pads the frontend produced. Plain C cannot generate .bpf_cleanup unless inline asm is used. The Rust compiler does not *properly* support BPF exception handling yet, but the kernel can support the table today, and inline assembly is enough to test it.

Add the UAPI to carry the .bpf_cleanup table into the kernel. BPF_PROG_LOAD grows cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, and struct bpf_cleanup_info describes one record as a triple of instruction indices:
the half-open call-site range [begin_off, end_off) and the landing_pad_off the frame resumes at. check_cleanup_info() validates the table a program is loaded with, so the rest of the kernel can rely on it.

[1] https://github.com/llvm/llvm-project/pull/192164

---
 include/linux/bpf_verifier.h   |   2 +
 include/uapi/linux/bpf.h       |   9 ++
 kernel/bpf/check_btf.c         | 147 +++++++++++++++++++++++++++++++++
 kernel/bpf/syscall.c           |   2 +-
 kernel/bpf/verifier.c          |   1 +
 tools/include/uapi/linux/bpf.h |   9 ++
 6 files changed, 169 insertions(+), 1 deletion(-)
include/linux/bpf_verifier.h +2 -0
index cf85141ea167..c08505b9ba82 100644
@@ -987,6 +987,8 @@ struct bpf_verifier_env {
 	struct arg_track **callsite_at_stack;
 	u32 pass_cnt; /* number of times do_check() was called */
 	u32 subprog_cnt;
+	struct bpf_cleanup_info *cleanup_info;
+	u32 cleanup_info_cnt;
 	/* number of instructions analyzed by the verifier */
 	u32 prev_insn_processed, insn_processed;
 	/* number of jmps, calls, exits analyzed so far */
include/uapi/linux/bpf.h +9 -0
index 732b35cc08d1..f7dc121be094 100644
@@ -1669,6 +1669,9 @@ union bpf_attr {
 		 * verification.
 		 */
 		__s32		keyring_id;
+		__aligned_u64	cleanup_info;	/* exception cleanup table */
+		__u32		cleanup_info_rec_size; /* userspace bpf_cleanup_info size */
+		__u32		cleanup_info_cnt; /* number of bpf_cleanup_info records */
 	};
 
 	struct { /* anonymous struct used by BPF_OBJ_* commands */
@@ -7588,6 +7591,12 @@ struct bpf_line_info {
 	__u32	line_col;
 };
 
+struct bpf_cleanup_info {
+	__u32	begin_off;
+	__u32	end_off;
+	__u32	landing_pad_off;
+};
+
 struct bpf_spin_lock {
 	__u32	val;
 };
kernel/bpf/check_btf.c +147 -0

159 lines not on this page · raw message

kernel/bpf/syscall.c +1 -1
index def57bddb092..ac7091469766 100644
@@ -2912,7 +2912,7 @@ int __init __used bpf_multi_func(void) { return 0; }
 BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
 
 /* last field in 'union bpf_attr' used by this command */
-#define BPF_PROG_LOAD_LAST_FIELD keyring_id
+#define BPF_PROG_LOAD_LAST_FIELD cleanup_info_cnt
 
 static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
 {
kernel/bpf/verifier.c +1 -0
index 6c6b8d8520cd..c65ff2e326bf 100644
@@ -21845,6 +21845,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	kvfree(env->succ);
 	kvfree(env->gotox_tmp_buf);
 	bpf_diag_free(env);
+	kvfree(env->cleanup_info);
 	kvfree(env);
 	return ret;
 }
tools/include/uapi/linux/bpf.h +9 -0
index 732b35cc08d1..f7dc121be094 100644
@@ -1669,6 +1669,9 @@ union bpf_attr {
 		 * verification.
 		 */
 		__s32		keyring_id;
+		__aligned_u64	cleanup_info;	/* exception cleanup table */
+		__u32		cleanup_info_rec_size; /* userspace bpf_cleanup_info size */
+		__u32		cleanup_info_cnt; /* number of bpf_cleanup_info records */
 	};
 
 	struct { /* anonymous struct used by BPF_OBJ_* commands */
@@ -7588,6 +7591,12 @@ struct bpf_line_info {
 	__u32	line_col;
 };
 
+struct bpf_cleanup_info {
+	__u32	begin_off;
+	__u32	end_off;
+	__u32	landing_pad_off;
+};
+
 struct bpf_spin_lock {
 	__u32	val;
 };
Signature · 1 line
2.53.0-Meta
author reviewer
#2 lore raw

PATCH v2 2/20 bpf: Add the bpf_unwind_resume() kfunc

In reply to Yonghong Song

A compiler-emitted cleanup landing pad ends with a call to _Unwind_Resume(), which carries the unwind on once that frame's cleanups have run. The kernel provides the same terminator as a kfunc, named bpf_unwind_resume() to keep the 'bpf_' prefix kfunc convention; a later libbpf patch resolves the compiler's name to it.

The body never runs. A JIT emits the way back out of a landing pad in its place.

---
 kernel/bpf/helpers.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
kernel/bpf/helpers.c +12 -0
index 051b6654e57c..31d84de4d318 100644
@@ -3407,6 +3407,17 @@ __bpf_kfunc void bpf_throw(u64 cookie)
 	WARN(1, "A call to BPF exception callback should never return\n");
 }
 
+/*
+ * Terminator of a compiler-emitted cleanup landing pad. The compiler names
+ * this _Unwind_Resume, the base unwind ABI's entry point for carrying an
+ * unwind on once a frame's cleanups have run. To match kernel kfunc
+ * convention, the kernel calls it bpf_unwind_resume and libbpf maps the
+ * compiler's name onto it.
+ */
+__bpf_kfunc void bpf_unwind_resume(void)
+{
+}
+
 __bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, unsigned int flags)
 {
 	struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
@@ -4853,6 +4864,7 @@ BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_throw)
+BTF_ID_FLAGS(func, bpf_unwind_resume)
 #ifdef CONFIG_BPF_EVENTS
 BTF_ID_FLAGS(func, bpf_send_signal_task)
 #endif
Signature · 1 line
2.53.0-Meta
author reviewer
#3 lore raw

PATCH v2 3/20 bpf: Add lookups for exception cleanup resumes and landing pads

In reply to Yonghong Song

Add two new files, exception.h and exception.c, to host the exception handling code. Only a few helpers so far: recognising a call to bpf_unwind_resume(), and asking which landing pad, if any, a call site unwinds to.

The pad of a call site is kept in insn_aux_data, so the two places that move instructions around -- bpf_patch_insn_data() and verifier_remove_insns() -- learn to keep it in step. Nothing sets the mark yet; the next patch does.

---
 include/linux/bpf_verifier.h |  6 ++++++
 kernel/bpf/Makefile          |  2 +-
 kernel/bpf/exception.c       | 36 ++++++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h       | 12 ++++++++++++
 kernel/bpf/fixups.c          | 17 +++++++++++++++++
 5 files changed, 72 insertions(+), 1 deletion(-)
 create mode 100644 kernel/bpf/exception.c
 create mode 100644 kernel/bpf/exception.h
include/linux/bpf_verifier.h +6 -0
index c08505b9ba82..f9bccd3e0f4d 100644
@@ -681,6 +681,11 @@ struct bpf_insn_aux_data {
 	bool needs_zext; /* alu op needs to clear upper bits */
 	bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
 	bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
+	/*
+	 * 1 + the instruction index of the exception cleanup landing pad this
+	 * call site unwinds to, or 0 for none.
+	 */
+	u32 cleanup_pad;
 	bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
 	u8 alu_state; /* used in combination with alu_limit */
 	/* true if STX or LDX instruction is a part of a spill/fill
@@ -1518,6 +1523,7 @@ u32 btf_func_arg_align(const struct btf *btf, const struct btf_type *t);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn);
 int bpf_compute_const_regs(struct bpf_verifier_env *env);
 int bpf_prune_dead_branches(struct bpf_verifier_env *env);
 int bpf_check_cfg(struct bpf_verifier_env *env);
kernel/bpf/Makefile +1 -1
index 9a92c348bbda..af9bc60428ad 100644
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o exception.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
kernel/bpf/exception.c new file +36 -0
new file mode 100644
index 000000000000..3895b9453639
@@ -0,0 +1,36 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
+#include <linux/slab.h>
+#include "exception.h"
+
+#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
+
+enum exc_kfunc {
+	EXC_KF_bpf_unwind_resume,
+};
+
+BTF_ID_LIST(exc_kfunc_list)
+BTF_ID(func, bpf_unwind_resume)
+
+static bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf)
+{
+	return bpf_pseudo_kfunc_call(insn) && insn->off == 0 &&
+	       insn->imm == exc_kfunc_list[kf];
+}
+
+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn)
+{
+	return insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume);
+}
+
+int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx)
+{
+	u32 pad = env->insn_aux_data[idx].cleanup_pad;
+
+	return pad ? (int)pad - 1 : -1;
+}
kernel/bpf/exception.h new file +12 -0
new file mode 100644
index 000000000000..0f2b9624a2ce
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef _LINUX_BPF_EXCEPTION_H
+#define _LINUX_BPF_EXCEPTION_H
+
+#include <linux/types.h>
+
+struct bpf_verifier_env;
+
+int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx);
+
+#endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +17 -0
index 2add8001c3ec..82b00fac6bd6 100644
@@ -261,6 +261,11 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 		}
 	}
 
+	if (env->cleanup_info_cnt)
+		for (i = 0; i < prog_len; i++)
+			if (data[i].cleanup_pad > off + 1)
+				data[i].cleanup_pad += cnt - 1;
+
 	/*
 	 * Last slot instruction could be a newly generated
 	 * BPF_ST/BPF_LDX/BPF_STX, systematically mark it for non-stack access
@@ -549,6 +554,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 	struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
 	unsigned int orig_prog_len = env->prog->len;
 	int err;
+	u32 i;
 
 	if (bpf_prog_is_offloaded(env->prog->aux))
 		bpf_prog_offload_remove_insns(env, off, cnt);
@@ -573,6 +579,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 		sizeof(*aux_data) * (orig_prog_len - off - cnt));
 	env->insn_aux_data_len -= cnt;
 
+	if (env->cleanup_info_cnt) {
+		for (i = 0; i < env->insn_aux_data_len; i++) {
+			u32 pad = aux_data[i].cleanup_pad;
+
+			if (pad > off + cnt)
+				aux_data[i].cleanup_pad = pad - cnt;
+			else if (pad > off)
+				aux_data[i].cleanup_pad = 0;
+		}
+	}
+
 	return 0;
 }
 
Signature · 1 line
2.53.0-Meta
#4 lore raw

Re: [PATCH bpf-next v2 03/20] bpf: Add lookups for exception cleanup resumes and landing pads

Can this logic handle NOP removal correctly? When bpf_opt_remove_nops() calls verifier_remove_insns(env, i, 1) to remove a single fall-through goto +0, if a cleanup record names that instruction as…
Quoted 15 lines
include/linux/bpf_verifier.h +5 -0
index c08505b9ba821..f9bccd3e0f4db 100644
@@ -681,6 +681,11 @@ struct bpf_insn_aux_data {
 	bool needs_zext; /* alu op needs to clear upper bits */
 	bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
 	bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
+	/*
+	 * 1 + the instruction index of the exception cleanup landing pad this
+	 * call site unwinds to, or 0 for none.
+	 */
+	u32 cleanup_pad;
 	bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
 	u8 alu_state; /* used in combination with alu_limit */

[ ... ]

Quoted 13 lines
kernel/bpf/fixups.c +5 -0
index 2add8001c3ec3..82b00fac6bd6a 100644
@@ -261,6 +261,11 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 		}
 	}
 
+	if (env->cleanup_info_cnt)
+		for (i = 0; i < prog_len; i++)
+			if (data[i].cleanup_pad > off + 1)
+				data[i].cleanup_pad += cnt - 1;
+

[ ... ]

Quoted 15 lines
kernel/bpf/fixups.c +4 -0
@@ -549,6 +554,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 	struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
 	unsigned int orig_prog_len = env->prog->len;
 	int err;
+	u32 i;
 
 	if (bpf_prog_is_offloaded(env->prog->aux))
 		bpf_prog_offload_remove_insns(env, off, cnt);
@@ -573,6 +579,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 		sizeof(*aux_data) * (orig_prog_len - off - cnt));
 	env->insn_aux_data_len -= cnt;
 
+	if (env->cleanup_info_cnt) {
+		for (i = 0; i < env->insn_aux_data_len; i++) {
+			u32 pad = aux_data[i].cleanup_pad;

kernel/bpf/fixups.c static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) · line 594

Quoted 10 lines
kernel/bpf/fixups.c +8 -0
+
+			if (pad > off + cnt)
+				aux_data[i].cleanup_pad = pad - cnt;
+			else if (pad > off)
+				aux_data[i].cleanup_pad = 0;
+		}
+	}
+
 	return 0;
 }

Can this logic handle NOP removal correctly? When bpf_opt_remove_nops() calls verifier_remove_insns(env, i, 1) to remove a single fall-through goto +0, if a cleanup record names that instruction as its landing pad, then pad == off + 1, so pad > off is true and cleanup_pad becomes 0.

But removing the NOP leaves the pad's next instruction at the same index, so the correct landing pad index stays unchanged rather than being zeroed. For example:

    off:     some_call with cleanup_pad = off+1
    off+1:   goto +0  (NOP landing pad)
    off+2:   actual cleanup code

After removing the NOP at off+1:

off: some_call (cleanup_pad should still be off+1)
off+1: actual cleanup code (was at off+2)

The else if (pad > off) arm erases the record instead of preserving it. Nothing rejects NOP landing pads during validation - a goto +0 has code 0x05, not 0x00, so check_cleanup_info() accepts it.

The ordering makes the loss escape validation: bpf_check_cleanup_exceptions() runs before bpf_opt_remove_nops(), so the verifier proves the cleanup path releases acquired references, but then the NOP removal zeroes cleanup_pad, and later cleanup_info_for_subprog() sees if (!pad) continue; and omits the record from the runtime table. The program is accepted but bpf_throw() finds no pad for that call site and skips the cleanup, leaking whatever the pad was supposed to release.

Does the else if (pad > off) arm need to distinguish removals that consume the landing pad instruction itself versus removals that happen before it?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

#5 lore raw

Re: [PATCH bpf-next v2 03/20] bpf: Add lookups for exception cleanup resumes and landing pads

Quoted 90 lines · bot+bpf-ci
Quoted 15 lines
include/linux/bpf_verifier.h +5 -0
index c08505b9ba821..f9bccd3e0f4db 100644
@@ -681,6 +681,11 @@ struct bpf_insn_aux_data {
 	bool needs_zext; /* alu op needs to clear upper bits */
 	bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
 	bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
+	/*
+	 * 1 + the instruction index of the exception cleanup landing pad this
+	 * call site unwinds to, or 0 for none.
+	 */
+	u32 cleanup_pad;
 	bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
 	u8 alu_state; /* used in combination with alu_limit */

[ ... ]

Quoted 13 lines
kernel/bpf/fixups.c +5 -0
index 2add8001c3ec3..82b00fac6bd6a 100644
@@ -261,6 +261,11 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 		}
 	}
 
+	if (env->cleanup_info_cnt)
+		for (i = 0; i < prog_len; i++)
+			if (data[i].cleanup_pad > off + 1)
+				data[i].cleanup_pad += cnt - 1;
+

[ ... ]

Quoted 15 lines
kernel/bpf/fixups.c +4 -0
@@ -549,6 +554,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 	struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
 	unsigned int orig_prog_len = env->prog->len;
 	int err;
+	u32 i;
 
 	if (bpf_prog_is_offloaded(env->prog->aux))
 		bpf_prog_offload_remove_insns(env, off, cnt);
@@ -573,6 +579,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 		sizeof(*aux_data) * (orig_prog_len - off - cnt));
 	env->insn_aux_data_len -= cnt;
 
+	if (env->cleanup_info_cnt) {
+		for (i = 0; i < env->insn_aux_data_len; i++) {
+			u32 pad = aux_data[i].cleanup_pad;

kernel/bpf/fixups.c static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) · line 594

Quoted 10 lines
kernel/bpf/fixups.c +8 -0
+
+			if (pad > off + cnt)
+				aux_data[i].cleanup_pad = pad - cnt;
+			else if (pad > off)
+				aux_data[i].cleanup_pad = 0;
+		}
+	}
+
 	return 0;
 }

Can this logic handle NOP removal correctly? When bpf_opt_remove_nops() calls verifier_remove_insns(env, i, 1) to remove a single fall-through goto +0, if a cleanup record names that instruction as its landing pad, then pad == off + 1, so pad > off is true and cleanup_pad becomes 0.

But removing the NOP leaves the pad's next instruction at the same index, so the correct landing pad index stays unchanged rather than being zeroed. For example:

    off:     some_call with cleanup_pad = off+1
    off+1:   goto +0  (NOP landing pad)
    off+2:   actual cleanup code

After removing the NOP at off+1:

off: some_call (cleanup_pad should still be off+1)
off+1: actual cleanup code (was at off+2)

The else if (pad > off) arm erases the record instead of preserving it. Nothing rejects NOP landing pads during validation - a goto +0 has code 0x05, not 0x00, so check_cleanup_info() accepts it.

The ordering makes the loss escape validation: bpf_check_cleanup_exceptions() runs before bpf_opt_remove_nops(), so the verifier proves the cleanup path releases acquired references, but then the NOP removal zeroes cleanup_pad, and later cleanup_info_for_subprog() sees if (!pad) continue; and omits the record from the runtime table. The program is accepted but bpf_throw() finds no pad for that call site and skips the cleanup, leaking whatever the pad was supposed to release.

Quoted 3 lines

Does the else if (pad > off) arm need to distinguish removals that consume the landing pad instruction itself versus removals that happen before it?

bot is correct here.
pls fix

pw-bot: cr

author reviewer
#6 lore raw

Re: [PATCH bpf-next v2 03/20] bpf: Add lookups for exception cleanup resumes and landing pads

Quoted 88 lines · Alexei Starovoitov
Quoted 87 lines · bot+bpf-ci
Quoted 15 lines
include/linux/bpf_verifier.h +5 -0
index c08505b9ba821..f9bccd3e0f4db 100644
@@ -681,6 +681,11 @@ struct bpf_insn_aux_data {
  	bool needs_zext; /* alu op needs to clear upper bits */
  	bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
  	bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
+	/*
+	 * 1 + the instruction index of the exception cleanup landing pad this
+	 * call site unwinds to, or 0 for none.
+	 */
+	u32 cleanup_pad;
  	bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
  	u8 alu_state; /* used in combination with alu_limit */

[ ... ]

Quoted 13 lines
kernel/bpf/fixups.c +5 -0
index 2add8001c3ec3..82b00fac6bd6a 100644
@@ -261,6 +261,11 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
  		}
  	}
 
+	if (env->cleanup_info_cnt)
+		for (i = 0; i < prog_len; i++)
+			if (data[i].cleanup_pad > off + 1)
+				data[i].cleanup_pad += cnt - 1;
+

[ ... ]

Quoted 15 lines
kernel/bpf/fixups.c +4 -0
@@ -549,6 +554,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
  	struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
  	unsigned int orig_prog_len = env->prog->len;
  	int err;
+	u32 i;
 
  	if (bpf_prog_is_offloaded(env->prog->aux))
  		bpf_prog_offload_remove_insns(env, off, cnt);
@@ -573,6 +579,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
  		sizeof(*aux_data) * (orig_prog_len - off - cnt));
  	env->insn_aux_data_len -= cnt;
 
+	if (env->cleanup_info_cnt) {
+		for (i = 0; i < env->insn_aux_data_len; i++) {
+			u32 pad = aux_data[i].cleanup_pad;

kernel/bpf/fixups.c static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) · line 594

Quoted 10 lines
kernel/bpf/fixups.c +8 -0
+
+			if (pad > off + cnt)
+				aux_data[i].cleanup_pad = pad - cnt;
+			else if (pad > off)
+				aux_data[i].cleanup_pad = 0;
+		}
+	}
+
  	return 0;
  }

Can this logic handle NOP removal correctly? When bpf_opt_remove_nops() calls verifier_remove_insns(env, i, 1) to remove a single fall-through goto +0, if a cleanup record names that instruction as its landing pad, then pad == off + 1, so pad > off is true and cleanup_pad becomes 0.

But removing the NOP leaves the pad's next instruction at the same index, so the correct landing pad index stays unchanged rather than being zeroed. For example:

     off:     some_call with cleanup_pad = off+1
     off+1:   goto +0  (NOP landing pad)
     off+2:   actual cleanup code

After removing the NOP at off+1:

off: some_call (cleanup_pad should still be off+1)
off+1: actual cleanup code (was at off+2)

The else if (pad > off) arm erases the record instead of preserving it. Nothing rejects NOP landing pads during validation - a goto +0 has code 0x05, not 0x00, so check_cleanup_info() accepts it.

The ordering makes the loss escape validation: bpf_check_cleanup_exceptions() runs before bpf_opt_remove_nops(), so the verifier proves the cleanup path releases acquired references, but then the NOP removal zeroes cleanup_pad, and later cleanup_info_for_subprog() sees if (!pad) continue; and omits the record from the runtime table. The program is accepted but bpf_throw() finds no pad for that call site and skips the cleanup, leaking whatever the pad was supposed to release.

Quoted 5 lines
Quoted 3 lines

Does the else if (pad > off) arm need to distinguish removals that consume the landing pad instruction itself versus removals that happen before it?

bot is correct here.
pls fix

Agree with bot about for two different cases. Will fix.

Quoted 2 lines

pw-bot: cr

author reviewer
#7 lore raw

PATCH v2 4/20 bpf: Mark the call sites an exception cleanup table covers

In reply to Yonghong Song

Some plumbing work is done before bpf_check_cfg(). More specifically, insn_aux_data records cleanup_throw_site for every bpf_throw(), and cleanup_pad, the landing pad a frame resumes at, for every call within the [begin_off, end_off) range of a cleanup record. Subsequent commits consume both.

bpf_prepare_cleanup_exceptions() runs before bpf_check_cfg(), because what it produces is what the CFG walk consumes. It refuses a table on an offloaded program, on a program whose JIT cannot dispatch landing pads or which the JIT was not asked to compile, and on a program that also installs an exception callback -- two different answers to what runs on the way out. bpf_jit_supports_cleanup_pads() is weak here and says no; the arch patches provide the real ones.

---
 include/linux/bpf_verifier.h |  1 +
 include/linux/filter.h       |  1 +
 kernel/bpf/core.c            |  5 ++++
 kernel/bpf/exception.c       | 55 ++++++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h       |  1 +
 kernel/bpf/verifier.c        |  6 ++++
 6 files changed, 69 insertions(+)
include/linux/bpf_verifier.h +1 -0
index f9bccd3e0f4d..7463e86d15ee 100644
@@ -681,6 +681,7 @@ struct bpf_insn_aux_data {
 	bool needs_zext; /* alu op needs to clear upper bits */
 	bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
 	bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
+	bool cleanup_throw_site; /* call to bpf_throw() */
 	/*
 	 * 1 + the instruction index of the exception cleanup landing pad this
 	 * call site unwinds to, or 0 for none.
include/linux/filter.h +1 -0
index b17222db2efc..9682b98ad890 100644
@@ -1242,6 +1242,7 @@ bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_arena_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
+bool bpf_jit_supports_cleanup_pads(void);
 bool bpf_jit_supports_ptr_xchg(void);
 bool bpf_jit_supports_arena(void);
 bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
kernel/bpf/core.c +5 -0
index 4e208cc94752..f0dd851264f4 100644
@@ -3470,6 +3470,11 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp,
 {
 }
 
+bool __weak bpf_jit_supports_cleanup_pads(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_timed_may_goto(void)
 {
 	return false;
kernel/bpf/exception.c +55 -0

61 lines not on this page · raw message

kernel/bpf/exception.h +1 -0
index 0f2b9624a2ce..f51383fd775c 100644
@@ -7,6 +7,7 @@
 
 struct bpf_verifier_env;
 
+int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env);
 int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx);
 
 #endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/verifier.c +6 -0
index c65ff2e326bf..6496c30a10f7 100644
@@ -37,6 +37,7 @@
 
 #include "diagnostics.h"
 #include "disasm.h"
+#include "exception.h"
 
 static const struct bpf_verifier_ops * const bpf_verifier_ops[] = {
 #define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
@@ -21638,6 +21639,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/* The CFG needs an edge from a call in a cleanup range to its pad. */
+	ret = bpf_prepare_cleanup_exceptions(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	/* Validate instructions and resolve the program's referenced resources. */
 	ret = check_and_resolve_insns(env);
 	if (ret < 0)
Signature · 1 line
2.53.0-Meta
author reviewer
#8 lore raw

PATCH v2 5/20 bpf: Make exception landing pads reachable in the CFG

In reply to Yonghong Song

Both the CFG walk and liveness are involved. A bpf_throw() or a bpf2bpf call inside the [begin_off, end_off) range of a cleanup record can reach that record's landing pad, so both grow that edge; and a bpf_unwind_resume() reaches nothing after it, so it has no successors.

Liveness needs one more thing. bpf_stack_slot_alive() decides whether an outer frame's stack slot is still read after the call the frame is suspended at by looking at the instruction after the call. A slot whose only remaining reader is the landing pad -- which is every slot a compiler-generated pad reloads, since only four registers survive a call -- is dead by that measure, so clean_verifier_state() poisons it while the callee runs and the pad is then rejected for reading it. Ask about the pad as well when the call site names one.

---
 kernel/bpf/cfg.c      | 47 ++++++++++++++++++++++++++++++++++++++++---
 kernel/bpf/liveness.c | 20 ++++++++++++++++++
 2 files changed, 64 insertions(+), 3 deletions(-)
kernel/bpf/cfg.c +44 -3
index 842c7d1eabcc..9f8b8b54d5ea 100644
@@ -6,6 +6,7 @@
 #include <linux/sort.h>
 
 #include "diagnostics.h"
+#include "exception.h"
 
 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
 
@@ -158,17 +159,57 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
 	return DONE_EXPLORING;
 }
 
+static int visit_cleanup_pad_edge(int t, struct bpf_verifier_env *env)
+{
+	int *insn_stack = env->cfg.insn_stack;
+	int *insn_state = env->cfg.insn_state;
+	int w;
+
+	if (!env->cleanup_info_cnt)
+		return DONE_EXPLORING;
+	w = bpf_cleanup_pad_of_call(env, t);
+	if (w < 0)
+		return DONE_EXPLORING;
+
+	mark_prune_point(env, t);
+	mark_jmp_point(env, w);
+	mark_jump_target(env, w);
+
+	if (insn_state[w])
+		return DONE_EXPLORING;
+	if (env->cfg.cur_stack >= env->prog->len)
+		return -E2BIG;
+	insn_stack[env->cfg.cur_stack++] = w;
+	insn_state[w] |= DISCOVERED;
+	return KEEP_EXPLORING;
+}
+
+static int merge_visit_ret(int a, int b)
+{
+	if (a < 0)
+		return a;
+	if (b < 0)
+		return b;
+	if (a == KEEP_EXPLORING || b == KEEP_EXPLORING)
+		return KEEP_EXPLORING;
+	return DONE_EXPLORING;
+}
+
 static int visit_func_call_insn(int t, struct bpf_insn *insns,
 				struct bpf_verifier_env *env,
 				bool visit_callee)
 {
-	int ret, insn_sz;
+	int ret, insn_sz, pad_ret;
 	int w;
 
+	pad_ret = visit_cleanup_pad_edge(t, env);
+	if (pad_ret < 0)
+		return pad_ret;
+
 	insn_sz = bpf_is_ldimm64(&insns[t]) ? 2 : 1;
 	ret = push_insn(t, t + insn_sz, FALLTHROUGH, env);
 	if (ret)
-		return ret;
+		return merge_visit_ret(pad_ret, ret);
 
 	mark_prune_point(env, t + insn_sz);
 	/* when we exit from subprog, we need to record non-linear history */
@@ -180,7 +221,7 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,
 		merge_callee_effects(env, t, w);
 		ret = push_insn(t, w, BRANCH, env);
 	}
-	return ret;
+	return merge_visit_ret(pad_ret, ret);
 }
 
 struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem)
kernel/bpf/liveness.c +20 -0
index 44ecdc5b4ec2..9cfd05f970bc 100644
@@ -8,6 +8,8 @@
 #include <linux/slab.h>
 #include <linux/sort.h>
 
+#include "exception.h"
+
 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
 
 struct per_frame_masks {
@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 	succ = env->succ;
 	succ->cnt = 0;
 
+	if (unlikely(bpf_is_unwind_resume_kfunc(insn)))
+		return succ;
+
 	opcode_info = &opcode_info_tbl[BPF_CLASS(insn->code) | BPF_OP(insn->code)];
 	insn_sz = bpf_is_ldimm64(insn) ? 2 : 1;
 	if (opcode_info->can_fallthrough)
@@ -264,6 +269,13 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 	if (opcode_info->can_jump)
 		succ->items[succ->cnt++] = idx + bpf_jmp_offset(insn) + 1;
 
+	if (unlikely(env->cleanup_info_cnt)) {
+		int pad = bpf_cleanup_pad_of_call(env, idx);
+
+		if (pad >= 0)
+			succ->items[succ->cnt++] = pad;
+	}
+
 	return succ;
 }
 
@@ -397,6 +409,14 @@ bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp
 		alive = bpf_calls_callback(env, callsite)
 			? is_live_before(instance, callsite, rel, half_spi)
 			: is_live_before(instance, callsite + 1, rel, half_spi);
+
+		/* Control may also go to the landing pad. */
+		if (!alive && unlikely(env->cleanup_info_cnt)) {
+			int pad = bpf_cleanup_pad_of_call(env, callsite);
+
+			if (pad >= 0)
+				alive = is_live_before(instance, pad, rel, half_spi);
+		}
 		if (alive)
 			return true;
 	}
Signature · 1 line
2.53.0-Meta
#9 lore raw

Re: [PATCH bpf-next v2 05/20] bpf: Make exception landing pads reachable in the CFG

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] bpf: missing prune point at exception landing pad target --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: missing prune point at exception landing pad target
--

commit 4fbd0e36862ba7e3ee8480e535068a6821d5cad0
Author: Yonghong Song <[email protected]>

bpf: Make exception landing pads reachable in the CFG

This commit updates the verifier's control flow graph and liveness analysis to properly handle exception landing pads. It ensures that exception throwing calls can reach cleanup records, and prevents outer frame stack slots from being read incorrectly when control is transferred to landing pads.

Quoted 4 lines
kernel/bpf/cfg.c +0 -0
index 842c7d1eabccc..9f8b8b54d5ea7 100644

[ ... ]

Quoted 9 lines
kernel/bpf/cfg.c +5 -0
@@ -158,17 +159,57 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
 	return DONE_EXPLORING;
 }
 
+static int visit_cleanup_pad_edge(int t, struct bpf_verifier_env *env)
+{
+	int *insn_stack = env->cfg.insn_stack;
+	int *insn_state = env->cfg.insn_state;
+	int w;

kernel/bpf/cfg.c static int push_insn(int t, int w, int e, struct bpf_verifier_env *env) · line 176

Quoted 10 lines
kernel/bpf/cfg.c +10 -0
+
+	if (!env->cleanup_info_cnt)
+		return DONE_EXPLORING;
+	w = bpf_cleanup_pad_of_call(env, t);
+	if (w < 0)
+		return DONE_EXPLORING;
+
+	mark_prune_point(env, t);
+	mark_jmp_point(env, w);
+	mark_jump_target(env, w);

[Severity: High]
Should mark_prune_point() be applied to the landing pad target w instead of the call site t?

Because w is a branch target where control flow merges, it needs to be marked as a prune point so the verifier knows to check for state convergence. By marking t instead of w, the verifier will evaluate the landing pad and all subsequent cleanup instructions independently for every exception-throwing call that branches to w.

Can this lead to combinatorial path explosion and result in the verifier rejecting the program with -E2BIG?

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
author reviewer
#10 lore raw

Re: [PATCH bpf-next v2 05/20] bpf: Make exception landing pads reachable in the CFG

Quoted 43 lines · On 9/17/26 9:59 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: missing prune point at exception landing pad target
--

commit 4fbd0e36862ba7e3ee8480e535068a6821d5cad0
Author: Yonghong Song <[email protected]>

bpf: Make exception landing pads reachable in the CFG

This commit updates the verifier's control flow graph and liveness analysis to properly handle exception landing pads. It ensures that exception throwing calls can reach cleanup records, and prevents outer frame stack slots from being read incorrectly when control is transferred to landing pads.

Quoted 4 lines
kernel/bpf/cfg.c +0 -0
index 842c7d1eabccc..9f8b8b54d5ea7 100644

[ ... ]

Quoted 9 lines
kernel/bpf/cfg.c +5 -0
@@ -158,17 +159,57 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
  	return DONE_EXPLORING;
  }
 
+static int visit_cleanup_pad_edge(int t, struct bpf_verifier_env *env)
+{
+	int *insn_stack = env->cfg.insn_stack;
+	int *insn_state = env->cfg.insn_state;
+	int w;

kernel/bpf/cfg.c static int push_insn(int t, int w, int e, struct bpf_verifier_env *env) · line 176

Quoted 10 lines
kernel/bpf/cfg.c +10 -0
+
+	if (!env->cleanup_info_cnt)
+		return DONE_EXPLORING;
+	w = bpf_cleanup_pad_of_call(env, t);
+	if (w < 0)
+		return DONE_EXPLORING;
+
+	mark_prune_point(env, t);
+	mark_jmp_point(env, w);
+	mark_jump_target(env, w);

[Severity: High]
Should mark_prune_point() be applied to the landing pad target w instead of the call site t?

Quoted 8 lines

Because w is a branch target where control flow merges, it needs to be marked as a prune point so the verifier knows to check for state convergence. By marking t instead of w, the verifier will evaluate the landing pad and all subsequent cleanup instructions independently for every exception-throwing call that branches to w.

Can this lead to combinatorial path explosion and result in the verifier rejecting the program with -E2BIG?

We can add mark_prune_point(env, w) as well, esp. there are cases there are multiple cleanup regions but many (or all) of them may pointing into the same landing pad.

Quoted 3 lines

[ ... ]

#11 lore raw

Re: [PATCH bpf-next v2 05/20] bpf: Make exception landing pads reachable in the CFG

In reply to Yonghong Song
Should this early return be gated on env->cleanup_info_cnt?
Quoted 14 lines
kernel/bpf/liveness.c +2 -0
index 44ecdc5b4ec2d..9cfd05f970bc9 100644
@@ -8,6 +8,8 @@
 #include <linux/slab.h>
 #include <linux/sort.h>
 
+#include "exception.h"
+
 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
 
 struct per_frame_masks {

kernel/bpf/liveness.c bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) · line 263

Quoted 6 lines
kernel/bpf/liveness.c +3 -0
@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 	succ = env->succ;
 	succ->cnt = 0;
 
+	if (unlikely(bpf_is_unwind_resume_kfunc(insn)))
+		return succ;
+

Should this early return be gated on env->cleanup_info_cnt?

bpf_unwind_resume() is an unconditionally registered generic kfunc with no KF_ flag restricting it to programs that carry a cleanup table. Its body is empty, so at run time the call simply returns and execution continues at the next instruction.

visit_insn() in cfg.c still gives the call a fallthrough edge, and at this commit nothing in do_check() rejects a bpf_unwind_resume() call that is not the tail of a cleanup landing pad. So any CAP_BPF program with no cleanup table can call bpf_unwind_resume() and keep executing after it.

For such a program, bpf_insn_successors() reports zero successors, so bpf_compute_live_registers() treats everything after the call as dead. The register-cleaning side fails closed (a poisoned slot is rejected), but the 32-bit zero-extension side fails open:

insn_aux[i].zext_dst for an insn preceding the call is only set when the hi half of the defined register is live-after, so it stays false, and bpf_opt_subreg_zext_lo32_rnd_hi32() (kernel/bpf/fixups.c:706) then does 'if (!aux[adj_idx].zext_dst) { if (!rnd_hi32) continue; ... }' and emits no BPF_ZEXT_REG.

On architectures where bpf_jit_needs_zext() is true and kfunc calls are supported (s390, riscv64, powerpc64, parisc, x86-32), the upper 32 bits of a register the verifier proved to be 32-bit bounded are left with stale JIT-defined contents, which defeats the range checks the verifier derived for it.

Compare with the pad lookup ten lines below:

kernel/bpf/liveness.c bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) · line 277

Quoted 13 lines
kernel/bpf/liveness.c +6 -0
 	opcode_info = &opcode_info_tbl[BPF_CLASS(insn->code) | BPF_OP(insn->code)];
 	insn_sz = bpf_is_ldimm64(insn) ? 2 : 1;
 	if (opcode_info->can_fallthrough)
@@ -264,6 +269,13 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 	if (opcode_info->can_jump)
 		succ->items[succ->cnt++] = idx + bpf_jmp_offset(insn) + 1;
 
+	if (unlikely(env->cleanup_info_cnt)) {
+		int pad = bpf_cleanup_pad_of_call(env, idx);
+
+		if (pad >= 0)
+			succ->items[succ->cnt++] = pad;
+	}

which gates the pad lookup on cleanup_info_cnt, and with visit_cleanup_pad_edge() in kernel/bpf/cfg.c which returns DONE_EXPLORING when !env->cleanup_info_cnt.

Would gating the early return on env->cleanup_info_cnt fix this? That would be sound because a program that carries a cleanup table is additionally constrained (bpf_jit_supports_cleanup_pads() plus, from commit fa6890df61539 onward, the requirement that a resume sit at the end of a landing pad), whereas a program with cleanup_info_cnt == 0 must keep the ordinary fallthrough successor of the call.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#12 lore raw

Re: [PATCH bpf-next v2 05/20] bpf: Make exception landing pads reachable in the CFG

Quoted 22 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 14 lines
kernel/bpf/liveness.c +2 -0
index 44ecdc5b4ec2d..9cfd05f970bc9 100644
@@ -8,6 +8,8 @@
  #include <linux/slab.h>
  #include <linux/sort.h>
 
+#include "exception.h"
+
  #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
 
  struct per_frame_masks {

kernel/bpf/liveness.c bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) · line 263

Quoted 6 lines
kernel/bpf/liveness.c +3 -0
@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
  	succ = env->succ;
  	succ->cnt = 0;
 
+	if (unlikely(bpf_is_unwind_resume_kfunc(insn)))
+		return succ;
+

Should this early return be gated on env->cleanup_info_cnt?

Quoted 4 lines

bpf_unwind_resume() is an unconditionally registered generic kfunc with no KF_ flag restricting it to programs that carry a cleanup table. Its body is empty, so at run time the call simply returns and execution continues at the next instruction.

This should be okay without env->cleanup_info_cnt guard. Eventually if bpf_unwind_resume() is not in cleanup region, a verifier error message will be in the log.

Quoted 42 lines

visit_insn() in cfg.c still gives the call a fallthrough edge, and at this commit nothing in do_check() rejects a bpf_unwind_resume() call that is not the tail of a cleanup landing pad. So any CAP_BPF program with no cleanup table can call bpf_unwind_resume() and keep executing after it.

For such a program, bpf_insn_successors() reports zero successors, so bpf_compute_live_registers() treats everything after the call as dead. The register-cleaning side fails closed (a poisoned slot is rejected), but the 32-bit zero-extension side fails open:

insn_aux[i].zext_dst for an insn preceding the call is only set when the hi half of the defined register is live-after, so it stays false, and bpf_opt_subreg_zext_lo32_rnd_hi32() (kernel/bpf/fixups.c:706) then does 'if (!aux[adj_idx].zext_dst) { if (!rnd_hi32) continue; ... }' and emits no BPF_ZEXT_REG.

On architectures where bpf_jit_needs_zext() is true and kfunc calls are supported (s390, riscv64, powerpc64, parisc, x86-32), the upper 32 bits of a register the verifier proved to be 32-bit bounded are left with stale JIT-defined contents, which defeats the range checks the verifier derived for it.

Compare with the pad lookup ten lines below:

kernel/bpf/liveness.c bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) · line 277

Quoted 13 lines
kernel/bpf/liveness.c +6 -0
  	opcode_info = &opcode_info_tbl[BPF_CLASS(insn->code) | BPF_OP(insn->code)];
  	insn_sz = bpf_is_ldimm64(insn) ? 2 : 1;
  	if (opcode_info->can_fallthrough)
@@ -264,6 +269,13 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
  	if (opcode_info->can_jump)
  		succ->items[succ->cnt++] = idx + bpf_jmp_offset(insn) + 1;
 
+	if (unlikely(env->cleanup_info_cnt)) {
+		int pad = bpf_cleanup_pad_of_call(env, idx);
+
+		if (pad >= 0)
+			succ->items[succ->cnt++] = pad;
+	}

which gates the pad lookup on cleanup_info_cnt, and with visit_cleanup_pad_edge() in kernel/bpf/cfg.c which returns DONE_EXPLORING when !env->cleanup_info_cnt.

Quoted 6 lines

Would gating the early return on env->cleanup_info_cnt fix this? That would be sound because a program that carries a cleanup table is additionally constrained (bpf_jit_supports_cleanup_pads() plus, from commit fa6890df61539 onward, the requirement that a resume sit at the end of a landing pad), whereas a program with cleanup_info_cnt == 0 must keep the ordinary fallthrough successor of the call.

As I mentioned, previous gating with "env->cleanup_info_cnt" is not necessary.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#13 lore raw

PATCH v2 6/20 bpf: Explore the landing pads no call site reaches

In reply to Yonghong Song

A cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving bpf_check_cfg() to refuse the program over code its own frontend had no way not to emit:

0: call bpf_preempt_disable
1: call bpf_preempt_enable record = { begin = 1, end = 2, pad = 4 }
2: r0 = 0
3: exit
4: r1 = pads_ran ll landing pad
6: r2 = *(u64 *)(r1 + 0)
7: r2 |= RAN_NOUNWIND_REC
8: *(u64 *)(r1 + 0) = r2
9: call bpf_unwind_resume
10: exit

The range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it. cleanup_mark_call_sites() marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.

Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other: do_check() never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.

---
 kernel/bpf/cfg.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
kernel/bpf/cfg.c +17 -0
index 9f8b8b54d5ea..5a2b48b6a9e1 100644
@@ -633,6 +633,7 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
 	int insn_cnt = env->prog->len;
 	int *insn_stack, *insn_state;
 	int ex_insn_beg, i, ret = 0;
+	u32 pad_idx = 0;
 
 	insn_state = env->cfg.insn_state = kvzalloc_objs(int, insn_cnt,
 							 GFP_KERNEL_ACCOUNT);
@@ -688,6 +689,22 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
 		goto walk_cfg;
 	}
 
+	/*
+	 * A landing pad no call site was marked with -- a record whose range
+	 * holds no call an exception can unwind out of -- is reached by
+	 * nothing. Walk it from here, and let the dead code sweep remove it.
+	 */
+	while (pad_idx < env->cleanup_info_cnt) {
+		u32 pad = env->cleanup_info[pad_idx++].landing_pad_off;
+
+		if (insn_state[pad] != EXPLORED) {
+			insn_state[pad] = DISCOVERED;
+			insn_stack[0] = pad;
+			env->cfg.cur_stack = 1;
+			goto walk_cfg;
+		}
+	}
+
 	for (i = 0; i < insn_cnt; i++) {
 		struct bpf_insn *insn = &env->prog->insnsi[i];
 
Signature · 1 line
2.53.0-Meta
#14 lore raw

Re: [PATCH bpf-next v2 06/20] bpf: Explore the landing pads no call site reaches

This looks like a bug fix for the earlier commit that made exception landing pads reachable in the CFG. Should this include:
Quoted 28 lines

commit c93117f24a1a9cc39bfeaef79c42dbff02d8c455
Author: Yonghong Song <[email protected]>

bpf: Explore the landing pads no call site reaches

A cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving bpf_check_cfg() to refuse the program over code its own frontend had no way not to emit:

0: call bpf_preempt_disable
1: call bpf_preempt_enable record = { begin = 1, end = 2, pad = 4 }
2: r0 = 0
3: exit
4: r1 = pads_ran ll landing pad
6: r2 = *(u64 *)(r1 + 0)
7: r2 |= RAN_NOUNWIND_REC
8: *(u64 *)(r1 + 0) = r2
9: call bpf_unwind_resume
10: exit

The range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it. cleanup_mark_call_sites() marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.

Quoted 7 lines

Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other: do_check() never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.

This looks like a bug fix for the earlier commit that made exception landing pads reachable in the CFG. Should this include:

Fixes a67011bbbca1ba95 ("bpf: Make exception landing pads reachable in the CFG")

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#15 lore raw

Re: [PATCH bpf-next v2 06/20] bpf: Explore the landing pads no call site reaches

Quoted 34 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 34 lines

commit c93117f24a1a9cc39bfeaef79c42dbff02d8c455
Author: Yonghong Song <[email protected]>

bpf: Explore the landing pads no call site reaches

A cleanup record need not cover a call an exception can unwind out of: a frontend is free to emit a region around a helper or an ordinary kfunc, both nounwind here. Nothing marks a call site then, and that record's landing pad is reached by nothing at all -- leaving bpf_check_cfg() to refuse the program over code its own frontend had no way not to emit:

0: call bpf_preempt_disable
1: call bpf_preempt_enable record = { begin = 1, end = 2, pad = 4 }
2: r0 = 0
3: exit
4: r1 = pads_ran ll landing pad
6: r2 = *(u64 *)(r1 + 0)
7: r2 |= RAN_NOUNWIND_REC
8: *(u64 *)(r1 + 0) = r2
9: call bpf_unwind_resume
10: exit

The range [1,2) holds one call, and it is a kfunc, so an exception cannot come out of it. cleanup_mark_call_sites() marks nothing, nothing pushes an edge to 4, and 4 through 10 are reachable from nothing: "unreachable insn 4". The pad is dead, which is correct -- no exception can ever arrive at it -- but the program is fine and has to load.

Walk every pad the table names that the edges did not reach, the way the walk is already re-seeded at an exception callback. From there the pad is code like any other: do_check() never enters it, because no call site dispatches to it, so the dead code sweep removes it along with everything else that was not reached.

Quoted 5 lines
Quoted 1 line

This looks like a bug fix for the earlier commit that made exception landing pads reachable in the CFG. Should this include:

Fixes a67011bbbca1ba95 ("bpf: Make exception landing pads reachable in the CFG")

No need for this Fixes tag.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#16 lore raw

PATCH v2 7/20 bpf: Refuse exception cleanup shapes bpf_throw() cannot dispatch

In reply to Yonghong Song

The table is on the instructions and the landing pads are in the control flow graph. What is left before bpf_throw() can be taught to dispatch them is to work out what that walk will need, and refuse the shapes it could not handle.

bpf_check_cleanup_exceptions() runs after bpf_check_cfg(). Everything it needs is control flow, so it reads what that walk has already worked out rather than working it out again:

subprog_info.might_throw which subprograms an exception may leave,
closed over the call graph by merge_callee_effects() as the walk pops each callee
cleanup_reachability() what each instruction can reach -- a resume, a
plain exit, a throw, an indirect jump
cleanup_mark_pad_bodies() which instructions only ever run with an
exception already in flight

What it refuses:

- a pad that reaches both a resume and a plain exit, or neither: nothing says whether it is a cleanup pad or a catch pad
- a catch pad, which ends in a plain exit: the walker calls a pad as a subroutine and cannot hand a frame back its own execution
- a throw in a pad, or a call from a pad to a subprogram that can throw:
a second unwind over frames the first is still discarding
- a bpf_unwind_resume() outside a pad body
- a subprogram that may unwind used as a helper callback: the helper's own kernel frame would end the walk before it found a boundary
- a tail call, an indirect jump, or an outgoing on-stack call argument in a pad body, all of which touch a stack the pad does not own

---
 include/linux/bpf_verifier.h |   1 +
 kernel/bpf/exception.c       | 397 +++++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h       |   2 +
 kernel/bpf/fixups.c          |   1 +
 kernel/bpf/verifier.c        |   8 +
 5 files changed, 409 insertions(+)
include/linux/bpf_verifier.h +1 -0
index 7463e86d15ee..fe8b26351a15 100644
@@ -682,6 +682,7 @@ struct bpf_insn_aux_data {
 	bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
 	bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
 	bool cleanup_throw_site; /* call to bpf_throw() */
+	bool in_cleanup_pad; /* only runs with an exception in flight */
 	/*
 	 * 1 + the instruction index of the exception cleanup landing pad this
 	 * call site unwinds to, or 0 for none.
kernel/bpf/exception.c +397 -0

421 lines not on this page · raw message

kernel/bpf/exception.h +2 -0
index f51383fd775c..7313dd2b65a1 100644
@@ -8,6 +8,8 @@
 struct bpf_verifier_env;
 
 int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env);
+int bpf_check_cleanup_exceptions(struct bpf_verifier_env *env);
+int bpf_cleanup_check_callback(struct bpf_verifier_env *env, int subprog);
 int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx);
 
 #endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +1 -0
index 82b00fac6bd6..f120ae66dbbc 100644
@@ -252,6 +252,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 		/* Expand insni[off]'s seen count to the patched range. */
 		data[i].seen = old_seen;
 		data[i].zext_dst = bpf_insn_def32(new_prog, insn + i) >= 0;
+		data[i].in_cleanup_pad = data[off + cnt - 1].in_cleanup_pad;
 		if (!memcmp(insn + i, original_insn, sizeof(struct bpf_insn))) {
 			data[i].non_stack_access =
 				data[off + cnt - 1].non_stack_access;
kernel/bpf/verifier.c +8 -0
index 6496c30a10f7..9cbdb8339701 100644
@@ -10485,6 +10485,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
 	 * callbacks
 	 */
 	env->subprog_info[subprog].is_cb = true;
+	err = bpf_cleanup_check_callback(env, subprog);
+	if (err)
+		return err;
+
 	if (bpf_pseudo_kfunc_call(insn) &&
 	    !is_callback_calling_kfunc(insn->imm)) {
 		verifier_bug(env, "kfunc %s#%d not marked as callback-calling",
@@ -21664,6 +21668,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	ret = bpf_check_cleanup_exceptions(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	ret = bpf_compute_postorder(env);
 	if (ret < 0)
 		goto skip_full_check;
Signature · 1 line
2.53.0-Meta
author reviewer
#17 lore raw

PATCH v2 8/20 bpf: Walk the exception unwind in the verifier

In reply to Yonghong Song

bpf_throw() is about to stop discarding the frames it unwinds and start running their landing pads instead. Have the verifier walk the same thing, step for step:

at a throw, with the frame chain in env->cur_state->frame[]
if a record covers the call control is at
run that record's landing pad in this frame on reaching its resume, pop the frame and carry on
else
pop the frame and carry on
at the boundary, deliver

Doing it this way is what keeps the resource rules unchanged. Whatever a pad releases is released in the verifier state too, so by the time the walk reaches the boundary the state says exactly what the program will really hold there -- and check_resource_leak(), which used to fire the moment a throw was seen, simply moves to the end of the walk. A frame that no record covers contributes nothing, so anything it held is still held when the walk ends, and that is what gets reported.

---
 include/linux/bpf_cleanup_abi.h |  16 +++++
 include/linux/bpf_verifier.h    |   1 +
 kernel/bpf/states.c             |   3 +
 kernel/bpf/verifier.c           | 102 +++++++++++++++++++++++++-------
 4 files changed, 99 insertions(+), 23 deletions(-)
 create mode 100644 include/linux/bpf_cleanup_abi.h
include/linux/bpf_cleanup_abi.h new file +16 -0
new file mode 100644
index 000000000000..b6c1d589abda
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef _LINUX_BPF_CLEANUP_ABI_H
+#define _LINUX_BPF_CLEANUP_ABI_H
+
+/*
+ * Value arch_bpf_run_cleanup_pad() leaves in r0 on the way into a landing pad.
+ * It has to be a constant the verifier knows: LLVM names r0 as both the
+ * exception pointer and the exception selector register, so every pad reads it
+ * before anything else and is free to store what it read. Kept on its own
+ * because the verifier and the arch dispatchers, which are assembly, have to
+ * agree on it.
+ */
+#define BPF_PAD_ENTRY_R0	1
+
+#endif /* _LINUX_BPF_CLEANUP_ABI_H */
include/linux/bpf_verifier.h +1 -0
index fe8b26351a15..09fb89fda40a 100644
@@ -509,6 +509,7 @@ struct bpf_verifier_state {
 
 	bool speculative;
 	bool in_sleepable;
+	bool unwinding;
 
 	/* first and last insn idx of this verifier state */
 	u32 first_insn_idx;
kernel/bpf/states.c +3 -0
index 66fb11b6c6a7..be0f529f7ecc 100644
@@ -996,6 +996,9 @@ static bool states_equal(struct bpf_verifier_env *env,
 	if (old->in_sleepable != cur->in_sleepable)
 		return false;
 
+	if (old->unwinding != cur->unwinding)
+		return false;
+
 	if (!refsafe(old, cur, &env->idmap_scratch))
 		return false;
 
kernel/bpf/verifier.c +79 -23

160 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
author reviewer
#18 lore raw

PATCH v2 9/20 bpf: Refuse a private stack for a program with an exception cleanup table

In reply to Yonghong Song

A landing pad is about to start running outside its own frame, on the bpf_throw() walker's stack, with that frame's registers put back from a spill area whose contents arch_bpf_run_cleanup_pad() knows how to read. The x86-64 JIT addresses a private-stack program's frame through a scratch register that it recomputes after each call rather than through rbp, and no spill area holds that register, so a pad there would address its frame through whatever the kernel left behind.

Refuse the combination in check_max_stack_depth(), where the choice is made. Everywhere rather than arch-conditionally: arm64 keeps its private stack pointer in x27, which is in the prologue spill and so survives, but a rule that holds on one arch and not the other is not worth the second code path when nothing is lost but an optimization.

---
 kernel/bpf/verifier.c | 8 ++++++++
 1 file changed, 8 insertions(+)
kernel/bpf/verifier.c +8 -0
index a3b34ded1392..43ecf79baa4a 100644
@@ -5591,6 +5591,14 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
 		}
 	}
 
+	/*
+	 * A pad rebuilds its frame from a spill area, and on x86-64 a private
+	 * stack's frame pointer is in no spill area. Refused on every arch
+	 * rather than just that one.
+	 */
+	if (env->cleanup_info_cnt)
+		priv_stack_mode = NO_PRIV_STACK;
+
 	if (priv_stack_mode == PRIV_STACK_UNKNOWN)
 		priv_stack_mode = bpf_enable_priv_stack(env->prog);
 
Signature · 1 line
2.53.0-Meta
#19 lore raw

Re: [PATCH bpf-next v2 09/20] bpf: Refuse a private stack for a program with an exception cleanup table

The description says the x86-64 JIT 'recomputes after each call'. Does the JIT actually recompute it?
Quoted 7 lines

A landing pad is about to start running outside its own frame, on the bpf_throw() walker's stack, with that frame's registers put back from a spill area whose contents arch_bpf_run_cleanup_pad() knows how to read. The x86-64 JIT addresses a private-stack program's frame through a scratch register that it recomputes after each call rather than through rbp, and no spill area holds that register, so a pad there would address its frame through whatever the kernel left behind.

The description says the x86-64 JIT 'recomputes after each call'. Does the JIT actually recompute it?

Looking at the x86 JIT, emit_priv_frame_ptr() (arch/x86/net/bpf_jit_comp.c) is emitted once in the prologue, and the value in X86_REG_R9 is carried across calls by push_r9()/pop_r9() around each call site; BPF_REG_FP accesses are rewritten to X86_REG_R9.

The load-bearing half of the argument is still correct - arch_bpf_run_cleanup_pad() (added later in the series, arch/x86/net/bpf_cleanup_pad.S) restores only r15/r14/r13/rbx/r12 and rbp from the spill area and never restores r9, so a pad in a private-stack program would address its frame through a stale register.

Only the described mechanism (recompute vs save/restore around calls) is wrong, which makes the justification harder to check against the JIT.

Quoted 4 lines

Everywhere rather than arch-conditionally: arm64 keeps its private stack pointer in x27, which is in the prologue spill and so survives, but a rule that holds on one arch and not the other is not worth the second code path when nothing is lost but an optimization.

Does 'nothing is lost but an optimization' understate the effect?

The private-stack mode also changes the verifier's stack-depth accounting, not just JIT codegen. In check_max_stack_depth_subprog() a PRIV_STACK_ADAPTIVE subprog is checked against MAX_BPF_STACK on its own ('if (subprog_depth > MAX_BPF_STACK)'), while a NO_PRIV_STACK subprog accumulates into 'depth' and the whole bpf2bpf chain must fit in MAX_BPF_STACK ('combined stack size of %d calls is %d. Too large').

So for the prog types bpf_enable_priv_stack() would have returned PRIV_STACK_ADAPTIVE for (kprobe/tracepoint/perf_event/raw_tp, and tracing/lsm/struct_ops with a recursion or an explicit request), a program carrying a cleanup table whose nested subprog depths sum above 512 bytes now fails to load rather than losing performance.

Worth stating in the changelog, since it is a load-time visible restriction on the new feature rather than an optimization.

Related: kernel/sched/ext/ext.c sets prog->aux->priv_stack_requested for sched_ext dispatch/sub_caps_updated, and verifier.c rejects such a request when the JIT cannot honour it; here the request is silently ignored instead.

This looks like a bug fix for private stack support. Should this include:

Fixes a76ab5731e32d ("bpf: Find eligible subprogs for private stack support")

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#20 lore raw

Re: [PATCH bpf-next v2 09/20] bpf: Refuse a private stack for a program with an exception cleanup table

Quoted 37 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 7 lines

A landing pad is about to start running outside its own frame, on the bpf_throw() walker's stack, with that frame's registers put back from a spill area whose contents arch_bpf_run_cleanup_pad() knows how to read. The x86-64 JIT addresses a private-stack program's frame through a scratch register that it recomputes after each call rather than through rbp, and no spill area holds that register, so a pad there would address its frame through whatever the kernel left behind.

The description says the x86-64 JIT 'recomputes after each call'. Does the JIT actually recompute it?

Looking at the x86 JIT, emit_priv_frame_ptr() (arch/x86/net/bpf_jit_comp.c) is emitted once in the prologue, and the value in X86_REG_R9 is carried across calls by push_r9()/pop_r9() around each call site; BPF_REG_FP accesses are rewritten to X86_REG_R9.

The load-bearing half of the argument is still correct - arch_bpf_run_cleanup_pad() (added later in the series, arch/x86/net/bpf_cleanup_pad.S) restores only r15/r14/r13/rbx/r12 and rbp from the spill area and never restores r9, so a pad in a private-stack program would address its frame through a stale register.

Only the described mechanism (recompute vs save/restore around calls) is wrong, which makes the justification harder to check against the JIT.

Quoted 4 lines

Everywhere rather than arch-conditionally: arm64 keeps its private stack pointer in x27, which is in the prologue spill and so survives, but a rule that holds on one arch and not the other is not worth the second code path when nothing is lost but an optimization.

Does 'nothing is lost but an optimization' understate the effect?

The private-stack mode also changes the verifier's stack-depth accounting, not just JIT codegen. In check_max_stack_depth_subprog() a PRIV_STACK_ADAPTIVE subprog is checked against MAX_BPF_STACK on its own ('if (subprog_depth > MAX_BPF_STACK)'), while a NO_PRIV_STACK subprog accumulates into 'depth' and the whole bpf2bpf chain must fit in MAX_BPF_STACK ('combined stack size of %d calls is %d. Too large').

Quoted 8 lines

So for the prog types bpf_enable_priv_stack() would have returned PRIV_STACK_ADAPTIVE for (kprobe/tracepoint/perf_event/raw_tp, and tracing/lsm/struct_ops with a recursion or an explicit request), a program carrying a cleanup table whose nested subprog depths sum above 512 bytes now fails to load rather than losing performance.

Worth stating in the changelog, since it is a load-time visible restriction on the new feature rather than an optimization.

Okay, comments and commit message may have some stale things or something like (when nothing is lost but an optimization) is not correct. Will fix.

Quoted 8 lines

Related: kernel/sched/ext/ext.c sets prog->aux->priv_stack_requested for sched_ext dispatch/sub_caps_updated, and verifier.c rejects such a request when the JIT cannot honour it; here the request is silently ignored instead.

This looks like a bug fix for private stack support. Should this include:

Fixes a76ab5731e32d ("bpf: Find eligible subprogs for private stack support")

Fixes tag is unnecessary.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#21 lore raw

PATCH v2 10/20 bpf: Dispatch exception cleanup pads from bpf_throw()

In reply to Yonghong Song

Stop discarding the frames an exception unwinds through. bpf_throw() already walks the BPF call stack with arch_bpf_stack_walk() to find the exception boundary; have it look each frame's return address up in that (sub)program's cleanup table on the way and run the landing pad a matching record names.

A pad is run as a subroutine of the walker, not jumped to. It executes with the unwinding frame's frame pointer and BPF callee-saved registers, so everything it reads is that frame's, but on the current stack far below it, so nothing it calls can disturb the frame it is cleaning up after. It ends in what the JIT emits for its bpf_unwind_resume(), which hands control back to the walker rather than to the frame's caller. That is exactly the procedure the LLVM commit emitting the section describes.

Restoring the frame's r6-r9 is what makes this work, and it is only possible because the callee about to be discarded spilled them in its own prologue. bpf_cleanup_force_spill() tells a JIT to make that spill unconditional and of a known shape for every subprogram of a program carrying a cleanup table, and aux->exc->spill_off records where it starts, so the walker needs no per-frame metadata. The frame that called bpf_throw() has no callee to have spilled anything and never runs its own epilogue, so the JIT spills that frame's registers at the throw site instead, in the area aux->exc->throw_spill_off names.

The main program needs the table handed to it rather than built for it. jit_subprogs() compiles it as func[0], but the ksym covering that image is the one bpf_prog_load() registers for the outer bpf_prog, and that is what the walker finds -- so without the handover a landing pad in the main program's own frame is never dispatched, silently, since the exception still reaches the boundary and the cookie still comes back.

All of it hangs off bpf_prog_aux by a single pointer. bpf_prog_aux is on every BPF program and this is a niche feature, so struct bpf_exception_info exists only for a program that carries a table, and aux->exc being set is what says the program carries one at all.

---
 include/linux/bpf.h    |  75 +++++++++++++++++++
 include/linux/filter.h |   1 +
 kernel/bpf/core.c      |  30 +++++++-
 kernel/bpf/exception.c | 166 +++++++++++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h |   7 ++
 kernel/bpf/fixups.c    | 125 +++++++++++++++++++++++++++++++
 kernel/bpf/helpers.c   |  34 +++++++++
 7 files changed, 435 insertions(+), 3 deletions(-)
include/linux/bpf.h +75 -0
index 2a5fa346aada..83f2b0d7e596 100644
@@ -1770,6 +1770,80 @@ enum bpf_sig_keyring {
 	BPF_SIG_KEYRING_BPF,
 };
 
+/*
+ * One cleanup region of a JITed (sub)program: @pad is the landing pad to run
+ * for a return address in (begin, end], the native code of its call sites.
+ */
+struct bpf_cleanup_range {
+	u64 begin;
+	u64 end;
+	u64 pad;
+};
+
+struct bpf_exception_info {
+	struct bpf_cleanup_info *info;
+	struct bpf_cleanup_range *ranges;
+	/* Landing pad instruction indices, sorted and deduplicated. */
+	u32 *pad_at;
+	/* bpf_throw() call instruction indices, sorted. */
+	u32 *throw_at;
+	/* One bit per instruction that only runs while unwinding. */
+	unsigned long *pad_body;
+	u32 nr_info;
+	u32 nr_ranges;
+	u32 nr_pad_at;
+	u32 nr_throw_at;
+	u32 nr_pad_body;
+	/* Offset from a frame's FP to the caller's spilled r6-r9. */
+	s32 spill_off;
+	/* Likewise, to the registers a frame spills before calling bpf_throw(). */
+	s32 throw_spill_off;
+};
+
+#ifdef CONFIG_BPF_SYSCALL
+bool bpf_cleanup_force_spill(const struct bpf_prog *prog);
+bool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx);
+bool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx);
+bool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx);
+int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env, struct bpf_prog *prog);
+void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image);
+void bpf_cleanup_free_info(struct bpf_prog_aux *aux);
+#else
+static inline bool bpf_cleanup_force_spill(const struct bpf_prog *prog)
+{
+	return false;
+}
+
+static inline bool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx)
+{
+	return false;
+}
+
+static inline bool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx)
+{
+	return false;
+}
+
+static inline bool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx)
+{
+	return false;
+}
+
+static inline int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env,
+					       struct bpf_prog *prog)
+{
+	return 0;
+}
+
+static inline void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image)
+{
+}
+
+static inline void bpf_cleanup_free_info(struct bpf_prog_aux *aux)
+{
+}
+#endif
+
 struct bpf_prog_aux {
 	atomic64_t refcnt;
 	u32 used_map_cnt;
@@ -1850,6 +1924,7 @@ struct bpf_prog_aux {
 	char name[BPF_OBJ_NAME_LEN];
 	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
 	u16 stack_arg_sp_adjust;
+	struct bpf_exception_info *exc;
 #ifdef CONFIG_SECURITY
 	void *security;
 #endif
include/linux/filter.h +1 -0
index 9682b98ad890..287cd9b59aa9 100644
@@ -1243,6 +1243,7 @@ bool bpf_jit_supports_arena_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
 bool bpf_jit_supports_cleanup_pads(void);
+void arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base);
 bool bpf_jit_supports_ptr_xchg(void);
 bool bpf_jit_supports_arena(void);
 bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
kernel/bpf/core.c +27 -3
index f0dd851264f4..bd2919063cec 100644
@@ -292,6 +292,7 @@ void __bpf_prog_free(struct bpf_prog *fp)
 		mutex_destroy(&fp->aux->dst_mutex);
 		mutex_destroy(&fp->aux->st_ops_assoc_mutex);
 		kfree(fp->aux->poke_tab);
+		bpf_cleanup_free_info(fp->aux);
 		kfree(fp->aux);
 	}
 	free_percpu(fp->stats);
@@ -2625,13 +2626,21 @@ static bool bpf_prog_select_interpreter(struct bpf_prog *fp)
 	return select_interpreter;
 }
 
-static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
+static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog,
+					     int *err)
 {
 #ifdef CONFIG_BPF_JIT
 	struct bpf_prog *orig_prog;
+	int ret;
 
-	if (!bpf_prog_need_blind(prog))
+	if (!bpf_prog_need_blind(prog)) {
+		ret = bpf_cleanup_attach_main_prog(env, prog);
+		if (ret) {
+			*err = ret;
+			return prog;
+		}
 		return bpf_int_jit_compile(env, prog);
+	}
 
 	orig_prog = prog;
 	prog = bpf_jit_blind_constants(env, prog);
@@ -2642,6 +2651,13 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
 	if (IS_ERR(prog))
 		goto out_restore;
 
+	ret = bpf_cleanup_attach_main_prog(env, prog);
+	if (ret) {
+		*err = ret;
+		bpf_jit_prog_release_other(orig_prog, prog);
+		goto out_restore;
+	}
+
 	prog = bpf_int_jit_compile(env, prog);
 	if (prog->jited) {
 		bpf_jit_prog_release_other(prog, orig_prog);
@@ -2681,8 +2697,10 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
 		if (*err)
 			return fp;
 
-		fp = bpf_prog_jit_compile(env, fp);
+		fp = bpf_prog_jit_compile(env, fp, err);
 		bpf_prog_jit_attempt_done(fp);
+		if (*err)
+			return fp;
 		if (!fp->jited && jit_needed) {
 			*err = -ENOTSUPP;
 			return fp;
@@ -3475,6 +3493,12 @@ bool __weak bpf_jit_supports_cleanup_pads(void)
 	return false;
 }
 
+/* Call @pad with the frame pointer @frame_fp and r6-r9 spilled at @spill_base. */
+void __weak arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base)
+{
+	WARN_ON_ONCE(1);
+}
+
 bool __weak bpf_jit_supports_timed_may_goto(void)
 {
 	return false;
kernel/bpf/exception.c +166 -0

174 lines not on this page · raw message

kernel/bpf/exception.h +7 -0
index 7313dd2b65a1..c0e68ce227c8 100644
@@ -5,11 +5,18 @@
 
 #include <linux/types.h>
 
+struct bpf_cleanup_info;
+struct bpf_cleanup_range;
+struct bpf_prog;
+struct bpf_prog_aux;
 struct bpf_verifier_env;
 
 int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env);
 int bpf_check_cleanup_exceptions(struct bpf_verifier_env *env);
 int bpf_cleanup_check_callback(struct bpf_verifier_env *env, int subprog);
 int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx);
+int bpf_cleanup_alloc_info(struct bpf_prog_aux *aux);
+int bpf_cleanup_attach_info(struct bpf_prog_aux *aux, struct bpf_cleanup_info *recs, u32 cnt);
+const struct bpf_cleanup_range *bpf_cleanup_pad_for_ip(const struct bpf_prog *prog, u64 ip);
 
 #endif /* _LINUX_BPF_EXCEPTION_H */
kernel/bpf/fixups.c +125 -0

166 lines not on this page · raw message

kernel/bpf/helpers.c +34 -0
index 31d84de4d318..ffef72804fc9 100644
@@ -31,6 +31,7 @@
 #include <linux/buildid.h>
 
 #include "../../lib/kstrtox.h"
+#include "exception.h"
 
 /* If kernel subsystem is allowing eBPF programs to call this function,
  * inside its own verifier_ops->get_func_proto() callback it should return
@@ -3360,8 +3361,36 @@ struct bpf_throw_ctx {
 	u64 sp;
 	u64 bp;
 	int cnt;
+	const struct bpf_prog *callee;
+	u64 callee_fp;
 };
 
+static void bpf_run_cleanup_pad(struct bpf_throw_ctx *ctx, const struct bpf_prog *prog,
+				u64 ip, u64 fp)
+{
+	const struct bpf_exception_info *exc = prog->aux->exc;
+	const struct bpf_cleanup_range *rec;
+	u64 spill_base;
+
+	if (!exc || !exc->nr_ranges)
+		return;
+	rec = bpf_cleanup_pad_for_ip(prog, ip);
+	if (!rec)
+		return;
+
+	/*
+	 * The callee is always another subprogram of this program -- the walk
+	 * ends at any frame that is not one -- so its prologue spilled these
+	 * registers and its exc is there to say where.
+	 */
+	if (ctx->callee)
+		spill_base = ctx->callee_fp + ctx->callee->aux->exc->spill_off;
+	else
+		spill_base = fp + exc->throw_spill_off;
+
+	arch_bpf_run_cleanup_pad(rec->pad, fp, spill_base);
+}
+
 static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
 {
 	struct bpf_throw_ctx *ctx = cookie;
@@ -3378,6 +3407,11 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
 	if (!prog)
 		return !ctx->cnt;
 	ctx->cnt++;
+
+	bpf_run_cleanup_pad(ctx, prog, ip, bp);
+	ctx->callee = prog;
+	ctx->callee_fp = bp;
+
 	if (bpf_is_subprog(prog))
 		return true;
 	ctx->aux = prog->aux;
Signature · 1 line
2.53.0-Meta
#22 lore raw

Re: [PATCH bpf-next v2 10/20] bpf: Dispatch exception cleanup pads from bpf_throw()

This isn't a bug, but could insn_idx_in() just use bsearch() from <linux/bsearch.h>, reusing the cmp_u32() already defined above for the sort()? kernel/bpf/fixups.c in this series uses bsearch() for…
Quoted 4 lines
kernel/bpf/exception.c +0 -0
index b2bf831242b85..521086d084a35 100644

[ ... ]

Quoted 6 lines
kernel/bpf/exception.c +6 -0
+static int cmp_u32(const void *a, const void *b)
+{
+	u32 x = *(const u32 *)a, y = *(const u32 *)b;
+
+	return x < y ? -1 : x > y;
+}

[ ... ]

Quoted 7 lines
kernel/bpf/exception.c +7 -0
+/* Is @idx in the sorted array @at of @n instruction indices? */
+static bool insn_idx_in(const u32 *at, u32 n, u32 idx)
+{
+	u32 l = 0, r = n;
+
+	while (l < r) {
+		u32 m = l + (r - l) / 2;

kernel/bpf/exception.c int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx) · line 632

Quoted 10 lines
kernel/bpf/exception.c +10 -0
+
+		if (idx < at[m])
+			r = m;
+		else if (idx > at[m])
+			l = m + 1;
+		else
+			return true;
+	}
+	return false;
+}

This isn't a bug, but could insn_idx_in() just use bsearch() from <linux/bsearch.h>, reusing the cmp_u32() already defined above for the sort()? kernel/bpf/fixups.c in this series uses bsearch() for similar lookups.

Quoted 4 lines
kernel/bpf/fixups.c +0 -0
index f120ae66dbbcb..134aafa6a6c9b 100644

[ ... ]

Quoted 55 lines
kernel/bpf/fixups.c +51 -0
@@ -1113,6 +1120,116 @@ static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_s
 	env->subprog_info[env->subprog_cnt].start = env->prog->len;
 }
 
+static int cleanup_info_for_subprog(struct bpf_verifier_env *env, struct bpf_prog *sub,
+				    u32 start, u32 end)
+{
+	struct bpf_cleanup_info *recs;
+	u32 i, cnt = 0;
+	int err;
+
+	if (!env->cleanup_info_cnt)
+		return 0;
+
+	err = bpf_cleanup_alloc_info(sub->aux);
+	if (err)
+		return err;
+
+	err = cleanup_throw_sites_for_subprog(env, sub, start, end);
+	if (err)
+		return err;
+
+	err = cleanup_pad_body_for_subprog(env, sub, start, end);
+	if (err)
+		return err;
+
+	for (i = start; i < end; i++)
+		if (env->insn_aux_data[i].cleanup_pad)
+			cnt++;
+	if (!cnt)
+		return 0;
+
+	recs = kvmalloc_array(cnt, sizeof(*recs), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+	if (!recs)
+		return -ENOMEM;
+
+	for (i = start, cnt = 0; i < end; i++) {
+		u32 pad = env->insn_aux_data[i].cleanup_pad;
+
+		if (!pad)
+			continue;
+		pad--;
+		if (verifier_bug_if(pad < start || pad >= end, env,
+				    "insn %u is covered by a landing pad at %u outside its subprog [%u, %u)",
+				    i, pad, start, end)) {
+			kvfree(recs);
+			return -EFAULT;
+		}
+		recs[cnt].begin_off = i - start;
+		recs[cnt].end_off = i - start + 1;
+		recs[cnt].landing_pad_off = pad - start;
+		cnt++;
+	}
+	return bpf_cleanup_attach_info(sub->aux, recs, cnt);
+}

[ ... ]

Quoted 3 lines

static int jit_subprogs(struct bpf_verifier_env *env) {
struct bpf_prog *prog = env->prog, **func, *tmp;

[ ... ]

kernel/bpf/fixups.c static int jit_subprogs(struct bpf_verifier_env *env) · line 1371

Quoted 6 lines
kernel/bpf/fixups.c +2 -0
@@ -1250,6 +1367,10 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 		func[i]->aux->token = prog->aux->token;
 		if (!i)
 			func[i]->aux->exception_boundary = env->seen_exception;
+		err = cleanup_info_for_subprog(env, func[i], subprog_start,
+					       env->subprog_info[i + 1].start);

This isn't a bug, but would passing subprog_end (already computed a few lines above from env->subprog_info[i + 1].start) be clearer here? It keeps the cleanup table's end bound explicitly tied to the same value jit_subprogs() uses for func[i]->len.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#23 lore raw

Re: [PATCH bpf-next v2 10/20] bpf: Dispatch exception cleanup pads from bpf_throw()

Quoted 27 lines · On 9/17/26 10:58 PM, [email protected] wrote:
Quoted 4 lines
kernel/bpf/exception.c +0 -0
index b2bf831242b85..521086d084a35 100644

[ ... ]

Quoted 6 lines
kernel/bpf/exception.c +6 -0
+static int cmp_u32(const void *a, const void *b)
+{
+	u32 x = *(const u32 *)a, y = *(const u32 *)b;
+
+	return x < y ? -1 : x > y;
+}

[ ... ]

Quoted 13 lines
kernel/bpf/exception.c +13 -0
+/* Is @idx in the sorted array @at of @n instruction indices? */
+static bool insn_idx_in(const u32 *at, u32 n, u32 idx)
+{
+	u32 l = 0, r = n;
+
+	while (l < r) {
+		u32 m = l + (r - l) / 2;
+
+		if (idx < at[m])
+			r = m;
+		else if (idx > at[m])
+			l = m + 1;
+		else
Quoted 8 lines

kernel/bpf/exception.c int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx) · line 632

Quoted 4 lines
kernel/bpf/exception.c +4 -0
+			return true;
+	}
+	return false;
+}

This isn't a bug, but could insn_idx_in() just use bsearch() from <linux/bsearch.h>, reusing the cmp_u32() already defined above for the sort()? kernel/bpf/fixups.c in this series uses bsearch() for similar lookups.

Good point. will do.

Quoted 69 lines
Quoted 5 lines
kernel/bpf/fixups.c +0 -0
index f120ae66dbbcb..134aafa6a6c9b 100644

[ ... ]

Quoted 55 lines
kernel/bpf/fixups.c +51 -0
@@ -1113,6 +1120,116 @@ static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_s
  	env->subprog_info[env->subprog_cnt].start = env->prog->len;
  }
 
+static int cleanup_info_for_subprog(struct bpf_verifier_env *env, struct bpf_prog *sub,
+				    u32 start, u32 end)
+{
+	struct bpf_cleanup_info *recs;
+	u32 i, cnt = 0;
+	int err;
+
+	if (!env->cleanup_info_cnt)
+		return 0;
+
+	err = bpf_cleanup_alloc_info(sub->aux);
+	if (err)
+		return err;
+
+	err = cleanup_throw_sites_for_subprog(env, sub, start, end);
+	if (err)
+		return err;
+
+	err = cleanup_pad_body_for_subprog(env, sub, start, end);
+	if (err)
+		return err;
+
+	for (i = start; i < end; i++)
+		if (env->insn_aux_data[i].cleanup_pad)
+			cnt++;
+	if (!cnt)
+		return 0;
+
+	recs = kvmalloc_array(cnt, sizeof(*recs), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+	if (!recs)
+		return -ENOMEM;
+
+	for (i = start, cnt = 0; i < end; i++) {
+		u32 pad = env->insn_aux_data[i].cleanup_pad;
+
+		if (!pad)
+			continue;
+		pad--;
+		if (verifier_bug_if(pad < start || pad >= end, env,
+				    "insn %u is covered by a landing pad at %u outside its subprog [%u, %u)",
+				    i, pad, start, end)) {
+			kvfree(recs);
+			return -EFAULT;
+		}
+		recs[cnt].begin_off = i - start;
+		recs[cnt].end_off = i - start + 1;
+		recs[cnt].landing_pad_off = pad - start;
+		cnt++;
+	}
+	return bpf_cleanup_attach_info(sub->aux, recs, cnt);
+}

[ ... ]

Quoted 3 lines
  static int jit_subprogs(struct bpf_verifier_env *env)
  {
  	struct bpf_prog *prog = env->prog, **func, *tmp;

[ ... ]

Quoted 10 lines

kernel/bpf/fixups.c static int jit_subprogs(struct bpf_verifier_env *env) · line 1371

Quoted 6 lines
kernel/bpf/fixups.c +2 -0
@@ -1250,6 +1367,10 @@ static int jit_subprogs(struct bpf_verifier_env *env)
  		func[i]->aux->token = prog->aux->token;
  		if (!i)
  			func[i]->aux->exception_boundary = env->seen_exception;
+		err = cleanup_info_for_subprog(env, func[i], subprog_start,
+					       env->subprog_info[i + 1].start);

This isn't a bug, but would passing subprog_end (already computed a few lines above from env->subprog_info[i + 1].start) be clearer here? It keeps the cleanup table's end bound explicitly tied to the same value jit_subprogs() uses for func[i]->len.

Good point. subprog_end is a good choice.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#24 lore raw

PATCH v2 11/20 bpf, x86: Dispatch exception cleanup pads at run time

In reply to Yonghong Song

Provide the arch half for x86-64: force the full callee-saved spill for a program carrying a cleanup table so the walker can find a frame's r6-r9 in its callee's prologue, record where that spill area starts, build the native cleanup table from the JIT's addrs[], emit a bare return for a pad's bpf_unwind_resume(), and hand control to a pad from arch_bpf_run_cleanup_pad().

Support is gated on CONFIG_UNWINDER_ORC, the same requirement arch_bpf_stack_walk() and therefore bpf_throw() already have here.

---
 arch/x86/net/Makefile          |  2 +-
 arch/x86/net/bpf_cleanup_pad.S | 74 ++++++++++++++++++++++++++++++++
 arch/x86/net/bpf_jit_comp.c    | 78 ++++++++++++++++++++++++++++------
 3 files changed, 141 insertions(+), 13 deletions(-)
 create mode 100644 arch/x86/net/bpf_cleanup_pad.S
arch/x86/net/Makefile +1 -1
index dddbefc0f439..9d574d972df3 100644
@@ -6,5 +6,5 @@
 ifeq ($(CONFIG_X86_32),y)
         obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o
 else
-        obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+        obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o
 endif
arch/x86/net/bpf_cleanup_pad.S new file +74 -0

74 lines not on this page · raw message

arch/x86/net/bpf_jit_comp.c +66 -12

170 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#25 lore raw

Re: [PATCH bpf-next v2 11/20] bpf, x86: Dispatch exception cleanup pads at run time

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] bpf, x86: garbage kfunc stack arguments in exception cleanup pads --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf, x86: garbage kfunc stack arguments in exception cleanup pads --

commit 1e32297a3c2c4f6bbc4d458bc25563bed8167f6c
Author: Yonghong Song <[email protected]>

bpf, x86: Dispatch exception cleanup pads at run time

This commit provides the architecture-specific support for x86-64 exception cleanup pads. It forces a full callee-saved spill for programs with a cleanup table and introduces arch_bpf_run_cleanup_pad to execute the pads.

Quoted 5 lines
arch/x86/net/bpf_cleanup_pad.S new file +0 -0
new file mode 100644
index 0000000000000..da4b448ecf095

[ ... ]

Quoted 17 lines
arch/x86/net/bpf_cleanup_pad.S new file +17 -0
+SYM_FUNC_START(arch_bpf_run_cleanup_pad)
+	ANNOTATE_NOENDBR
+
+	pushq %rbp
+	pushq %rbx
+	pushq %r12
+	pushq %r13
+	pushq %r14
+	pushq %r15
+	/* Keep the pad's entry rsp congruent to a normal call's. */
+	subq $8, %rsp
+
+	movq 0(%rdx), %r15
+	movq 8(%rdx), %r14
+	movq 16(%rdx), %r13
+	movq 24(%rdx), %rbx
+	movq 32(%rdx), %r12

arch/x86/net/bpf_cleanup_pad.S line 64

Quoted 10 lines
arch/x86/net/bpf_cleanup_pad.S new file +10 -0
+	/* rbp is BPF r10, so this is the whole of the pad's frame setup. */
+	movq %rsi, %rbp
+
+	/* CALL_NOSPEC needs the target in a register; rcx is BPF r4, dead. */
+	movq %rdi, %rcx
+
+	/* BPF r0 on the way into a pad, not whatever the kernel left in rax. */
+	movl $BPF_PAD_ENTRY_R0, %eax
+
+	CALL_NOSPEC rcx

[Severity: High]
Does this execution environment cause kfuncs with more than 5 arguments to read uninitialized garbage stack arguments?

In arch_bpf_run_cleanup_pad, the pad is executed on the unwinder's stack (rsp), while rbp is set to the original BPF frame.

However, the JIT unconditionally writes outgoing kfunc stack arguments relative to BPF_REG_FP (rbp), as seen in emit_kfunc_arg_moves in arch/x86/net/bpf_jit_comp.c:

else if (dst_mem)
emit_stx(&prog, BPF_DW, BPF_REG_FP, reg,
stack_base + (dst - nreg) * 8);

Because rbp and rsp are desynchronized in the pad, the arguments are written to the old frame. When the kfunc is called, the C calling convention requires it to read stack arguments from [rsp], which points to the unwinder's stack instead of the original frame, leading to garbage reads.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
author reviewer
#26 lore raw

Re: [PATCH bpf-next v2 11/20] bpf, x86: Dispatch exception cleanup pads at run time

Quoted 42 lines · On 9/17/26 10:03 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf, x86: garbage kfunc stack arguments in exception cleanup pads --

commit 1e32297a3c2c4f6bbc4d458bc25563bed8167f6c
Author: Yonghong Song <[email protected]>

bpf, x86: Dispatch exception cleanup pads at run time

This commit provides the architecture-specific support for x86-64 exception cleanup pads. It forces a full callee-saved spill for programs with a cleanup table and introduces arch_bpf_run_cleanup_pad to execute the pads.

Quoted 5 lines
arch/x86/net/bpf_cleanup_pad.S new file +0 -0
new file mode 100644
index 0000000000000..da4b448ecf095

[ ... ]

Quoted 22 lines
arch/x86/net/bpf_cleanup_pad.S new file +22 -0
+SYM_FUNC_START(arch_bpf_run_cleanup_pad)
+	ANNOTATE_NOENDBR
+
+	pushq %rbp
+	pushq %rbx
+	pushq %r12
+	pushq %r13
+	pushq %r14
+	pushq %r15
+	/* Keep the pad's entry rsp congruent to a normal call's. */
+	subq $8, %rsp
+
+	movq 0(%rdx), %r15
+	movq 8(%rdx), %r14
+	movq 16(%rdx), %r13
+	movq 24(%rdx), %rbx
+	movq 32(%rdx), %r12
+	/* rbp is BPF r10, so this is the whole of the pad's frame setup. */
+	movq %rsi, %rbp
+
+	/* CALL_NOSPEC needs the target in a register; rcx is BPF r4, dead. */
+	movq %rdi, %rcx
Quoted 8 lines

arch/x86/net/bpf_cleanup_pad.S line 64

Quoted 5 lines
arch/x86/net/bpf_cleanup_pad.S new file +5 -0
+
+	/* BPF r0 on the way into a pad, not whatever the kernel left in rax. */
+	movl $BPF_PAD_ENTRY_R0, %eax
+
+	CALL_NOSPEC rcx

[Severity: High]
Does this execution environment cause kfuncs with more than 5 arguments to read uninitialized garbage stack arguments?

We should be okay since we already have:

+ if (bpf_pseudo_kfunc_call(insn)) {
+ struct bpf_call_summary cs;
+
+ if (bpf_get_call_summary(env, insn, &cs) &&
+ cs.arg_slot_cnt > MAX_BPF_FUNC_REG_ARGS) {
+ verbose(env,
+ "insn %u passes an on-stack call argument in an exception cleanup landing pad\n",
+ i);
+ return -EINVAL;
+ }
+ }

Quoted 18 lines

In arch_bpf_run_cleanup_pad, the pad is executed on the unwinder's stack (rsp), while rbp is set to the original BPF frame.

However, the JIT unconditionally writes outgoing kfunc stack arguments relative to BPF_REG_FP (rbp), as seen in emit_kfunc_arg_moves in arch/x86/net/bpf_jit_comp.c:

else if (dst_mem)
emit_stx(&prog, BPF_DW, BPF_REG_FP, reg,
stack_base + (dst - nreg) * 8);

Because rbp and rsp are desynchronized in the pad, the arguments are written to the old frame. When the kfunc is called, the C calling convention requires it to read stack arguments from [rsp], which points to the unwinder's stack instead of the original frame, leading to garbage reads.

#27 lore raw

Re: [PATCH bpf-next v2 11/20] bpf, x86: Dispatch exception cleanup pads at run time

In reply to Yonghong Song
Will this build with CONFIG_STACK_VALIDATION enabled?
Quoted 16 lines
arch/x86/net/Makefile +1 -1
index dddbefc0f4398..9d574d972df37 100644
@@ -6,5 +6,5 @@
 ifeq ($(CONFIG_X86_32),y)
         obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o
 else
-        obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+        obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o
 endif
arch/x86/net/bpf_cleanup_pad.S new file +0 -0
new file mode 100644
index 0000000000000..da4b448ecf095

[ ... ]

Quoted 27 lines
arch/x86/net/bpf_cleanup_pad.S new file +27 -0
+SYM_FUNC_START(arch_bpf_run_cleanup_pad)
+	ANNOTATE_NOENDBR
+
+	pushq %rbp
+	pushq %rbx
+	pushq %r12
+	pushq %r13
+	pushq %r14
+	pushq %r15
+	/* Keep the pad's entry rsp congruent to a normal call's. */
+	subq $8, %rsp
+
+	movq 0(%rdx), %r15
+	movq 8(%rdx), %r14
+	movq 16(%rdx), %r13
+	movq 24(%rdx), %rbx
+	movq 32(%rdx), %r12
+	/* rbp is BPF r10, so this is the whole of the pad's frame setup. */
+	movq %rsi, %rbp
+
+	/* CALL_NOSPEC needs the target in a register; rcx is BPF r4, dead. */
+	movq %rdi, %rcx
+
+	/* BPF r0 on the way into a pad, not whatever the kernel left in rax. */
+	movl $BPF_PAD_ENTRY_R0, %eax
+
+	CALL_NOSPEC rcx

arch/x86/net/bpf_cleanup_pad.S line 74

Quoted 10 lines
arch/x86/net/bpf_cleanup_pad.S new file +10 -0
+
+	addq $8, %rsp
+	popq %r15
+	popq %r14
+	popq %r13
+	popq %r12
+	popq %rbx
+	popq %rbp
+	RET
+SYM_FUNC_END(arch_bpf_run_cleanup_pad)

Will this build with CONFIG_STACK_VALIDATION enabled?

arch_bpf_run_cleanup_pad() is declared with SYM_FUNC_START (an STT_FUNC symbol) and saves rbp with 'pushq %rbp', but never establishes a frame pointer with 'mov %rsp, %rbp'. Instead, it overwrites rbp with the BPF frame pointer (movq %rsi, %rbp) and then makes a call (CALL_NOSPEC rcx).

objtool's frame-pointer validation (CONFIG_STACK_VALIDATION with UNWINDER_FRAME_POINTER) checks for two conditions in validate_call() and validate_return():

tools/objtool/check.c:has_valid_stack_frame() requires cfi->cfa.base == CFI_BP, which is only set when the function does 'mov %rsp, %rbp' to establish a frame. This function never does that, so the CALL_NOSPEC would trigger:

    WARN_INSN(insn, "call without frame pointer save/setup");

Additionally, the 'pushq %rbp' while cfa->base != CFI_BP sets cfi->bp_scratch = true, which is only cleared by 'mov %rsp, %rbp'. This flag is still set at the RET, where validate_return() warns:

    WARN_INSN(insn, "BP used as a scratch register");

The Makefile adds bpf_cleanup_pad.o unconditionally under obj-$(CONFIG_BPF_JIT), even though the feature is gated on CONFIG_UNWINDER_ORC per bpf_jit_supports_cleanup_pads(). This means in a CONFIG_STACK_VALIDATION=y build (frame-pointer unwinder), the object would fail objtool validation, and with CONFIG_OBJTOOL_WERROR=y the build would fail.

The neighbouring arch/x86/net/bpf_timed_may_goto.S does not have this issue because it sets up a proper frame:

arch/x86/net/bpf_timed_may_goto.S:
pushq %rbp
movq %rsp, %rbp
...
leave

rbp cannot be the frame pointer here because it must carry BPF r10 into the pad. This is the same situation arch/x86/kernel/ftrace_64.S handles with STACK_FRAME_NON_STANDARD_FP(ftrace_caller). Could either STACK_FRAME_NON_STANDARD_FP(arch_bpf_run_cleanup_pad) after SYM_FUNC_END or building the object only when CONFIG_UNWINDER_ORC is set (matching the feature gate) address this?

arch/x86/net/bpf_jit_comp.c struct jit_context { · line 363

Quoted 12 lines
arch/x86/net/bpf_jit_comp.c +4 -0
index d4a980140b48d..9d0dd54773e83 100644
@@ -357,6 +357,11 @@ struct jit_context {
 /* Number of bytes that will be skipped on tailcall */
 #define X86_TAIL_CALL_OFFSET	(12 + ENDBR_INSN_SIZE)
 
+/* Throw-site spill: r15, r14, r13, rbx, r12 low to high, the layout the
+ * prologue's pushes leave, so arch_bpf_run_cleanup_pad() reads both alike.
+ */
+#define X86_CLEANUP_SPILL_SZ	(5 * 8)

This isn't a bug, but could the comment format be updated to match the BPF subsystem style? The BPF subsystem guide requires the modern kernel multi-line comment style, where the opening /* is on its own line. The patch already uses this format for its other new multi-line comments (the "Lowest address of each spill area" block and the "Spill r6-r9 and r12" block later in do_jit), so is there a reason for the inconsistency here?

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#28 lore raw

Re: [PATCH bpf-next v2 11/20] bpf, x86: Dispatch exception cleanup pads at run time

Quoted 95 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 16 lines
arch/x86/net/Makefile +1 -1
index dddbefc0f4398..9d574d972df37 100644
@@ -6,5 +6,5 @@
  ifeq ($(CONFIG_X86_32),y)
          obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o
  else
-        obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+        obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o
  endif
arch/x86/net/bpf_cleanup_pad.S new file +0 -0
new file mode 100644
index 0000000000000..da4b448ecf095

[ ... ]

Quoted 27 lines
arch/x86/net/bpf_cleanup_pad.S new file +27 -0
+SYM_FUNC_START(arch_bpf_run_cleanup_pad)
+	ANNOTATE_NOENDBR
+
+	pushq %rbp
+	pushq %rbx
+	pushq %r12
+	pushq %r13
+	pushq %r14
+	pushq %r15
+	/* Keep the pad's entry rsp congruent to a normal call's. */
+	subq $8, %rsp
+
+	movq 0(%rdx), %r15
+	movq 8(%rdx), %r14
+	movq 16(%rdx), %r13
+	movq 24(%rdx), %rbx
+	movq 32(%rdx), %r12
+	/* rbp is BPF r10, so this is the whole of the pad's frame setup. */
+	movq %rsi, %rbp
+
+	/* CALL_NOSPEC needs the target in a register; rcx is BPF r4, dead. */
+	movq %rdi, %rcx
+
+	/* BPF r0 on the way into a pad, not whatever the kernel left in rax. */
+	movl $BPF_PAD_ENTRY_R0, %eax
+
+	CALL_NOSPEC rcx

arch/x86/net/bpf_cleanup_pad.S line 74

Quoted 10 lines
arch/x86/net/bpf_cleanup_pad.S new file +10 -0
+
+	addq $8, %rsp
+	popq %r15
+	popq %r14
+	popq %r13
+	popq %r12
+	popq %rbx
+	popq %rbp
+	RET
+SYM_FUNC_END(arch_bpf_run_cleanup_pad)

Will this build with CONFIG_STACK_VALIDATION enabled?

arch_bpf_run_cleanup_pad() is declared with SYM_FUNC_START (an STT_FUNC symbol) and saves rbp with 'pushq %rbp', but never establishes a frame pointer with 'mov %rsp, %rbp'. Instead, it overwrites rbp with the BPF frame pointer (movq %rsi, %rbp) and then makes a call (CALL_NOSPEC rcx).

objtool's frame-pointer validation (CONFIG_STACK_VALIDATION with UNWINDER_FRAME_POINTER) checks for two conditions in validate_call() and validate_return():

tools/objtool/check.c:has_valid_stack_frame() requires cfi->cfa.base == CFI_BP, which is only set when the function does 'mov %rsp, %rbp' to establish a frame. This function never does that, so the CALL_NOSPEC would trigger:

     WARN_INSN(insn, "call without frame pointer save/setup");

Additionally, the 'pushq %rbp' while cfa->base != CFI_BP sets cfi->bp_scratch = true, which is only cleared by 'mov %rsp, %rbp'. This flag is still set at the RET, where validate_return() warns:

     WARN_INSN(insn, "BP used as a scratch register");

The Makefile adds bpf_cleanup_pad.o unconditionally under obj-$(CONFIG_BPF_JIT), even though the feature is gated on CONFIG_UNWINDER_ORC per bpf_jit_supports_cleanup_pads(). This means in a CONFIG_STACK_VALIDATION=y build (frame-pointer unwinder), the object would fail objtool validation, and with CONFIG_OBJTOOL_WERROR=y the build would fail.

The neighbouring arch/x86/net/bpf_timed_may_goto.S does not have this issue because it sets up a proper frame:

arch/x86/net/bpf_timed_may_goto.S:
pushq %rbp
movq %rsp, %rbp
...
leave

Quoted 6 lines

rbp cannot be the frame pointer here because it must carry BPF r10 into the pad. This is the same situation arch/x86/kernel/ftrace_64.S handles with STACK_FRAME_NON_STANDARD_FP(ftrace_caller). Could either STACK_FRAME_NON_STANDARD_FP(arch_bpf_run_cleanup_pad) after SYM_FUNC_END or building the object only when CONFIG_UNWINDER_ORC is set (matching the feature gate) address this?

Yes, jit only supports CONFIG_UNWINDER_ORC for x86, and CONFIG_STACK_VALIDATION requires CONFIG_UNWINDER_FRAME_POINTER.

Quoted 9 lines
Quoted 9 lines
arch/x86/net/bpf_jit_comp.c +0 -0
index d4a980140b48d..9d0dd54773e83 100644
@@ -357,6 +357,11 @@ struct jit_context {
  /* Number of bytes that will be skipped on tailcall */
  #define X86_TAIL_CALL_OFFSET	(12 + ENDBR_INSN_SIZE)
 
Quoted 11 lines

arch/x86/net/bpf_jit_comp.c struct jit_context { · line 363

Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +4 -0
+/* Throw-site spill: r15, r14, r13, rbx, r12 low to high, the layout the
+ * prologue's pushes leave, so arch_bpf_run_cleanup_pad() reads both alike.
+ */
+#define X86_CLEANUP_SPILL_SZ	(5 * 8)

This isn't a bug, but could the comment format be updated to match the BPF subsystem style? The BPF subsystem guide requires the modern kernel multi-line comment style, where the opening /* is on its own line. The patch already uses this format for its other new multi-line comments (the "Lowest address of each spill area" block and the "Spill r6-r9 and r12" block later in do_jit), so is there a reason for the inconsistency here?

Will adjust comment.

Quoted 9 lines

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#29 lore raw

PATCH v2 12/20 bpf, arm64: Dispatch exception cleanup pads at run time

In reply to Yonghong Song

The same arch half for arm64: force the full callee-saved spill, record where it starts, build the native table from the JIT's byte offsets, and hand control to a pad from arch_bpf_run_cleanup_pad().

Unlike x86-64 a pad cannot simply return: every call it makes clobbers x30, so no return address survives to its resume. x23 carries it instead. bpf2a64[] maps nothing to x23 or x24 -- they are pushed only to keep the frame shape the exception callback expects -- so nothing else in generated code touches them, and being callee-saved they survive every kfunc the pad calls. The JIT emits "br x23" for a pad's bpf_unwind_resume().

x24 is the other one, and it is what makes a pad able to touch its own frame at all. Generated code addresses the BPF frame through the stack pointer -- the frame sits directly on top of it, which turns every offset positive and each access into one instruction -- and in a pad the stack pointer is the walker's. So the JIT has a pad recompute the equivalent of its frame's stack pointer from BPF r10 on entry, into x24, and addresses the frame off x24 for every instruction the previous patches marked as running only while unwinding. One instruction per pad, and the accesses keep the shape and the immediate range they have everywhere else.

---
 arch/arm64/net/Makefile          |  2 +-
 arch/arm64/net/bpf_cleanup_pad.S | 95 +++++++++++++++++++++++++++++++
 arch/arm64/net/bpf_jit_comp.c    | 97 ++++++++++++++++++++++++++++++--
 3 files changed, 187 insertions(+), 7 deletions(-)
 create mode 100644 arch/arm64/net/bpf_cleanup_pad.S
arch/arm64/net/Makefile +1 -1
index 3ae382bfca87..ebec2a44a52b 100644
@@ -2,4 +2,4 @@
 #
 # ARM64 networking code
 #
-obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o
arch/arm64/net/bpf_cleanup_pad.S new file +95 -0

95 lines not on this page · raw message

arch/arm64/net/bpf_jit_comp.c +91 -6

196 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#30 lore raw

Re: [PATCH bpf-next v2 12/20] bpf, arm64: Dispatch exception cleanup pads at run time

This isn't a bug, but would it read better to put throw_spill next to stack_size and stack_arg_size, which it is always used with, rather than ahead of prog at the head of the struct?
Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +0 -0
index 6c04fee468766..560eba305bca2 100644

[ ... ]

Quoted 12 lines
arch/arm64/net/bpf_jit_comp.c +8 -0
@@ -75,7 +76,21 @@ static const int bpf2a64[] = {
 	[ARENA_VM_START] = A64_R(28),
 };
 
+/* Throw-site spill: the five pairs push_callee_regs() forces on, same size and
+ * slot order, so arch_bpf_run_cleanup_pad() reads both alike.
+ */
+#define A64_CLEANUP_SPILL_SZ	(5 * 16)
+
+/*
+ * Where a landing pad's frame is anchored, since the stack pointer generated
+ * code normally addresses it through is the walker's inside a pad. bpf2a64[]

arch/arm64/net/bpf_jit_comp.c static const int bpf2a64[] = { · line 96

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +6 -0
+ * maps nothing to x24, so nothing else in generated code touches it.
+ */
+#define A64_CLEANUP_FP		A64_R(24)
+
 struct jit_ctx {
+	/* Bytes reserved for the throw-site spill; see bpf_cleanup_force_spill(). */
+	u32 throw_spill;
 	const struct bpf_prog *prog;
 	int idx;
 	int epilogue_offset;

This isn't a bug, but would it read better to put throw_spill next to stack_size and stack_arg_size, which it is always used with, rather than ahead of prog at the head of the struct?

[ ... ]

Quoted 11 lines
arch/arm64/net/bpf_jit_comp.c +1 -0
@@ -1367,6 +1411,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 	const s16 off = insn->off;
 	const s32 imm = insn->imm;
 	const int i = insn - ctx->prog->insnsi;
+	const bool in_pad = bpf_cleanup_insn_in_pad(ctx->prog, i);
 	const bool is64 = BPF_CLASS(code) == BPF_ALU64 ||
 			  BPF_CLASS(code) == BPF_JMP;
 	u8 jmp_cond;
@@ -1378,9 +1423,14 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 	int ret;
 	bool sign_extend;

arch/arm64/net/bpf_jit_comp.c static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn · line 1433

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +6 -1
 
-	if (bpf_insn_is_indirect_target(env, ctx->prog, i))
+	if (bpf_insn_is_indirect_target(env, ctx->prog, i) ||
+	    bpf_cleanup_insn_is_pad(ctx->prog, i))
 		emit_bti(A64_BTI_J, ctx);
 
+	if (bpf_cleanup_insn_is_pad(ctx->prog, i))
+		emit(A64_SUB_I(1, A64_CLEANUP_FP, fp,
+			       ctx->stack_size + ctx->stack_arg_size), ctx);
+

This isn't a bug, but could the two bpf_cleanup_insn_is_pad() calls share a local the way in_pad does, and could the throw-site check further down use the existing 'i' instead of recomputing insn - ctx->prog->insnsi?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#31 lore raw

Re: [PATCH bpf-next v2 12/20] bpf, arm64: Dispatch exception cleanup pads at run time

Quoted 23 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +0 -0
index 6c04fee468766..560eba305bca2 100644

[ ... ]

Quoted 17 lines
arch/arm64/net/bpf_jit_comp.c +12 -0
@@ -75,7 +76,21 @@ static const int bpf2a64[] = {
  	[ARENA_VM_START] = A64_R(28),
  };
 
+/* Throw-site spill: the five pairs push_callee_regs() forces on, same size and
+ * slot order, so arch_bpf_run_cleanup_pad() reads both alike.
+ */
+#define A64_CLEANUP_SPILL_SZ	(5 * 16)
+
+/*
+ * Where a landing pad's frame is anchored, since the stack pointer generated
+ * code normally addresses it through is the walker's inside a pad. bpf2a64[]
+ * maps nothing to x24, so nothing else in generated code touches it.
+ */
+#define A64_CLEANUP_FP		A64_R(24)
+
  struct jit_ctx {
Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c static const int bpf2a64[] = { · line 96

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +2 -0
+	/* Bytes reserved for the throw-site spill; see bpf_cleanup_force_spill(). */
+	u32 throw_spill;
  	const struct bpf_prog *prog;
  	int idx;
  	int epilogue_offset;

This isn't a bug, but would it read better to put throw_spill next to stack_size and stack_arg_size, which it is always used with, rather than ahead of prog at the head of the struct?

Okay, will do.

Quoted 20 lines

[ ... ]

Quoted 17 lines
arch/arm64/net/bpf_jit_comp.c +3 -1
@@ -1367,6 +1411,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
  	const s16 off = insn->off;
  	const s32 imm = insn->imm;
  	const int i = insn - ctx->prog->insnsi;
+	const bool in_pad = bpf_cleanup_insn_in_pad(ctx->prog, i);
  	const bool is64 = BPF_CLASS(code) == BPF_ALU64 ||
  			  BPF_CLASS(code) == BPF_JMP;
  	u8 jmp_cond;
@@ -1378,9 +1423,14 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
  	int ret;
  	bool sign_extend;
 
-	if (bpf_insn_is_indirect_target(env, ctx->prog, i))
+	if (bpf_insn_is_indirect_target(env, ctx->prog, i) ||
+	    bpf_cleanup_insn_is_pad(ctx->prog, i))
  		emit_bti(A64_BTI_J, ctx);
 
Quoted 8 lines

arch/arm64/net/bpf_jit_comp.c static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn · line 1433

Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +4 -0
+	if (bpf_cleanup_insn_is_pad(ctx->prog, i))
+		emit(A64_SUB_I(1, A64_CLEANUP_FP, fp,
+			       ctx->stack_size + ctx->stack_arg_size), ctx);
+

This isn't a bug, but could the two bpf_cleanup_insn_is_pad() calls share a local the way in_pad does, and could the throw-site check further down use the existing 'i' instead of recomputing insn - ctx->prog->insnsi?

Will do.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#32 lore raw

PATCH v2 13/20 libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc

In reply to Yonghong Song

LLVM terminates a cleanup landing pad with a call to _Unwind_Resume: the base unwind ABI's entry point for carrying an unwind on once a frame's cleanups have run. The kernel provides that terminator as a kfunc, but not under that name: claiming _Unwind_Resume in the kernel's own symbol table, for a function whose body never runs, would be needlessly confusing, so it is called bpf_unwind_resume.

Both load paths take the detour. A direct load resolves the name against the kernel's BTF while libbpf runs. A light skeleton instead writes the name into the loader program's blob of bytes, for that program to resolve when it runs, so the name recorded there has to be translated as well.

---
 tools/lib/bpf/libbpf.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
tools/lib/bpf/libbpf.c +16 -5
index 33ff61b151f3..27f2eec13f76 100644
@@ -8351,6 +8351,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s
 	}
 }
 
+/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the
+ * base unwind ABI's entry point for carrying an unwind on once a frame's
+ * cleanups have run. The kernel knows it as bpf_unwind_resume.
+ */
+static const char *kern_extern_name(const char *name)
+{
+	return strcmp(name, "_Unwind_Resume") ? name : "bpf_unwind_resume";
+}
+
 static int bpf_program_record_relos(struct bpf_program *prog)
 {
 	struct bpf_object *obj = prog->obj;
@@ -8367,12 +8376,12 @@ static int bpf_program_record_relos(struct bpf_program *prog)
 				continue;
 			kind = btf_is_var(btf__type_by_id(obj->btf, ext->btf_id)) ?
 				BTF_KIND_VAR : BTF_KIND_FUNC;
-			bpf_gen__record_extern(obj->gen_loader, ext->name,
+			bpf_gen__record_extern(obj->gen_loader, kern_extern_name(ext->name),
 					       ext->is_weak, !ext->ksym.type_id,
 					       true, kind, relo->insn_idx);
 			break;
 		case RELO_EXTERN_CALL:
-			bpf_gen__record_extern(obj->gen_loader, ext->name,
+			bpf_gen__record_extern(obj->gen_loader, kern_extern_name(ext->name),
 					       ext->is_weak, false, false, BTF_KIND_FUNC,
 					       relo->insn_idx);
 			break;
@@ -8812,17 +8821,19 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,
 	struct module_btf *mod_btf = NULL;
 	const struct btf_type *kern_func;
 	struct btf *kern_btf = NULL;
+	const char *kern_name;
 	int ret;
 
 	local_func_proto_id = ext->ksym.type_id;
 
-	kfunc_id = find_ksym_btf_id(obj, ext->essent_name ?: ext->name, BTF_KIND_FUNC, &kern_btf,
-				    &mod_btf);
+	kern_name = kern_extern_name(ext->essent_name ?: ext->name);
+
+	kfunc_id = find_ksym_btf_id(obj, kern_name, BTF_KIND_FUNC, &kern_btf, &mod_btf);
 	if (kfunc_id < 0) {
 		if (kfunc_id == -ESRCH && ext->is_weak)
 			return 0;
 		pr_warn("extern (func ksym) '%s': not found in kernel or module BTFs\n",
-			ext->name);
+			strcmp(kern_name, "bpf_unwind_resume") ? ext->name : kern_name);
 		return kfunc_id;
 	}
 
Signature · 1 line
2.53.0-Meta
author reviewer
#33 lore raw

PATCH v2 14/20 libbpf: Add cleanup_info to bpf_prog_load_opts

In reply to Yonghong Song

Let a caller hand the kernel an exception cleanup table. Extend struct bpf_prog_load_opts with cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, pass them through to BPF_PROG_LOAD, and grow the attr size bpf_prog_load() computes to cover the new fields.

---
 tools/lib/bpf/bpf.c | 6 +++++-
 tools/lib/bpf/bpf.h | 7 ++++++-
 2 files changed, 11 insertions(+), 2 deletions(-)
tools/lib/bpf/bpf.c +5 -1
index 96819c082c77..bcf490570961 100644
@@ -295,7 +295,7 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
 		  const struct bpf_insn *insns, size_t insn_cnt,
 		  struct bpf_prog_load_opts *opts)
 {
-	const size_t attr_sz = offsetofend(union bpf_attr, keyring_id);
+	const size_t attr_sz = offsetofend(union bpf_attr, cleanup_info_cnt);
 	void *finfo = NULL, *linfo = NULL;
 	const char *func_info, *line_info;
 	__u32 log_size, log_level, attach_prog_fd, attach_btf_obj_fd;
@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
 	attr.fd_array = ptr_to_u64(OPTS_GET(opts, fd_array, NULL));
 	attr.fd_array_cnt = OPTS_GET(opts, fd_array_cnt, 0);
 
+	attr.cleanup_info = ptr_to_u64(OPTS_GET(opts, cleanup_info, NULL));
+	attr.cleanup_info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+	attr.cleanup_info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);
+
 	if (log_level) {
 		attr.log_buf = ptr_to_u64(log_buf);
 		attr.log_size = log_size;
tools/lib/bpf/bpf.h +6 -1
index 7534a593edae..6f62a99e1e3e 100644
@@ -128,9 +128,14 @@ struct bpf_prog_load_opts {
 
 	/* if set, provides the length of fd_array */
 	__u32 fd_array_cnt;
+
+	/* exception cleanup table, from the .bpf_cleanup section */
+	const void *cleanup_info;
+	__u32 cleanup_info_cnt;
+	__u32 cleanup_info_rec_size;
 	size_t :0;
 };
-#define bpf_prog_load_opts__last_field fd_array_cnt
+#define bpf_prog_load_opts__last_field cleanup_info_rec_size
 
 LIBBPF_API int bpf_prog_load(enum bpf_prog_type prog_type,
 			     const char *prog_name, const char *license,
Signature · 1 line
2.53.0-Meta
#34 lore raw

Re: [PATCH bpf-next v2 14/20] libbpf: Add cleanup_info to bpf_prog_load_opts

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] libbpf: missing E2BIG fallback for cleanup_info breaks forward compatibility --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] libbpf: missing E2BIG fallback for cleanup_info breaks forward compatibility --

commit d67b69538e39a329708fcc9fa943f0ad05afaa05
Author: Yonghong Song <[email protected]>

libbpf: Add cleanup_info to bpf_prog_load_opts

Let a caller hand the kernel an exception cleanup table. Extend struct bpf_prog_load_opts with cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, pass them through to BPF_PROG_LOAD, and grow the attr size bpf_prog_load() computes to cover the new fields.

Quoted 4 lines
tools/lib/bpf/bpf.c +0 -0
index 96819c082c772..bcf490570961c 100644

[ ... ]

tools/lib/bpf/bpf.c int bpf_prog_load(enum bpf_prog_type prog_type, · line 375

Quoted 7 lines
tools/lib/bpf/bpf.c +3 -0
@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
 	attr.fd_array = ptr_to_u64(OPTS_GET(opts, fd_array, NULL));
 	attr.fd_array_cnt = OPTS_GET(opts, fd_array_cnt, 0);
 
+	attr.cleanup_info = ptr_to_u64(OPTS_GET(opts, cleanup_info, NULL));
+	attr.cleanup_info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+	attr.cleanup_info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);

[Severity: Medium]
Will this break forward compatibility for bpf_prog_load() if cleanup_info triggers an -E2BIG error?

Later in bpf_prog_load(), there is a retry loop to handle -E2BIG when the userspace-provided record size is larger than what the kernel supports:

	while (errno == E2BIG && (!finfo || !linfo)) {
		if (!finfo && attr.func_info_cnt &&
		    attr.func_info_rec_size < func_info_rec_size) {
            /* try with corrected func info records */
            ...

The loop condition and body don't seem to include checks for cleanup_info.

If the kernel returns -E2BIG because of a larger cleanup_info_rec_size, won't this hit the else { break; } block or skip the loop entirely, causing the load to fail instead of retrying with truncated records?

Quoted 4 lines
tools/lib/bpf/bpf.c +1 -0
+
 	if (log_level) {
 		attr.log_buf = ptr_to_u64(log_buf);
 		attr.log_size = log_size;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
author reviewer
#35 lore raw

Re: [PATCH bpf-next v2 14/20] libbpf: Add cleanup_info to bpf_prog_load_opts

Quoted 39 lines · On 9/17/26 9:57 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] libbpf: missing E2BIG fallback for cleanup_info breaks forward compatibility --

commit d67b69538e39a329708fcc9fa943f0ad05afaa05
Author: Yonghong Song <[email protected]>

libbpf: Add cleanup_info to bpf_prog_load_opts

Let a caller hand the kernel an exception cleanup table. Extend struct bpf_prog_load_opts with cleanup_info, cleanup_info_cnt and cleanup_info_rec_size, pass them through to BPF_PROG_LOAD, and grow the attr size bpf_prog_load() computes to cover the new fields.

Quoted 4 lines
tools/lib/bpf/bpf.c +0 -0
index 96819c082c772..bcf490570961c 100644

[ ... ]

tools/lib/bpf/bpf.c int bpf_prog_load(enum bpf_prog_type prog_type, · line 375

Quoted 7 lines
tools/lib/bpf/bpf.c +3 -0
@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
  	attr.fd_array = ptr_to_u64(OPTS_GET(opts, fd_array, NULL));
  	attr.fd_array_cnt = OPTS_GET(opts, fd_array_cnt, 0);
 
+	attr.cleanup_info = ptr_to_u64(OPTS_GET(opts, cleanup_info, NULL));
+	attr.cleanup_info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+	attr.cleanup_info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);

[Severity: Medium]
Will this break forward compatibility for bpf_prog_load() if cleanup_info triggers an -E2BIG error?

Later in bpf_prog_load(), there is a retry loop to handle -E2BIG when the userspace-provided record size is larger than what the kernel supports:

	while (errno == E2BIG && (!finfo || !linfo)) {
		if (!finfo && attr.func_info_cnt &&
		    attr.func_info_rec_size < func_info_rec_size) {
             /* try with corrected func info records */
             ...
Quoted 5 lines

The loop condition and body don't seem to include checks for cleanup_info.

If the kernel returns -E2BIG because of a larger cleanup_info_rec_size, won't this hit the else { break; } block or skip the loop entirely, causing the load to fail instead of retrying with truncated records?

This is the first version to support .bpf_cleanup. It is too early to talk about the future extension for .bpf_cleanup although we do have some room to grow struct size if needed:
#define MAX_FUNCINFO_REC_SIZE 252
#define MAX_CLEANUP_INFO_REC_SIZE MAX_FUNCINFO_REC_SIZE

Quoted 5 lines
Quoted 5 lines
tools/lib/bpf/bpf.c +1 -0
+
  	if (log_level) {
  		attr.log_buf = ptr_to_u64(log_buf);
  		attr.log_size = log_size;
author reviewer
#36 lore raw

PATCH v2 15/20 libbpf: Collect .bpf_cleanup records and pass them to the kernel

In reply to Yonghong Song

Parse the compiler-emitted .bpf_cleanup section and hand the resulting table to BPF_PROG_LOAD.

Each record is three 4-byte fields, and each field is a byte offset into some code section named by a matching .rel.bpf_cleanup relocation. bpf_object__init_cleanup_info() resolves both halves once at open time and keeps (section, instruction index) pairs; it rejects a record whose field has no relocation, whose relocation is not one of the two 32-bit types that spell a data reference to a code section -- R_BPF_64_NODYLD32 from LLVM, R_BPF_64_ABS32 from GNU as -- or whose offset is not instruction aligned.

The records are sorted by begin_off once the offsets are final: the kernel wants the table sorted with disjoint ranges so that it can find the record covering a call site with a binary search, and records arrive in .bpf_cleanup order, which says nothing about where the subprograms they describe were appended. Overlapping ranges are reported here, where the program name and both regions are still at hand.

bpf_object_load_prog() then passes the per-program table through the bpf_prog_load() options added in the previous patch, with the record size carried on the program the way func_info and line_info carry theirs rather than taken from a sizeof() at the call site. bpf_program__clone() carries it too. That is a second load path -- the one veristat uses -- and without the table the kernel sees landing pads nothing reaches and refuses the program with "unreachable insn".

---
 tools/lib/bpf/libbpf.c          | 291 ++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |   3 +
 2 files changed, 294 insertions(+)
tools/lib/bpf/libbpf.c +291 -0
index 27f2eec13f76..1b8b978e7b7b 100644
@@ -514,6 +514,11 @@ struct bpf_program {
 	void *line_info;
 	__u32 line_info_rec_size;
 	__u32 line_info_cnt;
+
+	struct bpf_cleanup_info *cleanup_info;
+	__u32 cleanup_info_rec_size;
+	__u32 cleanup_info_cnt;
+
 	__u32 prog_flags;
 	__u8  hash[SHA256_DIGEST_LENGTH];
 
@@ -549,6 +554,7 @@ struct bpf_struct_ops {
 #define STRUCT_OPS_SEC ".struct_ops"
 #define STRUCT_OPS_LINK_SEC ".struct_ops.link"
 #define ARENA_SEC ".addr_space.1"
+#define CLEANUP_SEC ".bpf_cleanup"
 
 enum libbpf_map_type {
 	LIBBPF_MAP_UNSPEC,
@@ -677,6 +683,25 @@ struct elf_sec_desc {
 	Elf_Data *data;
 };
 
+#define CLEANUP_REC_FIELDS	(sizeof(struct bpf_cleanup_info) / sizeof(__u32))
+
+/* Index of each field of struct bpf_cleanup_info, read as an array of __u32. */
+enum {
+	CLEANUP_REC_BEGIN,
+	CLEANUP_REC_END,
+	CLEANUP_REC_PAD,
+};
+
+/* One (begin, end, landing_pad) triple from .bpf_cleanup, with each field
+ * resolved from its relocation to an ELF section plus a section-relative
+ * instruction index. The mapping to final program instruction indices can only
+ * happen after subprogram placement, which differs per main program.
+ */
+struct cleanup_raw_rec {
+	int sec_idx[CLEANUP_REC_FIELDS];
+	size_t insn_idx[CLEANUP_REC_FIELDS];
+};
+
 struct elf_state {
 	int fd;
 	const void *obj_buf;
@@ -696,6 +721,8 @@ struct elf_state {
 	bool has_st_ops;
 	int arena_data_shndx;
 	int jumptables_data_shndx;
+	Elf_Data *cleanup_data;
+	int cleanup_shndx;
 };
 
 struct usdt_manager;
@@ -771,6 +798,9 @@ struct bpf_object {
 	void *jumptables_data;
 	size_t jumptables_data_sz;
 
+	struct cleanup_raw_rec *cleanup_recs;
+	size_t cleanup_rec_cnt;
+
 	struct {
 		struct bpf_program *prog;
 		unsigned int sym_off;
@@ -817,7 +847,10 @@ static void bpf_program__exit(struct bpf_program *prog)
 	zfree(&prog->sec_name);
 	zfree(&prog->insns);
 	zfree(&prog->reloc_desc);
+	zfree(&prog->cleanup_info);
 
+	prog->cleanup_info_rec_size = 0;
+	prog->cleanup_info_cnt = 0;
 	prog->nr_reloc = 0;
 	prog->insns_cnt = 0;
 	prog->sec_idx = -1;
@@ -1559,6 +1592,7 @@ static struct bpf_object *bpf_object__new(const char *path,
 	obj->efile.obj_buf = obj_buf;
 	obj->efile.obj_buf_sz = obj_buf_sz;
 	obj->efile.btf_maps_shndx = -1;
+	obj->efile.cleanup_shndx = -1;
 	obj->kconfig_map_idx = -1;
 	obj->arena_map_idx = -1;
 
@@ -4045,6 +4079,9 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 				sec_desc->shdr = sh;
 				sec_desc->data = data;
 				obj->efile.has_st_ops = true;
+			} else if (strcmp(name, CLEANUP_SEC) == 0) {
+				obj->efile.cleanup_data = data;
+				obj->efile.cleanup_shndx = idx;
 			} else if (strcmp(name, ARENA_SEC) == 0) {
 				obj->efile.arena_data = data;
 				obj->efile.arena_data_shndx = idx;
@@ -4072,6 +4109,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 			    strcmp(name, ".rel" STRUCT_OPS_LINK_SEC) &&
 			    strcmp(name, ".rel?" STRUCT_OPS_SEC) &&
 			    strcmp(name, ".rel?" STRUCT_OPS_LINK_SEC) &&
+			    strcmp(name, ".rel" CLEANUP_SEC) &&
 			    strcmp(name, ".rel" MAPS_ELF_SEC)) {
 				pr_info("elf: skipping relo section(%d) %s for section(%d) %s\n",
 					idx, name, targ_sec_idx,
@@ -4852,6 +4890,221 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,
 	return NULL;
 }
 
+static int bpf_object__init_cleanup_info(struct bpf_object *obj)
+{
+	Elf_Data *data = obj->efile.cleanup_data;
+	Elf_Data *relo = NULL;
+	size_t i, nrels, nslots, nrecs;
+	struct cleanup_raw_rec *recs;
+	int *slot_sec, ret = 0;
+	size_t *slot_val;
+	const __u32 *vals;
+	bool native;
+
+	if (!data || obj->efile.cleanup_shndx < 0)
+		return 0;
+
+	native = is_native_endianness(obj);
+
+	for (i = 0; i < obj->efile.sec_cnt; i++) {
+		struct elf_sec_desc *sd = &obj->efile.secs[i];
+
+		if (sd->sec_type == SEC_RELO && sd->shdr &&
+		    sd->shdr->sh_info == (Elf64_Word)obj->efile.cleanup_shndx) {
+			relo = sd->data;
+			break;
+		}
+	}
+	if (!relo) {
+		pr_warn("%s present without relocations\n", CLEANUP_SEC);
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+	if (data->d_size % sizeof(struct bpf_cleanup_info)) {
+		pr_warn("%s size %zu is not a multiple of the record size %zu\n",
+			CLEANUP_SEC, data->d_size, sizeof(struct bpf_cleanup_info));
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+
+	vals = data->d_buf;
+	nslots = data->d_size / sizeof(__u32);
+	nrecs = data->d_size / sizeof(struct bpf_cleanup_info);
+
+	slot_sec = calloc(nslots, sizeof(*slot_sec));
+	slot_val = calloc(nslots, sizeof(*slot_val));
+	recs = calloc(nrecs ?: 1, sizeof(*recs));
+	if (!slot_sec || !slot_val || !recs) {
+		ret = -ENOMEM;
+		goto out;
+	}
+	for (i = 0; i < nslots; i++)
+		slot_sec[i] = -1;
+
+	/* One relocation per 4-byte field, naming the section it points into. */
+	nrels = relo->d_size / sizeof(Elf64_Rel);
+	for (i = 0; i < nrels; i++) {
+		Elf64_Rel *rel = elf_rel_by_idx(relo, i);
+		Elf64_Sym *sym = elf_sym_by_idx(obj, ELF64_R_SYM(rel->r_info));
+		size_t type = ELF64_R_TYPE(rel->r_info);
+		size_t slot = rel->r_offset / sizeof(__u32);
+
+		if (type != R_BPF_64_NODYLD32 && type != R_BPF_64_ABS32) {
+			pr_warn("%s: relocation %zu has unexpected type %zu\n",
+				CLEANUP_SEC, i, type);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		if (!sym || slot >= nslots || rel->r_offset % sizeof(__u32)) {
+			pr_warn("%s: bad relocation %zu\n", CLEANUP_SEC, i);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		slot_sec[slot] = sym->st_shndx;
+		/* The addend lives in the section data, which libelf leaves in
+		 * the object's byte order; a non-section symbol additionally
+		 * contributes its own value.
+		 */
+		slot_val[slot] = (native ? vals[slot] : bswap_32(vals[slot])) +
+				 sym->st_value;
+	}
+
+	for (i = 0; i < nslots; i++) {
+		struct cleanup_raw_rec *rec = &recs[i / CLEANUP_REC_FIELDS];
+		size_t field = i % CLEANUP_REC_FIELDS;
+
+		if (slot_sec[i] < 0) {
+			pr_warn("%s: field %zu has no relocation\n", CLEANUP_SEC, i);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		if (slot_val[i] % BPF_INSN_SZ) {
+			pr_warn("%s: field %zu offset %zu is not instruction aligned\n",
+				CLEANUP_SEC, i, slot_val[i]);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		rec->sec_idx[field] = slot_sec[i];
+		rec->insn_idx[field] = slot_val[i] / BPF_INSN_SZ;
+	}
+
+	obj->cleanup_recs = recs;
+	obj->cleanup_rec_cnt = nrecs;
+	recs = NULL;
+out:
+	free(recs);
+	free(slot_val);
+	free(slot_sec);
+	return ret;
+}
+
+static int cmp_cleanup_info(const void *a, const void *b)
+{
+	const struct bpf_cleanup_info *x = a, *y = b;
+
+	if (x->begin_off == y->begin_off)
+		return 0;
+	return x->begin_off < y->begin_off ? -1 : 1;
+}
+
+static int bpf_prog_collect_cleanup_info(struct bpf_object *obj,
+					 struct bpf_program *prog)
+{
+	size_t i;
+	int j;
+
+	for (i = 0; i < obj->cleanup_rec_cnt; i++) {
+		struct cleanup_raw_rec *raw = &obj->cleanup_recs[i];
+		struct bpf_program *owner = NULL;
+		struct bpf_cleanup_info ci = {};
+		__u32 fields[CLEANUP_REC_FIELDS];
+		void *tmp;
+
+		for (j = 0; j < CLEANUP_REC_FIELDS; j++) {
+			size_t idx = raw->insn_idx[j], final;
+			struct bpf_program *p;
+
+			/* The end of a range is exclusive, so it may name the
+			 * instruction just past the last one of a function,
+			 * which belongs to the next function or to nothing at
+			 * all. Ask about the last instruction the range covers,
+			 * the way the kernel does.
+			 */
+			if (j == CLEANUP_REC_END) {
+				if (!idx) {
+					pr_warn("%s: record %zu is an empty range\n",
+						CLEANUP_SEC, i);
+					return -LIBBPF_ERRNO__FORMAT;
+				}
+				idx--;
+			}
+
+			p = find_prog_by_sec_insn(obj, raw->sec_idx[j], idx);
+			if (!p) {
+				pr_warn("%s: record %zu field %d is not inside a function\n",
+					CLEANUP_SEC, i, j);
+				return -LIBBPF_ERRNO__FORMAT;
+			}
+			if (!owner) {
+				owner = p;
+			} else if (owner != p) {
+				pr_warn("%s: record %zu spans functions '%s' and '%s'\n",
+					CLEANUP_SEC, i, owner->name, p->name);
+				return -LIBBPF_ERRNO__FORMAT;
+			}
+
+			if (owner == prog) {
+				final = raw->insn_idx[j] - prog->sec_insn_off;
+			} else if (prog_is_subprog(obj, owner) && owner->sub_insn_off) {
+				/* sub_insn_off is where this subprogram was
+				 * appended to the main program being relocated;
+				 * zero means it is not part of it.
+				 */
+				final = owner->sub_insn_off +
+					raw->insn_idx[j] - owner->sec_insn_off;
+			} else {
+				owner = NULL;
+				break;
+			}
+			fields[j] = final;
+		}
+		if (!owner)
+			continue;
+
+		ci.begin_off = fields[CLEANUP_REC_BEGIN];
+		ci.end_off = fields[CLEANUP_REC_END];
+		ci.landing_pad_off = fields[CLEANUP_REC_PAD];
+
+		tmp = libbpf_reallocarray(prog->cleanup_info, prog->cleanup_info_cnt + 1,
+					  sizeof(*prog->cleanup_info));
+		if (!tmp)
+			return -ENOMEM;
+		prog->cleanup_info = tmp;
+		prog->cleanup_info_rec_size = sizeof(struct bpf_cleanup_info);
+		prog->cleanup_info[prog->cleanup_info_cnt++] = ci;
+
+		pr_debug("prog '%s': cleanup region [%u,%u) -> landing pad %u\n",
+			 prog->name, ci.begin_off, ci.end_off, ci.landing_pad_off);
+	}
+
+	if (!prog->cleanup_info_cnt)
+		return 0;
+
+	qsort(prog->cleanup_info, prog->cleanup_info_cnt,
+	      sizeof(*prog->cleanup_info), cmp_cleanup_info);
+	for (i = 1; i < prog->cleanup_info_cnt; i++) {
+		struct bpf_cleanup_info *prev = &prog->cleanup_info[i - 1];
+		struct bpf_cleanup_info *cur = &prog->cleanup_info[i];
+
+		if (cur->begin_off < prev->end_off) {
+			pr_warn("prog '%s': overlapping cleanup regions [%u,%u) and [%u,%u)\n",
+				prog->name, prev->begin_off, prev->end_off,
+				cur->begin_off, cur->end_off);
+			return -LIBBPF_ERRNO__FORMAT;
+		}
+	}
+
+	return 0;
+}
+
 static int
 bpf_object__collect_prog_relos(struct bpf_object *obj, Elf64_Shdr *shdr, Elf_Data *data)
 {
@@ -7561,6 +7814,13 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_pat
 					return err;
 			}
 		}
+
+		err = bpf_prog_collect_cleanup_info(obj, prog);
+		if (err) {
+			pr_warn("prog '%s': failed to collect cleanup info: %s\n",
+				prog->name, errstr(err));
+			return err;
+		}
 	}
 	for (i = 0; i < obj->nr_programs; i++) {
 		prog = &obj->programs[i];
@@ -7751,6 +8011,9 @@ static int bpf_object__collect_relos(struct bpf_object *obj)
 			return -LIBBPF_ERRNO__INTERNAL;
 		}
 
+		if (idx == obj->efile.cleanup_shndx)
+			continue;
+
 		if (obj->efile.secs[idx].sec_type == SEC_ST_OPS)
 			err = bpf_object__collect_st_ops_relos(obj, shdr, data);
 		else if (idx == obj->efile.btf_maps_shndx)
@@ -8023,6 +8286,11 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
 		load_attr.line_info_rec_size = prog->line_info_rec_size;
 		load_attr.line_info_cnt = prog->line_info_cnt;
 	}
+	if (prog->cleanup_info_cnt) {
+		load_attr.cleanup_info = prog->cleanup_info;
+		load_attr.cleanup_info_cnt = prog->cleanup_info_cnt;
+		load_attr.cleanup_info_rec_size = prog->cleanup_info_rec_size;
+	}
 	load_attr.log_level = log_level;
 	load_attr.prog_flags = prog->prog_flags;
 	load_attr.fd_array = obj->fd_array;
@@ -8579,6 +8847,7 @@ static struct bpf_object *bpf_object_open(const char *path, const void *obj_buf,
 	err = err ? : bpf_object__init_maps(obj, opts);
 	err = err ? : bpf_object_init_progs(obj, opts);
 	err = err ? : bpf_object__collect_relos(obj);
+	err = err ? : bpf_object__init_cleanup_info(obj);
 	if (err)
 		goto out;
 
@@ -9692,6 +9961,9 @@ void bpf_object__close(struct bpf_object *obj)
 	zfree(&obj->jumptables_data);
 	obj->jumptables_data_sz = 0;
 
+	zfree(&obj->cleanup_recs);
+	obj->cleanup_rec_cnt = 0;
+
 	for (i = 0; i < obj->jumptable_map_cnt; i++)
 		close(obj->jumptable_maps[i].fd);
 	zfree(&obj->jumptable_maps);
@@ -10084,6 +10356,25 @@ int bpf_program__clone(struct bpf_program *prog, const struct bpf_prog_load_opts
 		attr.line_info_rec_size = info ? info_rec_size : prog->line_info_rec_size;
 	}
 
+	/* exception cleanup table */
+	info = OPTS_GET(opts, cleanup_info, NULL);
+	info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+	info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);
+	if (!!info != !!info_cnt || !!info != !!info_rec_size) {
+		pr_warn("prog '%s': cleanup_info, cleanup_info_cnt, and cleanup_info_rec_size must all be specified or all omitted\n",
+			prog->name);
+		return libbpf_err(-EINVAL);
+	}
+	if (info) {
+		attr.cleanup_info = info;
+		attr.cleanup_info_cnt = info_cnt;
+		attr.cleanup_info_rec_size = info_rec_size;
+	} else if (prog->cleanup_info_cnt) {
+		attr.cleanup_info = prog->cleanup_info;
+		attr.cleanup_info_cnt = prog->cleanup_info_cnt;
+		attr.cleanup_info_rec_size = prog->cleanup_info_rec_size;
+	}
+
 	/* Logging is caller-controlled; no fallback to prog/obj log settings */
 	attr.log_buf = OPTS_GET(opts, log_buf, NULL);
 	attr.log_size = OPTS_GET(opts, log_size, 0);
tools/lib/bpf/libbpf_internal.h +3 -0
index cb4d96233844..3ba6d9090368 100644
@@ -56,6 +56,9 @@
 #ifndef R_BPF_64_ABS32
 #define R_BPF_64_ABS32 3
 #endif
+#ifndef R_BPF_64_NODYLD32
+#define R_BPF_64_NODYLD32 4
+#endif
 #ifndef R_BPF_64_32
 #define R_BPF_64_32 10
 #endif
Signature · 1 line
2.53.0-Meta
#37 lore raw

Re: [PATCH bpf-next v2 15/20] libbpf: Collect .bpf_cleanup records and pass them to the kernel

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] bpf: empty cleanup range check in bpf_prog_collect_cleanup_info() only catches offset 0…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] bpf: empty cleanup range check in bpf_prog_collect_cleanup_info() only catches offset 0
- [Medium] bpf: bpf_object__init_cleanup_info() rejects valid objects with 0-sized .bpf_cleanup sections --

commit a645d3bd249aceee4909edcc78e5fe4247fd3432
Author: Yonghong Song <[email protected]>

libbpf: Collect .bpf_cleanup records and pass them to the kernel

Parse the compiler-emitted .bpf_cleanup section and hand the resulting table to BPF_PROG_LOAD.

Quoted 4 lines
tools/lib/bpf/libbpf.c +0 -0
index 27f2eec13f76f..1b8b978e7b7b9 100644

[ ... ]

Quoted 23 lines
tools/lib/bpf/libbpf.c +19 -0
@@ -4852,6 +4890,221 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,
 	return NULL;
 }
 
+static int bpf_object__init_cleanup_info(struct bpf_object *obj)
+{
+	Elf_Data *data = obj->efile.cleanup_data;
+	Elf_Data *relo = NULL;
+	size_t i, nrels, nslots, nrecs;
+	struct cleanup_raw_rec *recs;
+	int *slot_sec, ret = 0;
+	size_t *slot_val;
+	const __u32 *vals;
+	bool native;
+
+	if (!data || obj->efile.cleanup_shndx < 0)
+		return 0;
+
+	native = is_native_endianness(obj);
+
+	for (i = 0; i < obj->efile.sec_cnt; i++) {
+		struct elf_sec_desc *sd = &obj->efile.secs[i];
+

tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 4921

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+		if (sd->sec_type == SEC_RELO && sd->shdr &&
+		    sd->shdr->sh_info == (Elf64_Word)obj->efile.cleanup_shndx) {
+			relo = sd->data;
+			break;
+		}
+	}
+	if (!relo) {
+		pr_warn("%s present without relocations\n", CLEANUP_SEC);
+		return -LIBBPF_ERRNO__FORMAT;
+	}

[Severity: Medium]
Could this reject valid objects if the compiler emits an empty 0-byte .bpf_cleanup section?

If the section is empty (e.g., exceptions were optimized out), no relocations are emitted and there is no corresponding .rel.bpf_cleanup section. However, bpf_object__init_cleanup_info() requires the relocation section to exist if .bpf_cleanup is present.

Would it be better to return early if data->d_size is 0 before checking for relocations?

[ ... ]

Quoted 21 lines
tools/lib/bpf/libbpf.c +21 -0
+static int bpf_prog_collect_cleanup_info(struct bpf_object *obj,
+					 struct bpf_program *prog)
+{
+	size_t i;
+	int j;
+
+	for (i = 0; i < obj->cleanup_rec_cnt; i++) {
+		struct cleanup_raw_rec *raw = &obj->cleanup_recs[i];
+		struct bpf_program *owner = NULL;
+		struct bpf_cleanup_info ci = {};
+		__u32 fields[CLEANUP_REC_FIELDS];
+		void *tmp;
+
+		for (j = 0; j < CLEANUP_REC_FIELDS; j++) {
+			size_t idx = raw->insn_idx[j], final;
+			struct bpf_program *p;
+
+			/* The end of a range is exclusive, so it may name the
+			 * instruction just past the last one of a function,
+			 * which belongs to the next function or to nothing at
+			 * all. Ask about the last instruction the range covers,

tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 5038

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+			 * the way the kernel does.
+			 */
+			if (j == CLEANUP_REC_END) {
+				if (!idx) {
+					pr_warn("%s: record %zu is an empty range\n",
+						CLEANUP_SEC, i);
+					return -LIBBPF_ERRNO__FORMAT;
+				}
+				idx--;
+			}

[Severity: Medium]
Will this check catch empty ranges emitted by the compiler at non-zero offsets?

If the compiler emits a .bpf_cleanup record where begin and end are both 5, the check for !idx evaluates to false. The code will then decrement idx to 4 and look up the program for the previous instruction.

If this empty range is located at the start of a function, looking up the previous instruction will fall into the preceding function, potentially triggering a spurious "spans functions" error or silently accepting invalid records.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
author reviewer
#38 lore raw

Re: [PATCH bpf-next v2 15/20] libbpf: Collect .bpf_cleanup records and pass them to the kernel

Quoted 56 lines · On 9/17/26 10:00 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] bpf: empty cleanup range check in bpf_prog_collect_cleanup_info() only catches offset 0
- [Medium] bpf: bpf_object__init_cleanup_info() rejects valid objects with 0-sized .bpf_cleanup sections --

commit a645d3bd249aceee4909edcc78e5fe4247fd3432
Author: Yonghong Song <[email protected]>

libbpf: Collect .bpf_cleanup records and pass them to the kernel

Parse the compiler-emitted .bpf_cleanup section and hand the resulting table to BPF_PROG_LOAD.

Quoted 4 lines
tools/lib/bpf/libbpf.c +0 -0
index 27f2eec13f76f..1b8b978e7b7b9 100644

[ ... ]

Quoted 23 lines
tools/lib/bpf/libbpf.c +19 -0
@@ -4852,6 +4890,221 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,
  	return NULL;
  }
 
+static int bpf_object__init_cleanup_info(struct bpf_object *obj)
+{
+	Elf_Data *data = obj->efile.cleanup_data;
+	Elf_Data *relo = NULL;
+	size_t i, nrels, nslots, nrecs;
+	struct cleanup_raw_rec *recs;
+	int *slot_sec, ret = 0;
+	size_t *slot_val;
+	const __u32 *vals;
+	bool native;
+
+	if (!data || obj->efile.cleanup_shndx < 0)
+		return 0;
+
+	native = is_native_endianness(obj);
+
+	for (i = 0; i < obj->efile.sec_cnt; i++) {
+		struct elf_sec_desc *sd = &obj->efile.secs[i];
+

tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 4921

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+		if (sd->sec_type == SEC_RELO && sd->shdr &&
+		    sd->shdr->sh_info == (Elf64_Word)obj->efile.cleanup_shndx) {
+			relo = sd->data;
+			break;
+		}
+	}
+	if (!relo) {
+		pr_warn("%s present without relocations\n", CLEANUP_SEC);
+		return -LIBBPF_ERRNO__FORMAT;
+	}

[Severity: Medium]
Could this reject valid objects if the compiler emits an empty 0-byte .bpf_cleanup section?

Quoted 7 lines

If the section is empty (e.g., exceptions were optimized out), no relocations are emitted and there is no corresponding .rel.bpf_cleanup section. However, bpf_object__init_cleanup_info() requires the relocation section to exist if .bpf_cleanup is present.

Would it be better to return early if data->d_size is 0 before checking for relocations?

I think it is okay, as you mentioned, there will be one error message later when relocation section checks.

Quoted 38 lines

[ ... ]

Quoted 21 lines
tools/lib/bpf/libbpf.c +21 -0
+static int bpf_prog_collect_cleanup_info(struct bpf_object *obj,
+					 struct bpf_program *prog)
+{
+	size_t i;
+	int j;
+
+	for (i = 0; i < obj->cleanup_rec_cnt; i++) {
+		struct cleanup_raw_rec *raw = &obj->cleanup_recs[i];
+		struct bpf_program *owner = NULL;
+		struct bpf_cleanup_info ci = {};
+		__u32 fields[CLEANUP_REC_FIELDS];
+		void *tmp;
+
+		for (j = 0; j < CLEANUP_REC_FIELDS; j++) {
+			size_t idx = raw->insn_idx[j], final;
+			struct bpf_program *p;
+
+			/* The end of a range is exclusive, so it may name the
+			 * instruction just past the last one of a function,
+			 * which belongs to the next function or to nothing at
+			 * all. Ask about the last instruction the range covers,

tools/lib/bpf/libbpf.c static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj, · line 5038

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+			 * the way the kernel does.
+			 */
+			if (j == CLEANUP_REC_END) {
+				if (!idx) {
+					pr_warn("%s: record %zu is an empty range\n",
+						CLEANUP_SEC, i);
+					return -LIBBPF_ERRNO__FORMAT;
+				}
+				idx--;
+			}

[Severity: Medium]
Will this check catch empty ranges emitted by the compiler at non-zero offsets?

Quoted 8 lines

If the compiler emits a .bpf_cleanup record where begin and end are both 5, the check for !idx evaluates to false. The code will then decrement idx to 4 and look up the program for the previous instruction.

If this empty range is located at the start of a function, looking up the previous instruction will fall into the preceding function, potentially triggering a spurious "spans functions" error or silently accepting invalid records.

Good point. Will fix this corner case.

Quoted 1 line
author reviewer
#39 lore raw

PATCH v2 16/20 libbpf: Carry the exception cleanup table through the light skeleton

In reply to Yonghong Song

A light skeleton does not call bpf_prog_load(). bpf_gen__prog_load() builds its own union bpf_attr field by field, and the loader program it emits is what issues BPF_PROG_LOAD when the skeleton runs -- so a program loaded this way reached the kernel without the table the previous patch collected for it. The load then fails, because the landing pads are code nothing reaches and the verifier says so. It says "unreachable insn", which names neither the skeleton nor the table.

Carry it the way func_info and line_info are carried: the records go into the loader's blob of bytes, the count and record size into the attr, and a relocation stores the blob's address into attr.cleanup_info once that address is known. The attr grows to its new last field, cleanup_info_cnt. Records are 4-byte fields like the other info blobs, so a cross-endian build has to swap them too.

Growing the attr is what makes the relocation conditional. Every other attr in this file stops at the last field it sets, and this one now runs to the end of the union: a kernel that predates cleanup_info accepts an attr longer than its own only while the tail it does not know reads as zero. The count and the record size are already zero for a program with no table, but the relocation stores a blob address, which never is, so a program that has no landing pads only keeps loading on such a kernel if the relocation is left out.

---
 tools/lib/bpf/gen_loader.c      | 29 +++++++++++++++++++++++++----
 tools/lib/bpf/libbpf_internal.h |  7 +++++++
 2 files changed, 32 insertions(+), 4 deletions(-)
tools/lib/bpf/gen_loader.c +25 -4
index af3a04f161ac..2345fbdd46f5 100644
@@ -981,13 +981,15 @@ static void cleanup_relos(struct bpf_gen *gen, int insns)
 	cleanup_core_relo(gen);
 }
 
-/* Convert func, line, and core relo info blobs to target endianness */
+/* Convert func, line, core relo and cleanup info blobs to target endianness */
 static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
-			    int core_relos, struct bpf_prog_load_opts *load_attr)
+			    int core_relos, int cleanup_info,
+			    struct bpf_prog_load_opts *load_attr)
 {
 	struct bpf_func_info *fi = gen->data_start + func_info;
 	struct bpf_line_info *li = gen->data_start + line_info;
 	struct bpf_core_relo *cr = gen->data_start + core_relos;
+	struct bpf_cleanup_info *ci = gen->data_start + cleanup_info;
 	int i;
 
 	for (i = 0; i < load_attr->func_info_cnt; i++)
@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
 
 	for (i = 0; i < gen->core_relo_cnt; i++)
 		bpf_core_relo_bswap(cr++);
+
+	for (i = 0; i < load_attr->cleanup_info_cnt; i++)
+		bpf_cleanup_info_bswap(ci++);
 }
 
 void bpf_gen__prog_load(struct bpf_gen *gen,
@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 			       load_attr->line_info_rec_size;
 	int core_relo_tot_sz = gen->core_relo_cnt *
 			       sizeof(struct bpf_core_relo);
+	int cleanup_info_tot_sz = load_attr->cleanup_info_cnt *
+				  load_attr->cleanup_info_rec_size;
 	int prog_load_attr, license_off, insns_off, func_info, line_info, core_relos;
-	int attr_size = offsetofend(union bpf_attr, core_relo_rec_size);
+	int attr_size = offsetofend(union bpf_attr, cleanup_info_cnt);
+	int cleanup_info;
 	union bpf_attr attr;
 
 	memset(&attr, 0, attr_size);
@@ -1061,9 +1069,17 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 		 core_relos, gen->core_relo_cnt,
 		 sizeof(struct bpf_core_relo));
 
+	attr.cleanup_info_rec_size = tgt_endian(load_attr->cleanup_info_rec_size);
+	attr.cleanup_info_cnt = tgt_endian(load_attr->cleanup_info_cnt);
+	cleanup_info = add_data(gen, load_attr->cleanup_info, cleanup_info_tot_sz);
+	pr_debug("gen: prog_load: cleanup_info: off %d cnt %u rec size %u\n",
+		 cleanup_info, load_attr->cleanup_info_cnt,
+		 load_attr->cleanup_info_rec_size);
+
 	/* convert all info blobs to target endianness */
 	if (gen->swapped_endian && !gen->error)
-		info_blob_bswap(gen, func_info, line_info, core_relos, load_attr);
+		info_blob_bswap(gen, func_info, line_info, core_relos, cleanup_info,
+				load_attr);
 
 	libbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name));
 	prog_load_attr = add_data(gen, &attr, attr_size);
@@ -1085,6 +1101,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 	/* populate union bpf_attr with a pointer to core_relos */
 	emit_rel_store(gen, attr_field(prog_load_attr, core_relos), core_relos);
 
+	/* populate union bpf_attr with a pointer to cleanup_info, if there is one */
+	if (load_attr->cleanup_info_cnt)
+		emit_rel_store(gen, attr_field(prog_load_attr, cleanup_info),
+			       cleanup_info);
+
 	/* populate union bpf_attr fd_array with a pointer to data where map_fds are saved */
 	emit_rel_store(gen, attr_field(prog_load_attr, fd_array), gen->fd_array);
 
tools/lib/bpf/libbpf_internal.h +7 -0
index 3ba6d9090368..78519f24fb40 100644
@@ -572,6 +572,13 @@ static inline void bpf_core_relo_bswap(struct bpf_core_relo *i)
 	i->kind = bswap_32(i->kind);
 }
 
+static inline void bpf_cleanup_info_bswap(struct bpf_cleanup_info *i)
+{
+	i->begin_off = bswap_32(i->begin_off);
+	i->end_off = bswap_32(i->end_off);
+	i->landing_pad_off = bswap_32(i->landing_pad_off);
+}
+
 enum btf_field_iter_kind {
 	BTF_FIELD_ITER_IDS,
 	BTF_FIELD_ITER_STRS,
Signature · 1 line
2.53.0-Meta
#40 lore raw

Re: [PATCH bpf-next v2 16/20] libbpf: Carry the exception cleanup table through the light skeleton

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] libbpf: integer overflow in cleanup_info_tot_sz leads to heap OOB read/write --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] libbpf: integer overflow in cleanup_info_tot_sz leads to heap OOB read/write --

commit b9300a1f6022da2aa8b343e05882b46e9f329c64
Author: Yonghong Song <[email protected]>

libbpf: Carry the exception cleanup table through the light skeleton

This patch modifies the light skeleton generation to carry the exception cleanup table (cleanup_info), similar to func_info and line_info. It updates bpf_gen__prog_load() to allocate space for the records, set up union bpf_attr, and perform endianness swapping for cross-compiled cases.

Quoted 4 lines
tools/lib/bpf/gen_loader.c +0 -0
index af3a04f161ac1..2345fbdd46f56 100644

[ ... ]

tools/lib/bpf/gen_loader.c static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info, · line 1005

Quoted 7 lines
tools/lib/bpf/gen_loader.c +3 -0
@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
 
 	for (i = 0; i < gen->core_relo_cnt; i++)
 		bpf_core_relo_bswap(cr++);
+
+	for (i = 0; i < load_attr->cleanup_info_cnt; i++)
+		bpf_cleanup_info_bswap(ci++);

[Severity: High]
Could this loop result in an out-of-bounds heap read and write?

If an attacker crafts a BPF ELF object with an extremely large .bpf_cleanup section, load_attr->cleanup_info_cnt can be very large. If the allocation size calculated in bpf_gen__prog_load() overflowed, this loop will process hundreds of millions of elements in an undersized buffer.

This could cause memory corruption when cross-compiling (gen->swapped_endian is true).

tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1020

Quoted 9 lines
tools/lib/bpf/gen_loader.c +2 -0
 }
 
 void bpf_gen__prog_load(struct bpf_gen *gen,
@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 			       load_attr->line_info_rec_size;
 	int core_relo_tot_sz = gen->core_relo_cnt *
 			       sizeof(struct bpf_core_relo);
+	int cleanup_info_tot_sz = load_attr->cleanup_info_cnt *
+				  load_attr->cleanup_info_rec_size;

[Severity: High]
Does this unbounded 32-bit integer multiplication risk an overflow?

If load_attr->cleanup_info_cnt is excessively large, multiplying it by cleanup_info_rec_size could wrap around modulo 2^32 to a small positive integer. This would result in add_data() allocating an undersized heap buffer for the cleanup info blob.

Combined with the loop in info_blob_bswap(), this could lead to crashes or potentially arbitrary code execution when processing untrusted files using the light skeleton generator.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
author reviewer
#41 lore raw

Re: [PATCH bpf-next v2 16/20] libbpf: Carry the exception cleanup table through the light skeleton

Quoted 29 lines · On 9/17/26 10:02 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] libbpf: integer overflow in cleanup_info_tot_sz leads to heap OOB read/write --

commit b9300a1f6022da2aa8b343e05882b46e9f329c64
Author: Yonghong Song <[email protected]>

libbpf: Carry the exception cleanup table through the light skeleton

This patch modifies the light skeleton generation to carry the exception cleanup table (cleanup_info), similar to func_info and line_info. It updates bpf_gen__prog_load() to allocate space for the records, set up union bpf_attr, and perform endianness swapping for cross-compiled cases.

Quoted 4 lines
tools/lib/bpf/gen_loader.c +0 -0
index af3a04f161ac1..2345fbdd46f56 100644

[ ... ]

tools/lib/bpf/gen_loader.c static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info, · line 1005

Quoted 7 lines
tools/lib/bpf/gen_loader.c +3 -0
@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
 
  	for (i = 0; i < gen->core_relo_cnt; i++)
  		bpf_core_relo_bswap(cr++);
+
+	for (i = 0; i < load_attr->cleanup_info_cnt; i++)
+		bpf_cleanup_info_bswap(ci++);

[Severity: High]
Could this loop result in an out-of-bounds heap read and write?

Quoted 7 lines

If an attacker crafts a BPF ELF object with an extremely large .bpf_cleanup section, load_attr->cleanup_info_cnt can be very large. If the allocation size calculated in bpf_gen__prog_load() overflowed, this loop will process hundreds of millions of elements in an undersized buffer.

This could cause memory corruption when cross-compiling (gen->swapped_endian is true).

It is user's problem if they intentionally want to have of lots and lots of .bpf_cleanup section.

Quoted 13 lines

tools/lib/bpf/gen_loader.c void bpf_gen__prog_load(struct bpf_gen *gen, · line 1020

Quoted 10 lines
tools/lib/bpf/gen_loader.c +2 -0
  }
 
  void bpf_gen__prog_load(struct bpf_gen *gen,
@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
  			       load_attr->line_info_rec_size;
  	int core_relo_tot_sz = gen->core_relo_cnt *
  			       sizeof(struct bpf_core_relo);
+	int cleanup_info_tot_sz = load_attr->cleanup_info_cnt *
+				  load_attr->cleanup_info_rec_size;

[Severity: High]
Does this unbounded 32-bit integer multiplication risk an overflow?

Quoted 8 lines

If load_attr->cleanup_info_cnt is excessively large, multiplying it by cleanup_info_rec_size could wrap around modulo 2^32 to a small positive integer. This would result in add_data() allocating an undersized heap buffer for the cleanup info blob.

Combined with the loop in info_blob_bswap(), this could lead to crashes or potentially arbitrary code execution when processing untrusted files using the light skeleton generator.

We should be okay. Similar to line_info, core_relo, etc., all of them (including cleanup_info) will have a guard in add_data which ensures the total size must be smaller than INT32_MAX.

Quoted 1 line
author reviewer
#42 lore raw

PATCH v2 17/20 libbpf: Let the static linker carry .bpf_cleanup relocations

In reply to Yonghong Song

An object that carries a compiler-emitted exception cleanup table cannot be linked today. The table's fields are byte offsets into a code section, materialised by a 32-bit relocation against that section's symbol with the offset itself as the implicit addend, and the linker rejects both halves of
that: the relocation type is not in the list it accepts, and a relocation against an STT_SECTION symbol from a non-executable section is an outright error.

Both spellings of that relocation have to be taken. LLVM emits R_BPF_64_NODYLD32 for a .long against a section symbol; GNU as emits R_BPF_64_ABS32, which is what bpf_reloc_type_lookup() maps BFD_RELOC_32 to. They describe the same value, and the selftests are built with both compilers.

---
 tools/lib/bpf/linker.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
tools/lib/bpf/linker.c +18 -1
index 78f92c39290a..e5c06023cb5b 100644
@@ -1036,7 +1036,8 @@ static int linker_sanity_check_elf_relos(struct src_obj *obj, struct src_sec *se
 		size_t sym_type = ELF64_R_TYPE(relo->r_info);
 
 		if (sym_type != R_BPF_64_64 && sym_type != R_BPF_64_32 &&
-		    sym_type != R_BPF_64_ABS64 && sym_type != R_BPF_64_ABS32) {
+		    sym_type != R_BPF_64_ABS64 && sym_type != R_BPF_64_ABS32 &&
+		    sym_type != R_BPF_64_NODYLD32) {
 			pr_warn("ELF relo #%d in section #%zu has unexpected type %zu in %s\n",
 				i, sec->sec_idx, sym_type, obj->filename);
 			return -EINVAL;
@@ -2274,6 +2275,22 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
 						insn->imm += sec->dst_off / sizeof(struct bpf_insn);
 					else
 						insn->imm += sec->dst_off;
+				} else if (sym_type == R_BPF_64_NODYLD32 ||
+					   sym_type == R_BPF_64_ABS32) {
+					__u32 *val;
+
+					/* Two spellings of the one thing: LLVM
+					 * emits NODYLD32 for a .long against a
+					 * section symbol, GNU as emits ABS32
+					 * (bpf_reloc_type_lookup() maps
+					 * BFD_RELOC_32 to it), and the value
+					 * they describe is the same.
+					 */
+					val = dst_linked_sec->raw_data + dst_rel->r_offset;
+					if (linker->swapped_endian)
+						*val = bswap_32(bswap_32(*val) + sec->dst_off);
+					else
+						*val += sec->dst_off;
 				} else {
 					pr_warn("relocation against STT_SECTION in non-exec section is not supported!\n");
 					return -EINVAL;
Signature · 1 line
2.53.0-Meta
#43 lore raw

Re: [PATCH bpf-next v2 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] bpf/linker: NULL pointer deref in linker_append_elf_relos() - [Medium] bpf/linker: unaligned…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] bpf/linker: NULL pointer deref in linker_append_elf_relos()
- [Medium] bpf/linker: unaligned memory access in data relocations
- [Low] bpf/linker: multi-line comment style violation
--

commit 2eadf13e135621b2e34d8946fe67664cd72e0e3d
Author: Yonghong Song <[email protected]>

libbpf: Let the static linker carry .bpf_cleanup relocations

This commit updates the BPF static linker to support .bpf_cleanup relocations. It specifically adds handling for R_BPF_64_NODYLD32 and R_BPF_64_ABS32 to process compiler-emitted exception cleanup tables.

Quoted 4 lines
tools/lib/bpf/linker.c +0 -0
index 78f92c39290af..e5c06023cb5b1 100644

[ ... ]

tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2282

Quoted 9 lines
tools/lib/bpf/linker.c +5 -0
@@ -2274,6 +2275,22 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
 						insn->imm += sec->dst_off / sizeof(struct bpf_insn);
 					else
 						insn->imm += sec->dst_off;
+				} else if (sym_type == R_BPF_64_NODYLD32 ||
+					   sym_type == R_BPF_64_ABS32) {
+					__u32 *val;
+
+					/* Two spellings of the one thing: LLVM

[Severity: Low]
This isn't a bug, but does this violate the BPF subsystem multi-line comment style? The preferred kernel style is to have the opening /* on its own line rather than putting text on the same line as the marker.

tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2289

Quoted 7 lines
tools/lib/bpf/linker.c +7 -0
+					 * emits NODYLD32 for a .long against a
+					 * section symbol, GNU as emits ABS32
+					 * (bpf_reloc_type_lookup() maps
+					 * BFD_RELOC_32 to it), and the value
+					 * they describe is the same.
+					 */
+					val = dst_linked_sec->raw_data + dst_rel->r_offset;

[Severity: High]
Could this lead to a NULL pointer dereference? Looking at linker_sanity_check_elf_relos(), SHT_NOBITS sections are explicitly allowed as relocatable sections.

For SHT_NOBITS sections (like .bss), extend_sec() typically skips allocating raw_data, leaving it as NULL. If we process a relocation against such a section here, we might compute val = NULL + dst_rel->r_offset and then dereference it on the next line when handling sec->dst_off.

[Severity: Medium]
Is it also possible to hit an unaligned memory access here?

While executable sections enforce alignment checks in linker_sanity_check_elf_relos(), there doesn't seem to be an equivalent alignment check for data section offsets. If an ELF provides an unaligned r_offset, the direct dereference of the unaligned val pointer might cause a crash on architectures with strict memory alignment requirements.

Quoted 7 lines
tools/lib/bpf/linker.c +4 -0
+					if (linker->swapped_endian)
+						*val = bswap_32(bswap_32(*val) + sec->dst_off);
+					else
+						*val += sec->dst_off;
 				} else {
 					pr_warn("relocation against STT_SECTION in non-exec section is not supported!\n");
 					return -EINVAL;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
author reviewer
#44 lore raw

Re: [PATCH bpf-next v2 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations

Quoted 52 lines · On 9/17/26 10:01 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] bpf/linker: NULL pointer deref in linker_append_elf_relos()
- [Medium] bpf/linker: unaligned memory access in data relocations
- [Low] bpf/linker: multi-line comment style violation
--

commit 2eadf13e135621b2e34d8946fe67664cd72e0e3d
Author: Yonghong Song <[email protected]>

libbpf: Let the static linker carry .bpf_cleanup relocations

This commit updates the BPF static linker to support .bpf_cleanup relocations. It specifically adds handling for R_BPF_64_NODYLD32 and R_BPF_64_ABS32 to process compiler-emitted exception cleanup tables.

Quoted 4 lines
tools/lib/bpf/linker.c +0 -0
index 78f92c39290af..e5c06023cb5b1 100644

[ ... ]

tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2282

Quoted 9 lines
tools/lib/bpf/linker.c +5 -0
@@ -2274,6 +2275,22 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
  						insn->imm += sec->dst_off / sizeof(struct bpf_insn);
  					else
  						insn->imm += sec->dst_off;
+				} else if (sym_type == R_BPF_64_NODYLD32 ||
+					   sym_type == R_BPF_64_ABS32) {
+					__u32 *val;
+
+					/* Two spellings of the one thing: LLVM

[Severity: Low]
This isn't a bug, but does this violate the BPF subsystem multi-line comment style? The preferred kernel style is to have the opening /* on its own line rather than putting text on the same line as the marker.

tools/lib/bpf/linker.c static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob · line 2289

Quoted 7 lines
tools/lib/bpf/linker.c +7 -0
+					 * emits NODYLD32 for a .long against a
+					 * section symbol, GNU as emits ABS32
+					 * (bpf_reloc_type_lookup() maps
+					 * BFD_RELOC_32 to it), and the value
+					 * they describe is the same.
+					 */
+					val = dst_linked_sec->raw_data + dst_rel->r_offset;

[Severity: High]
Could this lead to a NULL pointer dereference? Looking at linker_sanity_check_elf_relos(), SHT_NOBITS sections are explicitly allowed as relocatable sections.

For SHT_NOBITS sections (like .bss), extend_sec() typically skips allocating raw_data, leaving it as NULL. If we process a relocation against such a section here, we might compute val = NULL + dst_rel->r_offset and then dereference it on the next line when handling sec->dst_off.

Quoted 8 lines

[Severity: Medium]
Is it also possible to hit an unaligned memory access here?

While executable sections enforce alignment checks in linker_sanity_check_elf_relos(), there doesn't seem to be an equivalent alignment check for data section offsets. If an ELF provides an unaligned r_offset, the direct dereference of the unaligned val pointer might cause a crash on architectures with strict memory alignment requirements.

Okay, I will add a guard to check NULL pointer and unaligned memory access. If this is true, will return an error.

Quoted 8 lines
Quoted 8 lines
tools/lib/bpf/linker.c +4 -0
+					if (linker->swapped_endian)
+						*val = bswap_32(bswap_32(*val) + sec->dst_off);
+					else
+						*val += sec->dst_off;
  				} else {
  					pr_warn("relocation against STT_SECTION in non-exec section is not supported!\n");
  					return -EINVAL;
#45 lore raw

Re: [PATCH bpf-next v2 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations

In reply to Yonghong Song
This looks like a bug fix for objects that carry compiler-emitted exception cleanup tables. Should this include:
Quoted 8 lines

An object that carries a compiler-emitted exception cleanup table cannot be linked today. The table's fields are byte offsets into a code section, materialised by a 32-bit relocation against that section's symbol with the offset itself as the implicit addend, and the linker rejects both halves of
that: the relocation type is not in the list it accepts, and a relocation against an STT_SECTION symbol from a non-executable section is an outright error.

Quoted 7 lines

Both spellings of that relocation have to be taken. LLVM emits R_BPF_64_NODYLD32 for a .long against a section symbol; GNU as emits R_BPF_64_ABS32, which is what bpf_reloc_type_lookup() maps BFD_RELOC_32 to. They describe the same value, and the selftests are built with both compilers.

This looks like a bug fix for objects that carry compiler-emitted exception cleanup tables. Should this include:

Fixes 9f0c317f6aa1 ("libbpf: Add support for new llvm bpf relocations")

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#46 lore raw

Re: [PATCH bpf-next v2 17/20] libbpf: Let the static linker carry .bpf_cleanup relocations

Quoted 14 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 14 lines

An object that carries a compiler-emitted exception cleanup table cannot be linked today. The table's fields are byte offsets into a code section, materialised by a 32-bit relocation against that section's symbol with the offset itself as the implicit addend, and the linker rejects both halves of
that: the relocation type is not in the list it accepts, and a relocation against an STT_SECTION symbol from a non-executable section is an outright error.

Both spellings of that relocation have to be taken. LLVM emits R_BPF_64_NODYLD32 for a .long against a section symbol; GNU as emits R_BPF_64_ABS32, which is what bpf_reloc_type_lookup() maps BFD_RELOC_32 to. They describe the same value, and the selftests are built with both compilers.

Quoted 5 lines
Quoted 1 line

This looks like a bug fix for objects that carry compiler-emitted exception cleanup tables. Should this include:

Fixes 9f0c317f6aa1 ("libbpf: Add support for new llvm bpf relocations")

No need for a Fixes tag.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#47 lore raw

PATCH v2 18/20 selftests/bpf: Add an end-to-end .bpf_cleanup exception test

In reply to Yonghong Song

C has no unwinding, so nothing here comes out of the frontend: the frames that own a resource are written as __naked inline assembly, which spells out by hand exactly what a frontend emits -- a call site bracketed by two labels, a landing pad unreachable in the compiler's CFG, and a .bpf_cleanup record tying them together. The assembler turns ".long <text label>" into the same R_BPF_64_NODYLD32 relocation the BPF AsmPrinter emits, so libbpf and the kernel see an object indistinguishable from a compiler-generated one.

Call chain: entry -> foo1 -> foo1v -> foo2 -> foo3. foo3 holds a non-preemptible section and throws inside it; foo2 holds an RCU read lock and has two call sites sharing one pad, one of them its own throw; foo1v is a void frame whose pad ends in a jump to a resume block placed after an unrelated block that ends in a plain exit; foo1 owns nothing and gets no record; entry is the boundary.

There are also some shapes the kernel refuses -- the ones with no correct answer, and the ones a pad running on the bpf_throw() walker's stack cannot express -- plus the return-value rejection that delivering an exception at the boundary of a program type which constrains its return value produces. The test skips rather than fails where the JIT cannot dispatch a landing pad at all.

---
 .../selftests/bpf/exceptions_cleanup.h        |  29 +
 .../bpf/prog_tests/exceptions_cleanup.c       |  79 +++
 .../selftests/bpf/progs/exceptions_cleanup.c  | 152 +++++
 .../bpf/progs/exceptions_cleanup_fail.c       | 600 ++++++++++++++++++
 4 files changed, 860 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
tools/testing/selftests/bpf/exceptions_cleanup.h new file +29 -0
new file mode 100644
index 000000000000..630d2e207119
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef __EXCEPTIONS_CLEANUP_H__
+#define __EXCEPTIONS_CLEANUP_H__
+
+#define THROW_COOKIE		0x100
+
+/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */
+#define RAN_FOO3_PREEMPT	0x1
+#define RAN_FOO2_RCU		0x2
+#define RAN_FOO1V_PREEMPT	0x4
+#define RAN_FOO2_DROP		0x8
+#define RAN_BUMP		0x10
+
+#define CLEANUP_REC(begin, end, landing_pad)			\
+	".pushsection .bpf_cleanup,\"a\",@progbits;"		\
+	".long " begin ";"					\
+	".long " end ";"					\
+	".long " landing_pad ";"				\
+	".popsection;"
+
+/* Set a bit in @pads_ran. */
+#define PAD_RAN(bit)						\
+	"r1 = %[pads_ran] ll;"					\
+	"r2 = *(u64 *)(r1 + 0);"				\
+	"r2 |= " bit ";"					\
+	"*(u64 *)(r1 + 0) = r2;"
+
+#endif /* __EXCEPTIONS_CLEANUP_H__ */
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +79 -0

79 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup.c new file +152 -0

152 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +600 -0

600 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#48 lore raw

Re: [PATCH bpf-next v2 18/20] selftests/bpf: Add an end-to-end .bpf_cleanup exception test

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] bpf: multi-line comments do not follow modern BPF style --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: multi-line comments do not follow modern BPF style
--

commit 8ad5e7ea23b9dcb55029206d5cb36453b231f9c8
Author: Yonghong Song <[email protected]>

selftests/bpf: Add an end-to-end .bpf_cleanup exception test

This commit adds an end-to-end test for .bpf_cleanup exception handling. It uses inline assembly to construct precise call-site brackets, landing pads, and cleanup records that mirror compiler output. It also tests shapes that the kernel should refuse, ensuring the JIT correctly handles or rejects different landing pad configurations.

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +0 -0
new file mode 100644
index 0000000000000..d1e45b765af26

[ ... ]

Quoted 1 line
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +0 -0
@@ -0,0 +1,79 @@

[ ... ]

tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c line 72

Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +7 -0
+	/* foo3 throws; every pad runs and the cookie is delivered at entry. */
+	if (test__start_subtest("throw_from_foo3"))
+		run(skel, 101, THROW_COOKIE, PADS_FOO3_THREW);
+
+	/* foo3 returns 2 ^ 1 == 3, so foo2 throws from its own second region;
+	 * foo3's frame is long gone, so its pad must not run.
+	 */

[Severity: Low]
Does this comment block follow the proper multi-line comment formatting for the BPF subsystem?

The BPF subsystem strict guidelines require multi-line comments to have the opening /* on its own line:

/*
* foo3 returns 2 ^ 1 == 3, so foo2 throws from its own second region;
* foo3's frame is long gone, so its pad must not run.
*/

Quoted 13 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +7 -0
+	if (test__start_subtest("throw_from_foo2"))
+		run(skel, 2, THROW_COOKIE, PADS_FOO2_THREW);
+
+	exceptions_cleanup__destroy(skel);
+
+	RUN_TESTS(exceptions_cleanup_fail);
+}
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +0 -0
new file mode 100644
index 0000000000000..ce2dac306a84c

[ ... ]

Quoted 1 line
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +0 -0
@@ -0,0 +1,600 @@

[ ... ]

Quoted 16 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +16 -0
+SEC("socket")
+__failure __msg("reaches an indirect jump")
+__naked void gotox_in_pad(void)
+{
+	asm volatile (
+	".pushsection .jumptables,\"\",@progbits;"
+"jt0_%=:"
+	".quad l0_%= - socket;"
+	".quad l1_%= - socket;"
+	".size jt0_%=, 16;"
+	".global jt0_%=;"
+	".popsection;"
+
+"1:"	"call gotox_thrower;"		/* cleanup region */
+"2:"
+	"r0 = 0;"

tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c line 586

Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +10 -0
+	"exit;"
+"3:"					/* landing pad */
+	"r1 = jt0_%= ll;"
+	"r1 += 8;"
+	"r2 = *(u64 *)(r1 + 0);"
+	/* gotox r2. Spelled as a raw insn on purpose: the "gotox" mnemonic
+	 * only reached the LLVM assembler in llvm 22, and BPF_RAW_INSN()
+	 * needs <linux/bpf.h>, which this file cannot have -- vmlinux.h
+	 * already defines the uapi enums.
+	 */

[Severity: Low]
Does this multi-line comment format conform to the BPF style guide?

As with the previous comment in exceptions_cleanup.c, the BPF subsystem mandates that the opening /* resides on its own line for multi-line comments.

Quoted 14 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +14 -0
+	".8byte 0x20d;"
+"l0_%=:"
+	"call bpf_unwind_resume;"
+	"exit;"
+"l1_%=:"
+	"call bpf_unwind_resume;"
+	"exit;"
+	CLEANUP_REC("1b", "2b", "3b")
+	::: __clobber_all);
+}
+
+#endif /* x86 || arm64 */
+
+char _license[] SEC("license") = "GPL";
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
#49 lore raw

Re: [PATCH bpf-next v2 18/20] selftests/bpf: Add an end-to-end .bpf_cleanup exception test

In reply to Yonghong Song
Is bpf_unwind_resume() declared anywhere? It is called from C here but bpf_experimental.h declares only the other kfuncs this anchor exists for:
Quoted 14 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +8 -0
new file mode 100644
index 0000000000000..ce2dac306a84c
@@ -0,0 +1,600 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_experimental.h"
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+#include "exceptions_cleanup.h"

tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c line 18

Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +10 -0
+
+__u64 input = 0;
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+	bpf_throw(0);
+	bpf_preempt_disable();
+	bpf_preempt_enable();
+	bpf_unwind_resume();
+}

Is bpf_unwind_resume() declared anywhere? It is called from C here but bpf_experimental.h declares only the other kfuncs this anchor exists for:

tools/testing/selftests/bpf/bpf_experimental.h declares bpf_throw():

    extern void bpf_throw(u64 cookie) __ksym;

and the preempt pair:

    void bpf_preempt_disable(void) __weak __ksym;
    void bpf_preempt_enable(void) __weak __ksym;

but bpf_unwind_resume() does not appear in any header. vmlinux.h contains type definitions only, not kfunc prototypes.

BPF objects are built with BPF_CFLAGS = -g -Wall -Werror -std=gnu11 (tools/testing/selftests/bpf/Makefile). -Wimplicit-function-declaration is part of -Wall, and in C99+ mode modern Clang promotes it to an error, which with -Werror becomes fatal:

progs/exceptions_cleanup_fail.c:17:2: error: call to undeclared
function 'bpf_unwind_resume'; ISO C99 and later do not support implicit function declarations

This would break the build of the whole BPF selftests tree. The same C-level call appears in the three other new prog files added by this series (progs/exceptions_cleanup.c, and in follow-ups progs/exceptions_cleanup_shapes.c and progs/exceptions_cleanup_light.c), so all of them would fail to compile.

The anchor cannot simply drop the call either: the C call is what makes Clang emit the BTF extern FUNC for bpf_unwind_resume, which libbpf needs to resolve the inline-asm "call bpf_unwind_resume;" relocations. The fix would be to add the declaration:

    extern void bpf_unwind_resume(void) __ksym;

in bpf_experimental.h alongside bpf_throw(). For comparison, progs/exceptions_cleanup_shapes.c (commit 198e7e8d6d72) does declare its sibling explicitly:

    extern void _Unwind_Resume(void) __ksym;

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#50 lore raw

Re: [PATCH bpf-next v2 18/20] selftests/bpf: Add an end-to-end .bpf_cleanup exception test

Quoted 58 lines · On 9/17/26 10:58 PM, [email protected] wrote:
Quoted 14 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +8 -0
new file mode 100644
index 0000000000000..ce2dac306a84c
@@ -0,0 +1,600 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_experimental.h"
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+#include "exceptions_cleanup.h"

tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c line 18

Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +10 -0
+
+__u64 input = 0;
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+	bpf_throw(0);
+	bpf_preempt_disable();
+	bpf_preempt_enable();
+	bpf_unwind_resume();
+}

Is bpf_unwind_resume() declared anywhere? It is called from C here but bpf_experimental.h declares only the other kfuncs this anchor exists for:

tools/testing/selftests/bpf/bpf_experimental.h declares bpf_throw():

     extern void bpf_throw(u64 cookie) __ksym;

and the preempt pair:

     void bpf_preempt_disable(void) __weak __ksym;
     void bpf_preempt_enable(void) __weak __ksym;

but bpf_unwind_resume() does not appear in any header. vmlinux.h contains type definitions only, not kfunc prototypes.

BPF objects are built with BPF_CFLAGS = -g -Wall -Werror -std=gnu11 (tools/testing/selftests/bpf/Makefile). -Wimplicit-function-declaration is part of -Wall, and in C99+ mode modern Clang promotes it to an error, which with -Werror becomes fatal:

progs/exceptions_cleanup_fail.c:17:2: error: call to undeclared
function 'bpf_unwind_resume'; ISO C99 and later do not support implicit function declarations

This would break the build of the whole BPF selftests tree. The same C-level call appears in the three other new prog files added by this series (progs/exceptions_cleanup.c, and in follow-ups progs/exceptions_cleanup_shapes.c and progs/exceptions_cleanup_light.c), so all of them would fail to compile.

The anchor cannot simply drop the call either: the C call is what makes Clang emit the BTF extern FUNC for bpf_unwind_resume, which libbpf needs to resolve the inline-asm "call bpf_unwind_resume;" relocations. The fix would be to add the declaration:

Quoted 7 lines
     extern void bpf_unwind_resume(void) __ksym;

in bpf_experimental.h alongside bpf_throw(). For comparison, progs/exceptions_cleanup_shapes.c (commit 198e7e8d6d72) does declare its sibling explicitly:

     extern void _Unwind_Resume(void) __ksym;

I didn't find anything wrong with this patch set and with my normal environment, the test works fine.

Quoted 6 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#51 lore raw

PATCH v2 19/20 selftests/bpf: Cover the exception cleanup shapes the chain does not reach

In reply to Yonghong Song

The end-to-end test walks one call chain with a pad in most of its frames. This adds the shapes that chain does not reach: a callee called from both a covered and an uncovered site, a pad that reads its frame's callee-saved registers, a tail-call-reachable callee, a tail call that is taken, an extension standing in for a covered call, a pad in the main program's own frame, a record covering bpf_throw() itself, a record covering only a nounwind call, a throwing subprogram named by a BPF_PSEUDO_FUNC, a region ending on a 16-byte instruction, a pad that reloads from and writes to its own frame, a pad two frames up, a pad that calls a subprogram which tail calls, an extension carrying a table of its own, and a pad terminated by _Unwind_Resume rather than bpf_unwind_resume.

---
 .../selftests/bpf/exceptions_cleanup.h        |  20 +
 .../bpf/prog_tests/exceptions_cleanup.c       | 314 +++++++
 .../bpf/progs/exceptions_cleanup_ext_table.c  |  48 +
 .../bpf/progs/exceptions_cleanup_freplace.c   |  17 +
 .../progs/exceptions_cleanup_pad_freplace.c   |  17 +
 .../bpf/progs/exceptions_cleanup_shapes.c     | 863 ++++++++++++++++++
 6 files changed, 1279 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
tools/testing/selftests/bpf/exceptions_cleanup.h +20 -0
index 630d2e207119..5896cf83d15e 100644
@@ -4,6 +4,7 @@
 #define __EXCEPTIONS_CLEANUP_H__
 
 #define THROW_COOKIE		0x100
+#define INNER_COOKIE		0x200
 
 /* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */
 #define RAN_FOO3_PREEMPT	0x1
@@ -12,6 +13,25 @@
 #define RAN_FOO2_DROP		0x8
 #define RAN_BUMP		0x10
 
+/* progs/exceptions_cleanup_shapes.c: one bit per shape, numbered its own way. */
+#define RAN_SWEEP		0x1
+#define RAN_SHARED		0x2
+#define RAN_REGS		0x4
+#define RAN_TAIL_CALL		0x8
+#define RAN_MAIN_PAD		0x10
+#define RAN_TC_TAKEN		0x20
+#define RAN_FREPLACE		0x40
+#define RAN_ADDR_TAKEN		0x80
+#define RAN_NO_SUBPROG		0x100
+#define RAN_PAD_CALLS		0x200
+#define RAN_PAD_FIRST		0x400
+#define RAN_WIDE_REC		0x800
+#define RAN_PAD_STACK		0x1000
+#define RAN_DEEP_PAD		0x2000
+#define RAN_NOUNWIND_REC	0x4000
+#define RAN_RESUME_ALIAS	0x8000
+#define RAN_PAD_TAIL_CALL	0x10000
+
 #define CLEANUP_REC(begin, end, landing_pad)			\
 	".pushsection .bpf_cleanup,\"a\",@progbits;"		\
 	".long " begin ";"					\
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +314 -0

331 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c new file +48 -0

48 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c new file +17 -0
new file mode 100644
index 000000000000..afb358fd3d40
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "exceptions_cleanup.h"
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */
+int btf_int_anchor;
+
+SEC("freplace/fr_callee")
+__u64 new_fr_callee(__u64 x)
+{
+	bpf_throw(THROW_COOKIE);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c new file +17 -0
new file mode 100644
index 000000000000..eabac6baabb7
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "exceptions_cleanup.h"
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */
+int btf_int_anchor;
+
+SEC("freplace/pad_callee")
+__u64 new_pad_callee(__u64 x)
+{
+	bpf_throw(INNER_COOKIE);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +863 -0

863 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#52 lore raw

Re: [PATCH bpf-next v2 19/20] selftests/bpf: Cover the exception cleanup shapes the chain does not reach

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] bpf: multi-line comment style violation --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] bpf: multi-line comment style violation
--

commit 59a07b8ba8f5c3976a5da9bea3b5d6359874813a
Author: Yonghong Song <[email protected]>

selftests/bpf: Cover the exception cleanup shapes the chain does not reach

This commit adds end-to-end exception cleanup tests for various edge cases and shapes that the primary call chain does not reach. It tests things like tail calls, nounwind calls, subprogram pads, and landing pads that interact with the stack.

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +0 -0
index d1e45b765af26..ffc0b95191681 100644

[ ... ]

Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +2 -0
@@ -35,6 +39,314 @@ static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval,
+static void test_shapes(void)
+{

[ ... ]

tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 215

Quoted 3 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +3 -0
+	/* The same call site either way: the subprogram's throw unwinds into
+	 * this frame and runs its pad, an extension's stops at its own boundary.
+	 */

[Severity: Low]
This isn't a bug, but does this comment follow the BPF subsystem coding style? Multi-line comments should have their opening /* on its own line before the text begins.

tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 224

Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +9 -0
+	if (test__start_subtest("freplace_subprog_throws"))
+		run_shape(skel, skel->progs.entry_freplace, 7, THROW_COOKIE,
+			  RAN_FREPLACE);
+	if (test__start_subtest("freplace_extension_throws"))
+		test_freplace(skel);
+
+	/* A tail call that is taken: the walk ends at the target, so the cookie
+	 * comes back from there and this frame's pad does not run.
+	 */

[Severity: Low]
This isn't a bug, but is this the preferred multi-line comment format? The BPF subsystem coding style mandates that the opening /* should be on its own line.

Quoted 2 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +2 -0
+	if (test__start_subtest("tail_call_taken")) {
+		int key = 0, prog_fd = bpf_program__fd(skel->progs.tc_target);

[ ... ]

Quoted 5 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +0 -0
new file mode 100644
index 0000000000000..f5eb2ff15c896

[ ... ]

tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 5

Quoted 6 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +5 -0
@@ -0,0 +1,863 @@
+static __used __noinline __u64 tc_thrower(void *ctx)
+{
+	/* Never taken; its presence is what makes this frame, whose spill the
+	 * walker reads, tail-call-reachable.
+	 */

[Severity: Low]
This isn't a bug, but could this comment formatting be adjusted? Multi-line comments in the BPF subsystem should place their opening /* on a separate line.

Quoted 1 line
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +1 -0
+	bpf_tail_call_static(ctx, &jmp_table, 0);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19
#53 lore raw

Re: [PATCH bpf-next v2 19/20] selftests/bpf: Cover the exception cleanup shapes the chain does not reach

In reply to Yonghong Song
Does this comment accurately describe what the addr_taken shape tests?
Quoted 12 lines
tools/testing/selftests/bpf/exceptions_cleanup.h +1 -0
index 630d2e207119a..5896cf83d15ee 100644
@@ -4,6 +4,7 @@
 #define __EXCEPTIONS_CLEANUP_H__
 
 #define THROW_COOKIE		0x100
+#define INNER_COOKIE		0x200
 
 /* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */
 #define RAN_FOO3_PREEMPT	0x1

[ ... ]

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +0 -0
index d1e45b765af26..ffc0b95191681 100644

[ ... ]

Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +7 -0
+static void test_shapes(void)
+{
+	struct exceptions_cleanup_shapes *skel;
+
+	skel = exceptions_cleanup_shapes__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "shapes open_and_load"))
+		return;

[ ... ]

tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 242

Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +8 -0
+	/* A throwing subprog named by a BPF_PSEUDO_FUNC no helper is handed: the
+	 * callback check has to look at the bpf_loop(), not at the ld_imm64.
+	 */
+	if (test__start_subtest("addr_taken_no_throw"))
+		run_shape(skel, skel->progs.entry_addr_taken, 1, 2, 0);
+	if (test__start_subtest("addr_taken_throw"))
+		run_shape(skel, skel->progs.entry_addr_taken, 101, THROW_COOKIE,
+			  RAN_ADDR_TAKEN);

Does this comment accurately describe what the addr_taken shape tests?

Looking at addr_taken_callee() in progs/exceptions_cleanup_shapes.c, the function does hand cb_thrower to bpf_loop(). The ld_imm64 and the bpf_loop() call are on the same path, after bpf_throw(). Since bpf_throw() is not declared noreturn, LLVM keeps that tail, and what actually makes the program load is that the verifier never reaches the bpf_loop() at all (it is dead after the throw).

So push_callback_call() -> bpf_cleanup_check_callback() never fires.

The shape exercised is 'a BPF_PSEUDO_FUNC whose helper call the verifier never reaches', not 'a BPF_PSEUDO_FUNC no helper is handed'. Could the comment be rephrased to reflect what the verifier actually sees?

[ ... ]

Quoted 5 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +0 -0
new file mode 100644
index 0000000000000..f5eb2ff15c896

[ ... ]

Quoted 29 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +29 -0
+/*
+ * 2. A callee called from both a covered and an uncovered site: the pad is
+ * recorded on the call site, not on the callee. The lock sits between the two
+ * calls because the frame really would leak it if the uncovered call unwound.
+ */
+static __used __noinline __u64 shared_callee(__u64 x)
+{
+	if (x > 100)
+		bpf_throw(THROW_COOKIE);
+	return x + 1;
+}
+
+static __used __naked __noinline __u64 shared_frame(void)
+{
+	asm volatile (
+	"r1 = %[input] ll;"
+	"r6 = *(u64 *)(r1 + 0);"
+	"r1 = 0;"
+	"call shared_callee;"
+	"call bpf_rcu_read_lock;"
+	"r1 = r6;"
+"1:"	"call shared_callee;"		/* cleanup region */
+"2:"
+	"r6 = r0;"
+	"call bpf_rcu_read_unlock;"
+	"r0 = r6;"
+	"exit;"
+"3:"					/* landing pad */
+	"r7 = r0;"

tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 118

Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +10 -0
+	"call bpf_rcu_read_unlock;"
+	PAD_RAN("%[ran]")
+	"r1 = r7;"
+	"call bpf_unwind_resume;"
+	"exit;"
+	CLEANUP_REC("1b", "2b", "3b")
+	:
+	: [ran]"i"(RAN_SHARED), __imm_addr(input), __imm_addr(pads_ran)
+	: __clobber_all);
+}

Does the uncovered call site verify that the pad wouldn't be wrongly dispatched if an exception occurred there?

The uncovered site is reached with the constant 0 in r1, and shared_callee only throws when x > 100, so the verifier proves that call cannot throw and no unwind ever leaves the uncovered site - at verification time or at run time.

The structural half of the property (only the second call's return address falls inside the record's native range) is exercised, but the behavioral half is not: a kernel that wrongly matched the uncovered site's return address against the record and ran the pad would go unnoticed, because that site never unwinds.

[ ... ]

Quoted 44 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +44 -0
+/*
+ * 6. A tail call that is really taken: the target is a program in its own
+ * right, so the walk ends there and this frame's pad does not run. The callee
+ * can also throw on a path never taken, which keeps the pad out of the sweep.
+ */
+struct {
+	__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+	__uint(max_entries, 1);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(__u32));
+} taken_table SEC(".maps");
+
+SEC("syscall")
+int tc_target(void *ctx)
+{
+	bpf_throw(THROW_COOKIE);
+	return 0;
+}
+
+static __used __noinline __u64 tc_taken_callee(void *ctx, __u64 x)
+{
+	/* Never true at run time; the verifier cannot know that, and its
+	 * unwind out of here is what keeps the caller's pad alive.
+	 */
+	if (x == 7)
+		bpf_throw(THROW_COOKIE);
+	bpf_tail_call_static(ctx, &taken_table, 0);
+	return 0;
+}
+
+SEC("syscall")
+__naked int entry_tail_taken(void)
+{
+	asm volatile (
+	"*(u64 *)(r10 - 8) = r1;"	/* the context, straight from entry */
+	"r1 = %[input] ll;"
+	"r2 = *(u64 *)(r1 + 0);"
+	"if r2 < 101 goto 8f;"
+	"r1 = *(u64 *)(r10 - 8);"
+"1:"	"call tc_taken_callee;"		/* cleanup region */
+"2:"
+	"exit;"				/* the cookie, delivered at tc_target */
+"8:"
+	"r0 = 0;"

tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 344

Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +10 -0
+	"exit;"
+"3:"					/* landing pad: must not run */
+	PAD_RAN("%[ran]")
+	"call bpf_unwind_resume;"
+	"exit;"
+	CLEANUP_REC("1b", "2b", "3b")
+	:
+	: [ran]"i"(RAN_TC_TAKEN), __imm_addr(input), __imm_addr(pads_ran)
+	: __clobber_all);
+}

Can the tail_call_taken subtest fail if the kernel wrongly continues the walk past the tail-call boundary?

entry_tail_taken narrows the argument before the covered call:

"r2 = *(u64 *)(r1 + 0);" /* r2 = input */
"if r2 < 101 goto 8f;" /* fallthrough => r2 in [101, U64_MAX] */

tc_taken_callee is a static subprog, so check_func_call() copies the caller's r1-r5 verbatim into the callee frame. x therefore arrives with umin_value == 101, and is_branch_taken() resolves 'if (x == 7)' to never-taken, so the verifier never explores the bpf_throw(). Nothing then unwinds into entry_tail_taken during verification, the landing pad at label 3 is never marked seen, and opt_remove_dead_code() removes it.

fixups.c then clears the annotation for any call site whose cleanup_pad falls inside the removed window:

    if (env->cleanup_info_cnt) {
            for (i = 0; i < env->insn_aux_data_len; i++) {
                    u32 pad = aux_data[i].cleanup_pad;
                    if (pad > off + cnt) aux_data[i].cleanup_pad = pad - cnt;
                    else if (pad > off)  aux_data[i].cleanup_pad = 0;
            }
    }

so no native range is emitted for the region at all. The program still loads (mark_subprog_might_throw() is a static analysis, so bpf_check_cfg() still sees the pad edge and check_cleanup_info() runs before the sweep), and pads_ran stays 0 unconditionally. A kernel that wrongly continued bpf_stack_walker() past the tail-call boundary into entry_tail_taken's frame would find no record to match and would still leave pads_ran == 0, so the subtest passes on a correct and on a broken kernel alike.

The in-code comment is also wrong on both clauses: the verifier CAN know x != 7 here, and consequently the unwind out of tc_taken_callee is NOT what keeps the caller's pad alive - nothing does.

Contrast with shape 7 (entry_freplace, lines 862-878), the analogous 'walk ends in the callee's frame' shape, which deliberately passes the unnarrowed input value to fr_callee() with no 'if r1 < 101' guard, keeping fr_callee's identical 'if (x == 7) bpf_throw()' live. Dropping the 'if r2 < 101 goto 8f' guard from entry_tail_taken (or otherwise passing a value the verifier cannot exclude 7 from) would make the pad survive and give the subtest something to fail on.

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#54 lore raw

Re: [PATCH bpf-next v2 19/20] selftests/bpf: Cover the exception cleanup shapes the chain does not reach

Quoted 46 lines · On 9/17/26 10:44 PM, [email protected] wrote:
Quoted 12 lines
tools/testing/selftests/bpf/exceptions_cleanup.h +1 -0
index 630d2e207119a..5896cf83d15ee 100644
@@ -4,6 +4,7 @@
  #define __EXCEPTIONS_CLEANUP_H__
 
  #define THROW_COOKIE		0x100
+#define INNER_COOKIE		0x200
 
  /* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */
  #define RAN_FOO3_PREEMPT	0x1

[ ... ]

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +0 -0
index d1e45b765af26..ffc0b95191681 100644

[ ... ]

Quoted 7 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +7 -0
+static void test_shapes(void)
+{
+	struct exceptions_cleanup_shapes *skel;
+
+	skel = exceptions_cleanup_shapes__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "shapes open_and_load"))
+		return;

[ ... ]

tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval, · line 242

Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +8 -0
+	/* A throwing subprog named by a BPF_PSEUDO_FUNC no helper is handed: the
+	 * callback check has to look at the bpf_loop(), not at the ld_imm64.
+	 */
+	if (test__start_subtest("addr_taken_no_throw"))
+		run_shape(skel, skel->progs.entry_addr_taken, 1, 2, 0);
+	if (test__start_subtest("addr_taken_throw"))
+		run_shape(skel, skel->progs.entry_addr_taken, 101, THROW_COOKIE,
+			  RAN_ADDR_TAKEN);

Does this comment accurately describe what the addr_taken shape tests?

Looking at addr_taken_callee() in progs/exceptions_cleanup_shapes.c, the function does hand cb_thrower to bpf_loop(). The ld_imm64 and the bpf_loop() call are on the same path, after bpf_throw(). Since bpf_throw() is not declared noreturn, LLVM keeps that tail, and what actually makes the program load is that the verifier never reaches the bpf_loop() at all (it is dead after the throw).

Quoted 5 lines

So push_callback_call() -> bpf_cleanup_check_callback() never fires.

The shape exercised is 'a BPF_PSEUDO_FUNC whose helper call the verifier never reaches', not 'a BPF_PSEUDO_FUNC no helper is handed'. Could the comment be rephrased to reflect what the verifier actually sees?

Okay, will change comments.

Quoted 57 lines

[ ... ]

Quoted 5 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +0 -0
new file mode 100644
index 0000000000000..f5eb2ff15c896

[ ... ]

Quoted 29 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +29 -0
+/*
+ * 2. A callee called from both a covered and an uncovered site: the pad is
+ * recorded on the call site, not on the callee. The lock sits between the two
+ * calls because the frame really would leak it if the uncovered call unwound.
+ */
+static __used __noinline __u64 shared_callee(__u64 x)
+{
+	if (x > 100)
+		bpf_throw(THROW_COOKIE);
+	return x + 1;
+}
+
+static __used __naked __noinline __u64 shared_frame(void)
+{
+	asm volatile (
+	"r1 = %[input] ll;"
+	"r6 = *(u64 *)(r1 + 0);"
+	"r1 = 0;"
+	"call shared_callee;"
+	"call bpf_rcu_read_lock;"
+	"r1 = r6;"
+"1:"	"call shared_callee;"		/* cleanup region */
+"2:"
+	"r6 = r0;"
+	"call bpf_rcu_read_unlock;"
+	"r0 = r6;"
+	"exit;"
+"3:"					/* landing pad */
+	"r7 = r0;"

tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 118

Quoted 10 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +10 -0
+	"call bpf_rcu_read_unlock;"
+	PAD_RAN("%[ran]")
+	"r1 = r7;"
+	"call bpf_unwind_resume;"
+	"exit;"
+	CLEANUP_REC("1b", "2b", "3b")
+	:
+	: [ran]"i"(RAN_SHARED), __imm_addr(input), __imm_addr(pads_ran)
+	: __clobber_all);
+}

Does the uncovered call site verify that the pad wouldn't be wrongly dispatched if an exception occurred there?

The uncovered site is reached with the constant 0 in r1, and shared_callee only throws when x > 100, so the verifier proves that call cannot throw and no unwind ever leaves the uncovered site - at verification time or at run time.

Quoted 5 lines

The structural half of the property (only the second call's return address falls inside the record's native range) is exercised, but the behavioral half is not: a kernel that wrongly matched the uncovered site's return address against the record and ran the pad would go unnoticed, because that site never unwinds.

The above analysis exactly described the code so the above 'wrongly dispatched if an exception occurred there' does not happen. I guess this tries to capture incorrect kernel implementation.

Quoted 51 lines

[ ... ]

Quoted 48 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +48 -0
+/*
+ * 6. A tail call that is really taken: the target is a program in its own
+ * right, so the walk ends there and this frame's pad does not run. The callee
+ * can also throw on a path never taken, which keeps the pad out of the sweep.
+ */
+struct {
+	__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+	__uint(max_entries, 1);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(__u32));
+} taken_table SEC(".maps");
+
+SEC("syscall")
+int tc_target(void *ctx)
+{
+	bpf_throw(THROW_COOKIE);
+	return 0;
+}
+
+static __used __noinline __u64 tc_taken_callee(void *ctx, __u64 x)
+{
+	/* Never true at run time; the verifier cannot know that, and its
+	 * unwind out of here is what keeps the caller's pad alive.
+	 */
+	if (x == 7)
+		bpf_throw(THROW_COOKIE);
+	bpf_tail_call_static(ctx, &taken_table, 0);
+	return 0;
+}
+
+SEC("syscall")
+__naked int entry_tail_taken(void)
+{
+	asm volatile (
+	"*(u64 *)(r10 - 8) = r1;"	/* the context, straight from entry */
+	"r1 = %[input] ll;"
+	"r2 = *(u64 *)(r1 + 0);"
+	"if r2 < 101 goto 8f;"
+	"r1 = *(u64 *)(r10 - 8);"
+"1:"	"call tc_taken_callee;"		/* cleanup region */
+"2:"
+	"exit;"				/* the cookie, delivered at tc_target */
+"8:"
+	"r0 = 0;"
+	"exit;"
+"3:"					/* landing pad: must not run */
+	PAD_RAN("%[ran]")
+	"call bpf_unwind_resume;"
Quoted 8 lines

tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c line 344

Quoted 6 lines
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +6 -0
+	"exit;"
+	CLEANUP_REC("1b", "2b", "3b")
+	:
+	: [ran]"i"(RAN_TC_TAKEN), __imm_addr(input), __imm_addr(pads_ran)
+	: __clobber_all);
+}

Can the tail_call_taken subtest fail if the kernel wrongly continues the walk past the tail-call boundary?

This is a test which fixed some early regression tests. Yes, if kernel implementation is wrong, tail_call_taken test may fail.

Quoted 32 lines

entry_tail_taken narrows the argument before the covered call:

   "r2 = *(u64 *)(r1 + 0);"       /* r2 = input */
   "if r2 < 101 goto 8f;"         /* fallthrough => r2 in [101, U64_MAX] */

tc_taken_callee is a static subprog, so check_func_call() copies the caller's r1-r5 verbatim into the callee frame. x therefore arrives with umin_value == 101, and is_branch_taken() resolves 'if (x == 7)' to never-taken, so the verifier never explores the bpf_throw(). Nothing then unwinds into entry_tail_taken during verification, the landing pad at label 3 is never marked seen, and opt_remove_dead_code() removes it.

fixups.c then clears the annotation for any call site whose cleanup_pad falls inside the removed window:

     if (env->cleanup_info_cnt) {
             for (i = 0; i < env->insn_aux_data_len; i++) {
                     u32 pad = aux_data[i].cleanup_pad;
                     if (pad > off + cnt) aux_data[i].cleanup_pad = pad - cnt;
                     else if (pad > off)  aux_data[i].cleanup_pad = 0;
             }
     }

so no native range is emitted for the region at all. The program still loads (mark_subprog_might_throw() is a static analysis, so bpf_check_cfg() still sees the pad edge and check_cleanup_info() runs before the sweep), and pads_ran stays 0 unconditionally. A kernel that wrongly continued bpf_stack_walker() past the tail-call boundary into entry_tail_taken's frame would find no record to match and would still leave pads_ran == 0, so the subtest passes on a correct and on a broken kernel alike.

Quoted 3 lines

The in-code comment is also wrong on both clauses: the verifier CAN know x != 7 here, and consequently the unwind out of tc_taken_callee is NOT what keeps the caller's pad alive - nothing does.

The comment is:

        /* Never true at run time; the verifier cannot know that, and its
         * unwind out of here is what keeps the caller's pad alive.
         */
        if (x == 7)
                bpf_throw(THROW_COOKIE);

Yes, the 'x' will be a known value at run time. Comments need update.

Quoted 17 lines

Contrast with shape 7 (entry_freplace, lines 862-878), the analogous 'walk ends in the callee's frame' shape, which deliberately passes the unnarrowed input value to fr_callee() with no 'if r1 < 101' guard, keeping fr_callee's identical 'if (x == 7) bpf_throw()' live. Dropping the 'if r2 < 101 goto 8f' guard from entry_tail_taken (or otherwise passing a value the verifier cannot exclude 7 from) would make the pad survive and give the subtest something to fail on.

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35308528711

author reviewer
#55 lore raw

PATCH v2 20/20 selftests/bpf: Load an exception cleanup program from a light skeleton

In reply to Yonghong Song

For a light skeleton, libbpf hands the records to bpf_gen__prog_load(), which writes them into a blob and emits a loader program that issues BPF_PROG_LOAD from inside the kernel. Nothing about that is shared with the ordinary path: the attr is built field by field, and the kfunc names are resolved by the loader program when it runs.

exceptions_cleanup_light.c is the smallest program that can tell whether a table survives that trip: one record covering the bpf_throw() call itself, one pad that sets a bit and resumes. If the table arrives, the pad runs and the cookie comes back; if it does not, the program does not load at all.

---
 tools/testing/selftests/bpf/Makefile          |  2 +-
 .../selftests/bpf/exceptions_cleanup.h        |  3 ++
 .../bpf/prog_tests/exceptions_cleanup.c       | 28 +++++++++++++
 .../bpf/progs/exceptions_cleanup_light.c      | 39 +++++++++++++++++++
 4 files changed, 71 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c
tools/testing/selftests/bpf/Makefile +1 -1
index 7ea5ba1df29e..981cc2643049 100644
@@ -523,7 +523,7 @@ LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h		\
 LSKELS := fexit_sleep.c trace_printk.c trace_vprintk.c map_ptr_kern.c 	\
 	core_kern.c core_kern_overflow.c test_ringbuf.c			\
 	test_ringbuf_n.c test_ringbuf_map_key.c test_ringbuf_write.c    \
-	test_ringbuf_overwrite.c
+	test_ringbuf_overwrite.c exceptions_cleanup_light.c
 
 LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
 
tools/testing/selftests/bpf/exceptions_cleanup.h +3 -0
index 5896cf83d15e..0c088d96ca01 100644
@@ -32,6 +32,9 @@
 #define RAN_RESUME_ALIAS	0x8000
 #define RAN_PAD_TAIL_CALL	0x10000
 
+/* progs/exceptions_cleanup_light.c: the one pad it has. */
+#define RAN_LIGHT		0x1
+
 #define CLEANUP_REC(begin, end, landing_pad)			\
 	".pushsection .bpf_cleanup,\"a\",@progbits;"		\
 	".long " begin ";"					\
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +28 -0
index ffc0b9519168..ad0ff949f10d 100644
@@ -8,6 +8,7 @@
 #include "exceptions_cleanup_freplace.skel.h"
 #include "exceptions_cleanup_pad_freplace.skel.h"
 #include "exceptions_cleanup_ext_table.skel.h"
+#include "exceptions_cleanup_light.lskel.h"
 
 /* foo3 threw: every frame that has a pad ran it. */
 #define PADS_FOO3_THREW \
@@ -170,6 +171,30 @@ static void test_ext_table(struct exceptions_cleanup_shapes *skel)
 	exceptions_cleanup_ext_table__destroy(fr);
 }
 
+static void test_light_skeleton(void)
+{
+	struct exceptions_cleanup_light_lskel *skel;
+	__u64 ctx = 0;
+	int err;
+
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		    .ctx_in = &ctx,
+		    .ctx_size_in = sizeof(ctx),
+	);
+
+	skel = exceptions_cleanup_light_lskel__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "light open_and_load"))
+		return;
+
+	err = bpf_prog_test_run_opts(skel->progs.entry_light.prog_fd, &topts);
+	if (!ASSERT_OK(err, "run"))
+		goto out;
+	ASSERT_EQ(topts.retval, THROW_COOKIE, "retval");
+	ASSERT_EQ(skel->bss->pads_ran, RAN_LIGHT, "pads_ran");
+out:
+	exceptions_cleanup_light_lskel__destroy(skel);
+}
+
 static void test_shapes(void)
 {
 	struct exceptions_cleanup_shapes *skel;
@@ -387,6 +412,9 @@ void test_exceptions_cleanup(void)
 
 	exceptions_cleanup__destroy(skel);
 
+	if (test__start_subtest("light_skeleton"))
+		test_light_skeleton();
+
 	test_shapes();
 
 	RUN_TESTS(exceptions_cleanup_fail);
tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c new file +39 -0
new file mode 100644
index 000000000000..de7ae478d67f
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "exceptions_cleanup.h"
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+	bpf_throw(0);
+	bpf_preempt_disable();
+	bpf_preempt_enable();
+	bpf_unwind_resume();
+}
+
+__u64 pads_ran = 0;
+
+SEC("syscall")
+__naked int entry_light(void)
+{
+	asm volatile (
+	"call bpf_preempt_disable;"
+	"r1 = %[cookie];"
+"1:"	"call bpf_throw;"		/* cleanup region */
+"2:"
+	"exit;"
+"3:"					/* landing pad */
+	"call bpf_preempt_enable;"
+	PAD_RAN("%[ran]")
+	"call bpf_unwind_resume;"
+	"exit;"
+	CLEANUP_REC("1b", "2b", "3b")
+	:
+	: [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_LIGHT),
+	  __imm_addr(pads_ran)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
Signature · 1 line
2.53.0-Meta