Skip to contentSkip to conversation

bpf: Run exception cleanup landing pads when bpf_unwind() unwinds

PATCH In review Yonghong Song 5 participants 37 messages Oct 17

Subsystems (9)
x86 Kselftest BPF arm64 +5 more subsystems
lore
Share & tools
lore

Yonghong Song

author reviewer
#0 lore raw

v12 cover bpf: Run exception cleanup landing pads when bpf_unwind() unwinds

bpf_throw() walks the BPF call stack to the exception boundary and discards every frame in between. A frame that owns something -- an RCU read lock, a preemption-disabled section, a referenced kptr -- never gets to give it back, so the verifier refuses to let such a frame throw at all. That is the whole reason a Rust program cannot use bpf_throw() as its panic path right now: Rust's Drop glue *is* that give-back, and there is nowhere to run it.

LLVM 23 added the compiler half ([1]). A Rust function that owns a value across a call that can unwind

    fn foo() {
        let _guard = RcuReadGuard::new();  /* bpf_rcu_read_lock()   */
        may_throw();                       /* extern "C-unwind"     */
    }                                      /* Drop: rcu_read_unlock */

lowers to an invoke with a cleanup landing pad holding the Drop call, and the BPF backend writes one record per invoke region into a .bpf_cleanup
section: a flat table of 12-byte (begin, end, landing_pad) triples, each field a byte offset into the code section. The rule is "a frame suspended at a call in [begin, end) resumes at landing_pad when an unwind passes through". A pad ends with a call to _Unwind_Resume(), which the kernel provides as the bpf_unwind_resume() kfunc.

Rather than overload bpf_throw(), which keeps its own meaning -- leave for the exception boundary with the frames in between discarded -- a new bpf_unwind() kfunc raises the unwind this series dispatches.

This series is the kernel half: take that table at BPF_PROG_LOAD, teach the verifier to follow an unwind to the landing pad it reaches, and have bpf_unwind() send each frame it leaves there.

C has no unwinding, so the selftests spell out by hand what a frontend emits -- a call site bracketed by two labels, a landing pad, and a record tying them together. The frame above, written that way:

"call bpf_rcu_read_lock;"
"1:" "call foo3;" /* cleanup region */
"2:"
... normal path, ends in bpf_rcu_read_unlock ...
"6:" /* landing pad */
"call bpf_rcu_read_unlock;"
"call bpf_unwind_resume;" CLEANUP_REC("1b", "2b", "6b")

Design
======

A pad runs in the frame that owns it, entered by an ordinary return. bpf_unwind() walks the frames with arch_bpf_stack_walk_ra(), which hands out the slot each frame's return address came from, and rewrites that slot for every frame above the one that called it: to the landing pad where a record covers the call the address returns into, and to that frame's epilogue where nothing does. It finds the frames in the calling program itself, whose aux the verifier passes it as an implicit argument. Then it returns. The frame that called it goes on after the call, where the fixups put 'r0 = 0' and then a jump to its own pad, or an exit where no record covers the call.

Each frame therefore runs its own pad, on its own stack, and leaves through its own epilogue -- which is what puts its caller's r6-r9 back. The unwind needs no trampoline, no spill area and no per-frame metadata beyond the table itself. The fixups lower a pad's bpf_unwind_resume() to 'r0 = 0; exit', so the frame returns and the address rewritten below it carries the unwind on to the next pad. For main -> A -> B -> C with only A's call to B covered, an unwind raised in C runs:

  frame   runs
  -----   ----------------------------------------------------------
  C       'r0 = 0; exit', patched in after its bpf_unwind()
  B       its epilogue, putting back A's r6-r9
  A       its pad, which drops A's resources; the resume is an exit
  main    its epilogue, returning 0 to the kernel

The verifier follows the same path. A bpf_unwind() goes on at its own frame's pad, or pops frames to the first pad that covers a call, or to the main program's exit, where nothing may still be held and the zero returned has to suit the program type. Only there are an unwind's resources checked, since a pad may drop what another frame acquired, as when Rust moves a value into a callee. A resume goes on below its frame the same way. The pad is entered from the state the unwind leaves, not from a snapshot at the call: before unwinding, a callee may have written its caller's stack through a pointer, overwritten a spilled pointer, or changed packet data, and the callee's epilogue puts back none of that. A global subprog is verified on its own, so a call to one that can unwind gets a second successor, the unwind, taken from the state the call returns in. Precision backtracking follows the same edges back, into the frame the unwind left.

1-2 uapi: cleanup_info in BPF_PROG_LOAD and struct bpf_cleanup_info;
the bpf_unwind() and bpf_unwind_resume() kfuncs
3-5 keep a call site's pad in insn_aux_data, mark the covered call
sites, refuse what cannot carry a table, and make the landing pads reachable in the CFG and in liveness
6 verifier: follow an unwind through landing pads and epilogues
7-9 refuse a callback that can unwind and a pad that does not
resume; no private stack for a program that can unwind
10 lower the unwind and resume calls, and build the native cleanup
tables
11 keep an exit in every function the walk can pass, putting one in
a function that has none
12 let a subprog end with a call that does not return, as a compiler
emits one

  13     bpf_unwind(): rewrite the return addresses as it walks
  14     refuse a trampoline that calls a subprog that can unwind
  15-16  x86-64 and arm64 JITs
  17-21  libbpf: resolve _Unwind_Resume, collect .bpf_cleanup and pass it

to the kernel, carry it through the light skeleton and the static linker
22-25 selftests

Limitations
===========

- Cleanup pads only. A catch pad, which Rust's catch_unwind would need, is refused: bpf_unwind() rewrites every frame in one pass.
- Needs a JIT that dispatches pads: x86-64 with CONFIG_UNWINDER_ORC, and arm64. Elsewhere the load fails with -EOPNOTSUPP.
- Not with offload, a private stack, a verifier_ops epilogue, bpf_throw() or an exception callback.
- In a pad's frame: no tail call, BPF_LD_[ABS|IND] or indirect jump, and no second unwind in the pad or anything it calls.
- A subprog that can unwind cannot be a callback, nor be attached by fexit, fmod_ret or fsession. One with a callx counts as able to unwind when any subprog whose address is taken can.
- Until the Rust toolchain supports BPF exception handling, the selftests write their tables by hand in inline asm.

[1] https://github.com/llvm/llvm-project/pull/192164
llvm commit 9d51c891b719 ("[BPF] Add exception handling support with .bpf_cleanup section")

Changelog
=========
v11 -> v12:
- v11: https://lore.kernel.org/bpf/[email protected]/
- Rebase on bpf-next, and drop patch 1, which landed there as commit a3923834d155 ("bpf: Pack bpf_insn_aux_data flags into bit fields").
- Patch 12: put an exit after a call that ends a subprog before the CFG walk, so no later pass needs to know about such a call. check_subprogs() alone decides which final calls are accepted; a call to a global subprog or to bpf_throw() no longer is.
- Patch 11: drop the adjustment of a final call's offset, since no subprog ends with a call by the time it runs.
v10 -> v11:
- v10: https://lore.kernel.org/bpf/[email protected]/
- New patch 12, split out of patch 11: a subprog an unwind passes through without an exit, such as one ending in the loop rustc puts after a call that does not return, gets one after its last insn instead of being refused. The last insn keeps its aux data.
- New patch 13: let a subprog end with a call that does not return, the way rustc ends one with a call to a '-> !' function and LLVM a pad with its _Unwind_Resume call. check_subprogs() accepts a final call to a subprog with no exit, or to bpf_unwind(), bpf_unwind_resume() or bpf_throw(); do_check() refuses a path on which the call returns. The CFG walk, stack liveness and precision backtracking handle a call with no fall-through.
- When an unwind with no pad leaves a resource held, report it as "bpf_unwind cannot be used inside ..." at the unwinding insn, instead of as a BPF_EXIT in main.
- Run the in-pad checks from do_check_insn().
- arm64: sign the new return address with PACIA1716, so no assembler support for pointer authentication is needed.
- libbpf: read the .bpf_cleanup fields with memcpy(), as the section may be unaligned in a linked object; when _Unwind_Resume's prototype does not match, name bpf_unwind_resume in the warning.
v9 -> v10:
- v9: https://lore.kernel.org/bpf/[email protected]/
- Split the callback refusal into its own patch 8.
- Fix a verifier_bug() when a global call's pad is the next insn.
- On a cleanup_info record with a nonzero tail, fail with -E2BIG without writing a record size back into the attr.
v8 -> v9:
- v8: https://lore.kernel.org/bpf/[email protected]/
- unwind_frames() destroys popped frames' dynptrs.
- Drop patch 8: it refused a callee's pad dropping a reference its caller acquired, which is how Rust compiles moving a value into a callee. Held resources are checked only where main or a global subprog returns.
- Drop the per-insn in/outside-pad marks; compare in_pad when pruning.
- Split patch 11 into 10 (prepare the JITed program) and 11 (the walk).
- bpf_unwind() finds frames in its program's aux (KF_IMPLICIT_ARGS), not in kallsyms, which can go while an instance still runs.
- New patch 12: refuse fexit/fmod_ret/fsession on an unwinding subprog.
- x86, arm64: make bpf_unwind() notrace and NOKPROBE, drop the hooked-return warning; arm64: drop the PAC modifier check.
- Fix an unwind out of main inside a loop, and a global subprog returning in R0:R2; narrow the callx might_unwind marking.
- libbpf: skip records of an overridden weak function; leave table checks to the kernel.
v7 -> v8:
- v7: https://lore.kernel.org/bpf/[email protected]/
- Follow an unwind to its pad from the state it leaves, not from a snapshot at the call; patch 7 is renamed to match.
- Follow the unwind out of a call to a global subprogram as well: the call continues at the next insn and, from the state it returns in, at its pad or further out.
- Backtrack precision across an unwind; clear r1-r5 at a main-frame unwind exit.
- bpf_unwind() leaves its own caller alone; the fixups put 'r0 = 0' and a jump to its pad after the call.
- Refuse bpf_throw() and an exception callback with any unwind, not only with a table.
- Refuse a subprogram an unwind passes through that has no exit.
- Patch 8's rule that a frame an unwind leaves holds what it entered with no longer keeps the pads safe; it stays to refuse a program at the frame at fault.
- A speculative walk leaves no in-pad or outside-pad mark.
- x86, arm64: warn on a hooked return. arm64: take the PAC modifier as record + 16, and allow a shadow call stack.
- libbpf: narrow the _Unwind_Resume translation, close the opts hole, size the light skeleton's attr by need, and tighten the linker check.
v6 -> v7:
- v6: https://lore.kernel.org/bpf/[email protected]/
- New patch: require an unwind to leave a frame holding what it entered with, including at a call it passes through with no record over it.
- A resume ends the verifier's path instead of walking back to the instruction after the call, where an unwind never returns.
- Refuse a private stack for any program that can unwind, not only one carrying a table.
- Refuse a table for a program whose verifier_ops plants an epilogue, and scan the whole instruction stream for bpf_throw().
- Keep the last exit of every subprogram an unwind can pass through, not only the one after a bpf_unwind() call, and search for it.
- Fix precision backtracking across a resume, push a landing pad by itself so the CFG walk's DFS invariant holds, and mark callx sites.
- x86: leave a frame whose return a tracer has hooked alone; no ENDBR at a pad head, which is only ever reached by a return.
- arm64: no BTI at a pad head; ask the build whether a return address is signed; refuse cleanup pads where a shadow call stack is in use.
- Answer a speculative walk reaching a pad with a barrier, not a refusal.
- selftests: add the negative shapes for the above, allow repeated __set_global()/__ret_global(), and drop duplicate accepted shapes.
v5 -> v6:
- v5: https://lore.kernel.org/bpf/[email protected]/
- Run a pad in the frame that owns it: bpf_unwind() rewrites each frame's saved return address rather than calling the pad as a subroutine of the walker. The bpf_cleanup_pad.S trampolines, the per-frame spill area and the pad-entry register header all go away.
- Raise the unwind with a new bpf_unwind() kfunc, so that bpf_throw() keeps its meaning.
- Add arch_bpf_stack_walk_ra(), which also hands out the slot a return address came from. arm64 re-signs the address it writes there.
- A pad is an ordinary second successor of a covered call, so the verifier needs no unwind edge of its own: the cross-frame precision and liveness work is gone, with the v5 fixes it needed, and two patches become one.
- Refuse the undispatchable shapes per instruction in do_check(), keyed on a per-frame mark, rather than by walking each pad.
- Allow on-stack call arguments in a pad, which now has its own frame.
- Add __set_global() and __ret_global() test tags, so RUN_TESTS() drives the shapes and test_shapes() goes away (suggested by Eduard).
- Drop the shapes that needed a driver of their own, and the three extension objects with them.
v4 -> v5:
- v4: https://lore.kernel.org/bpf/[email protected]/
- Rebase on bpf-next.
- Clear r0 where a throw enters a landing pad: no instruction defines it, so a precision request for it outlived the state and oopsed the verifier.
- Defer entering the frames an unwind edge crossed until the backtrack reads an instruction from them; entering at the landing pad could leave bt->frame past the parent state's frames and oops the verifier.
- Stamp the popped frame count inside bpf_push_jmp_history(), so a pad's entry carries it even when the prune path is what creates the entry.
- Replace the hand-rolled CFG traversal and its separate pass with per-instruction checks in do_check(), keyed on the verifier's unwinding state; kernel/bpf/exception.c halves.
- Add a first patch packing bpf_insn_aux_data's flags into one bit field word, 144 bytes to 128, so the flags this series adds cost bits.
- Drop the per-subprogram arrays the JITs consulted for throw sites, resume sites and pad bodies, and read insn_aux_data, which a JIT already has.
- Drop the pad-entry r0 header: r0 at a pad is an unknown scalar, and a dispatcher writes a defined value there only to keep a kernel one out of BPF.
- Take the bool arguments back out of verifier_remove_insns() and the site collector, and share pop_frame() with prepare_func_exit().
- Rename the recorded call sites to throw_call and resume_call, give the exported functions a bpf_exc_ prefix, and drop cleanup_ from the statics.
v3 -> v4:
- v3: https://lore.kernel.org/bpf/[email protected]/
- Rebase on bpf-next due to conflict.
- Reserve the throw-site spill area only in a (sub)program that calls bpf_throw(): 40 bytes of stack per frame on x86-64, 80 on arm64.
- Bound the record count by the number of instructions in the program, and name both in the message.
- Refuse a .bpf_cleanup section in libbpf whose record count cannot be handed to the kernel as a count times a record size in an int.
- Fix the static linker's new bounds check, which could itself wrap, and refuse a section too small to hold one field.
- WARN once if the body of bpf_unwind_resume() is ever reached, the way bpf_throw() does where its exception callback should never return.
- Rename nr_pad_body to pad_body_bits, use BTF_ID_LIST_SINGLE, move cleanup_pad out of insn_aux_data's bools, drop an arm64 include.
v2 -> v3:
- v2: https://lore.kernel.org/bpf/[email protected]/
- Keep a landing pad's record when opt_remove_nops() deletes a pad that is a nop, instead of dropping it after the verifier has already checked the call site against it.
- Refuse a BPF_LD_[ABS|IND] in a pad body.
- Teach mark_chain_precision() about the throw-to-pad edge, which crosses frames with no instruction to account for them.
- Refuse a bpf_unwind_resume() in any frame but the one whose landing pad the walker entered.
- Check raw_data, alignment and bounds before the static linker writes through a relocation in a non-executable section, which may be SHT_NOBITS.
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Consolidate all usages of kern_extern_name() in a single patch in libbpf.
- Avoid compiler warning and add proper cleanup_info_cnt guard in libbpf when collecting .bpf_cleanup records.
- Add cleanup_info_cnt condition for emit_rel_store() with cleanup_info.

Yonghong Song (25):
bpf: Accept the compiler's exception cleanup table at program load
bpf: Add the bpf_unwind() and bpf_unwind_resume() kfuncs
bpf: Keep a call site's landing pad in insn_aux_data, add lookups
bpf: Mark covered call sites and check a program can take a table
bpf: Make exception landing pads reachable in the CFG
bpf: Verify an unwind through landing pads and epilogues
bpf: Refuse a callback that can unwind
bpf: Refuse a landing pad that does not resume
bpf: Do not use a private stack for a program that can unwind
bpf: Prepare JITed programs for dispatching cleanup pads
bpf: Keep an epilogue for each subprog an unwind can pass through
bpf: Allow a subprog to end with a call that does not return
bpf: Dispatch cleanup pads by rewriting return addresses
bpf: Refuse a trampoline that calls a subprog that can unwind bpf, x86: Dispatch exception cleanup pads at run time
bpf, arm64: Dispatch exception cleanup pads at run time
libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc
libbpf: Add cleanup_info to bpf_prog_load_opts
libbpf: Collect .bpf_cleanup records and pass them to the kernel
libbpf: Carry the exception cleanup table through the light skeleton
libbpf: Let the static linker carry .bpf_cleanup relocations selftests/bpf: Add end-to-end and negative .bpf_cleanup exception
tests
selftests/bpf: Add __set_global() and __ret_global() test tags selftests/bpf: Cover more accepted .bpf_cleanup exception shapes selftests/bpf: Load an exception cleanup program from a light skeleton

 arch/arm64/kernel/stacktrace.c                |   77 ++
 arch/arm64/net/bpf_jit_comp.c                 |   16 +
 arch/x86/net/bpf_jit_comp.c                   |   36 +
 include/linux/bpf.h                           |   31 +
 include/linux/bpf_verifier.h                  |   20 +-
 include/linux/filter.h                        |    3 +
 include/uapi/linux/bpf.h                      |   14 +
 kernel/bpf/Makefile                           |    2 +-
 kernel/bpf/backtrack.c                        |   40 +-
 kernel/bpf/cfg.c                              |  126 ++
 kernel/bpf/core.c                             |   12 +
 kernel/bpf/exception.c                        |  379 ++++++
 kernel/bpf/exception.h                        |   30 +
 kernel/bpf/fixups.c                           |  191 ++-
 kernel/bpf/helpers.c                          |   78 ++
 kernel/bpf/liveness.c                         |   24 +-
 kernel/bpf/states.c                           |   11 +-
 kernel/bpf/syscall.c                          |    2 +-
 kernel/bpf/verifier.c                         |  322 ++++-
 tools/include/uapi/linux/bpf.h                |   14 +
 tools/lib/bpf/bpf.c                           |    6 +-
 tools/lib/bpf/bpf.h                           |   10 +-
 tools/lib/bpf/gen_loader.c                    |   29 +-
 tools/lib/bpf/libbpf.c                        |  343 ++++-
 tools/lib/bpf/libbpf_internal.h               |   10 +
 tools/lib/bpf/linker.c                        |   39 +-
 tools/testing/selftests/bpf/Makefile.skel     |    2 +-
 .../selftests/bpf/exceptions_cleanup.h        |   48 +
 .../bpf/prog_tests/exceptions_cleanup.c       |  173 +++
 tools/testing/selftests/bpf/progs/bpf_misc.h  |   12 +
 .../selftests/bpf/progs/exceptions_cleanup.c  |  157 +++
 .../bpf/progs/exceptions_cleanup_fail.c       |  792 ++++++++++++
 .../bpf/progs/exceptions_cleanup_shapes.c     | 1134 +++++++++++++++++
 .../bpf/progs/exceptions_cleanup_tracing.c    |   26 +
 tools/testing/selftests/bpf/test_loader.c     |  319 ++++-
 35 files changed, 4467 insertions(+), 61 deletions(-)
 create mode 100644 kernel/bpf/exception.c
 create mode 100644 kernel/bpf/exception.h
 create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_tracing.c
Signature · 1 line
2.53.0-Meta
author reviewer
#1 lore raw

PATCH v12 1/25 bpf: Accept the compiler's exception cleanup table at program load

LLVM 23 added exception handling for BPF with the .bpf_cleanup section [1]: Rust code compiled with panic=unwind runs cleanup code (Drop glue) when an unwind passes through, and the BPF backend emits the section from the landing pads. rustc does not fully support BPF exception handling yet, and C can produce the section only with inline asm, which is enough to test it.

Add the UAPI to carry the table to BPF_PROG_LOAD: cleanup_info, cleanup_info_cnt and cleanup_info_rec_size. A record, struct bpf_cleanup_info, says that calls in [begin_off, end_off) that unwind resume at landing_pad_off, all instruction indices.

bpf_exc_check_info() checks the table at load time: records sorted by begin_off, with non-empty and disjoint ranges, each inside one subprog, no landing pad inside any range, and no offset naming the second half of an ld_imm64. A record longer than the kernel's is taken only if the rest is zero, and, unlike func_info, no size is written back for the loader to zero the rest and retry: a field the kernel does not know could change what a pad does. Nothing reads the table yet.

---
 include/linux/bpf_verifier.h   |   2 +
 include/uapi/linux/bpf.h       |  14 ++++
 kernel/bpf/Makefile            |   2 +-
 kernel/bpf/exception.c         | 137 +++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h         |  15 ++++
 kernel/bpf/syscall.c           |   2 +-
 kernel/bpf/verifier.c          |   6 ++
 tools/include/uapi/linux/bpf.h |  14 ++++
 8 files changed, 190 insertions(+), 2 deletions(-)
 create mode 100644 kernel/bpf/exception.c
 create mode 100644 kernel/bpf/exception.h
include/linux/bpf_verifier.h +2 -0
index 112ca1511da4..11cd90d85bad 100644
@@ -1135,6 +1135,8 @@ struct bpf_verifier_env {
 	struct spill_snapshot **callsite_at_stack;
 	u32 pass_cnt; /* number of times do_check() was called */
 	u32 subprog_cnt;
+	struct bpf_cleanup_info *cleanup_info;
+	u32 cleanup_info_cnt;
 	/* number of instructions analyzed by the verifier */
 	u32 prev_insn_processed, insn_processed;
 	/* number of jmps, calls, exits analyzed so far */
include/uapi/linux/bpf.h +14 -0
index 6f70db7515dd..9d7439c96e52 100644
@@ -1708,6 +1708,9 @@ union bpf_attr {
 		 * verification.
 		 */
 		__s32		keyring_id;
+		__aligned_u64	cleanup_info;	/* exception cleanup table */
+		__u32		cleanup_info_rec_size; /* userspace bpf_cleanup_info size */
+		__u32		cleanup_info_cnt; /* number of bpf_cleanup_info records */
 	};
 
 	struct { /* anonymous struct used by BPF_OBJ_* commands */
@@ -7669,6 +7672,17 @@ struct bpf_line_info {
 	__u32	line_col;
 };
 
+/*
+ * One record of an exception cleanup table: calls in [begin_off, end_off)
+ * that unwind resume at landing_pad_off. All three are instruction offsets
+ * in the program as loaded.
+ */
+struct bpf_cleanup_info {
+	__u32	begin_off;
+	__u32	end_off;
+	__u32	landing_pad_off;
+};
+
 struct bpf_spin_lock {
 	__u32	val;
 };
kernel/bpf/Makefile +1 -1
index 269810b128c4..df71ec19d0a0 100644
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o exception.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
kernel/bpf/exception.c new file +137 -0

137 lines not on this page · raw message

kernel/bpf/exception.h new file +15 -0
new file mode 100644
index 000000000000..cf099dcc5b74
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef __BPF_EXCEPTION_H
+#define __BPF_EXCEPTION_H
+
+#include <linux/bpfptr.h>
+#include <linux/types.h>
+
+union bpf_attr;
+struct bpf_verifier_env;
+
+int bpf_exc_check_info(struct bpf_verifier_env *env, const union bpf_attr *attr,
+		       bpfptr_t uattr);
+
+#endif /* __BPF_EXCEPTION_H */
kernel/bpf/syscall.c +1 -1
index 0f27604a54aa..e85a3e979f47 100644
@@ -2948,7 +2948,7 @@ int __init __used bpf_multi_func(void) { return 0; }
 BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
 
 /* last field in 'union bpf_attr' used by this command */
-#define BPF_PROG_LOAD_LAST_FIELD keyring_id
+#define BPF_PROG_LOAD_LAST_FIELD cleanup_info_cnt
 
 static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
 {
kernel/bpf/verifier.c +6 -0
index ddfa0c6d2755..1facec5485b3 100644
@@ -39,6 +39,7 @@
 
 #include "diagnostics.h"
 #include "disasm.h"
+#include "exception.h"
 
 static const struct bpf_verifier_ops * const bpf_verifier_ops[] = {
 #define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
@@ -23468,6 +23469,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	ret = bpf_exc_check_info(env, attr, uattr);
+	if (ret < 0)
+		goto skip_full_check;
+
 	/* Validate instructions and resolve the program's referenced resources. */
 	ret = check_and_resolve_insns(env);
 	if (ret < 0)
@@ -23713,6 +23718,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	kvfree(env->func_ptrs);
 	bpf_diag_free(env);
 	kvfree(env->idoms);
+	kvfree(env->cleanup_info);
 	kvfree(env);
 	return ret;
 }
tools/include/uapi/linux/bpf.h +14 -0
index 6f70db7515dd..9d7439c96e52 100644
@@ -1708,6 +1708,9 @@ union bpf_attr {
 		 * verification.
 		 */
 		__s32		keyring_id;
+		__aligned_u64	cleanup_info;	/* exception cleanup table */
+		__u32		cleanup_info_rec_size; /* userspace bpf_cleanup_info size */
+		__u32		cleanup_info_cnt; /* number of bpf_cleanup_info records */
 	};
 
 	struct { /* anonymous struct used by BPF_OBJ_* commands */
@@ -7669,6 +7672,17 @@ struct bpf_line_info {
 	__u32	line_col;
 };
 
+/*
+ * One record of an exception cleanup table: calls in [begin_off, end_off)
+ * that unwind resume at landing_pad_off. All three are instruction offsets
+ * in the program as loaded.
+ */
+struct bpf_cleanup_info {
+	__u32	begin_off;
+	__u32	end_off;
+	__u32	landing_pad_off;
+};
+
 struct bpf_spin_lock {
 	__u32	val;
 };
Signature · 1 line
2.53.0-Meta
author reviewer
#2 lore raw

PATCH v12 2/25 bpf: Add the bpf_unwind() and bpf_unwind_resume() kfuncs

In reply to Yonghong Song

An exception cleanup needs two terminators:

- bpf_unwind() begins an unwind: the frames it leaves run their landing pads on the way out to the program's exit. It is separate from bpf_throw(), which discards the frames in between.
- bpf_unwind_resume() ends a landing pad, carrying the unwind on. The compiler calls it _Unwind_Resume, which a later libbpf patch resolves to this name, and passes it the exception object, which the kfunc takes as ptr__ign.

Neither is registered yet: that waits for the patch that dispatches pads at run time.

---
 kernel/bpf/helpers.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
kernel/bpf/helpers.c +13 -0
index 439370d73cc5..8df3f4746b68 100644
@@ -3424,6 +3424,10 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
 	return false;
 }
 
+__bpf_kfunc void bpf_unwind(void)
+{
+}
+
 __bpf_kfunc void bpf_throw(u64 cookie)
 {
 	struct bpf_throw_ctx ctx = {};
@@ -3445,6 +3449,15 @@ __bpf_kfunc void bpf_throw(u64 cookie)
 	WARN(1, "A call to BPF exception callback should never return\n");
 }
 
+__bpf_kfunc void bpf_unwind_resume(void *ptr__ign)
+{
+	/*
+	 * Not meant to run: every call of this kfunc is to be lowered to
+	 * 'r0 = 0; exit' by the verifier.
+	 */
+	WARN_ONCE(1, "exception cleanup resume was not lowered to a return\n");
+}
+
 __bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, unsigned int flags)
 {
 	struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
Signature · 1 line
2.53.0-Meta
author reviewer
#3 lore raw

PATCH v12 3/25 bpf: Keep a call site's landing pad in insn_aux_data, add lookups

In reply to Yonghong Song

A call site's landing pad, if any, is kept in insn_aux_data as cleanup_pad. adjust_insn_aux_data(), which every instruction patch goes through, and verifier_remove_insns() keep it in step, and bpf_opt_remove_nops() moves a pad off a nop to the next instruction before removing the nop. Add lookups for it and for calls to bpf_unwind() and bpf_unwind_resume(); their users come in later patches.

---
 include/linux/bpf_verifier.h |  5 +++++
 kernel/bpf/exception.c       | 24 ++++++++++++++++++++++++
 kernel/bpf/exception.h       |  4 ++++
 kernel/bpf/fixups.c          | 26 +++++++++++++++++++++++++-
 4 files changed, 58 insertions(+), 1 deletion(-)
include/linux/bpf_verifier.h +5 -0
index 11cd90d85bad..89a714057820 100644
@@ -806,6 +806,11 @@ struct bpf_insn_aux_data {
 	u64 jump_target:1;
 
 	unsigned int orig_idx; /* original instruction index, initialized once */
+	/*
+	 * 1 + the instruction index of the exception cleanup landing pad
+	 * this call site unwinds to, or 0 for none.
+	 */
+	u32 cleanup_pad;
 	/*
 	 * CFG strongly connected component this instruction belongs to,
 	 * zero if it is a singleton SCC.
kernel/bpf/exception.c +24 -0
index ac3b575f935e..9ab29c73c44b 100644
@@ -2,6 +2,8 @@
 /* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
 #include <linux/bpf.h>
 #include <linux/bpf_verifier.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
 #include <linux/slab.h>
 #include "exception.h"
 
@@ -135,3 +137,25 @@ int bpf_exc_check_info(struct bpf_verifier_env *env, const union bpf_attr *attr,
 	kvfree(krecord);
 	return ret;
 }
+
+BTF_ID_LIST_SINGLE(bpf_unwind_id, func, bpf_unwind)
+BTF_ID_LIST_SINGLE(bpf_unwind_resume_id, func, bpf_unwind_resume)
+
+bool bpf_is_unwind_kfunc(const struct bpf_insn *insn)
+{
+	return bpf_pseudo_kfunc_call(insn) && insn->off == 0 &&
+	       insn->imm == bpf_unwind_id[0];
+}
+
+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn)
+{
+	return bpf_pseudo_kfunc_call(insn) && insn->off == 0 &&
+	       insn->imm == bpf_unwind_resume_id[0];
+}
+
+int bpf_exc_pad_of_call(struct bpf_verifier_env *env, u32 idx)
+{
+	u32 pad = env->insn_aux_data[idx].cleanup_pad;
+
+	return pad ? (int)pad - 1 : -1;
+}
kernel/bpf/exception.h +4 -0
index cf099dcc5b74..d5c6ac459870 100644
@@ -8,8 +8,12 @@
 
 union bpf_attr;
 struct bpf_verifier_env;
+struct bpf_insn;
 
 int bpf_exc_check_info(struct bpf_verifier_env *env, const union bpf_attr *attr,
 		       bpfptr_t uattr);
+int bpf_exc_pad_of_call(struct bpf_verifier_env *env, u32 idx);
+bool bpf_is_unwind_kfunc(const struct bpf_insn *insn);
+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn);
 
 #endif /* __BPF_EXCEPTION_H */
kernel/bpf/fixups.c +25 -1
index 608621d7f225..feb3b079dfbb 100644
@@ -244,11 +244,18 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 			data[i].non_stack_access =
 				data[off + cnt - 1].non_stack_access;
 			data[off + cnt - 1].non_stack_access = false;
+			data[i].cleanup_pad = data[off + cnt - 1].cleanup_pad;
+			data[off + cnt - 1].cleanup_pad = 0;
 		} else if (bpf_is_mem_insn(insn + i)) {
 			data[i].non_stack_access = true;
 		}
 	}
 
+	if (env->cleanup_info_cnt)
+		for (i = 0; i < prog_len; i++)
+			if (data[i].cleanup_pad > off + 1)
+				data[i].cleanup_pad += cnt - 1;
+
 	/*
 	 * Last slot instruction could be a newly generated
 	 * BPF_ST/BPF_LDX/BPF_STX, systematically mark it for non-stack access
@@ -604,6 +611,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 	struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
 	unsigned int orig_prog_len = env->prog->len;
 	int err;
+	u32 i;
 
 	if (bpf_rewrite_must_abort())
 		return -EINTR;
@@ -634,6 +642,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 		sizeof(*aux_data) * (orig_prog_len - off - cnt));
 	env->insn_aux_data_len -= cnt;
 
+	if (env->cleanup_info_cnt) {
+		for (i = 0; i < env->insn_aux_data_len; i++) {
+			u32 pad = aux_data[i].cleanup_pad;
+
+			if (pad > off + cnt)
+				aux_data[i].cleanup_pad = pad - cnt;
+			else if (pad > off)
+				aux_data[i].cleanup_pad = 0;
+		}
+	}
+
 	return 0;
 }
 
@@ -739,7 +758,7 @@ int bpf_opt_remove_nops(struct bpf_verifier_env *env)
 	struct bpf_insn *insn = env->prog->insnsi;
 	int insn_cnt = env->prog->len;
 	bool is_may_goto_0, is_ja;
-	int i, err;
+	int i, j, err;
 
 	for (i = 0; i < insn_cnt; i++) {
 		is_may_goto_0 = !memcmp(&insn[i], &MAY_GOTO_0, sizeof(MAY_GOTO_0));
@@ -750,6 +769,11 @@ int bpf_opt_remove_nops(struct bpf_verifier_env *env)
 		if (aux[i].indirect_target)
 			continue;
 
+		if (env->cleanup_info_cnt)
+			for (j = 0; j < insn_cnt; j++)
+				if (aux[j].cleanup_pad == i + 1)
+					aux[j].cleanup_pad = i + 2;
+
 		err = verifier_remove_insns(env, i, 1);
 		if (err)
 			return err;
Signature · 1 line
2.53.0-Meta
author reviewer
#4 lore raw

PATCH v12 4/25 bpf: Mark covered call sites and check a program can take a table

In reply to Yonghong Song

Before the CFG walk, bpf_exc_mark_call_sites() records the landing pad of every call in a cleanup record's range that can unwind: a BPF-to-BPF call, direct or indirect, or a call to bpf_unwind().

bpf_exc_check_prog() refuses a table where pad dispatch cannot work. An offloaded program cannot take one, and neither can a program without a JIT that dispatches pads. Nor can a program whose verifier_ops has a gen_epilogue, as bpf_qdisc's does: the epilogue is planted on the exits bpf_convert_ctx_accesses() sees, and the exits an unwind returns through are added after it. A table cannot be combined with an exception callback or with a bpf_throw() anywhere in the program either, since a throw leaves without rewriting return addresses, so no pad would run.

bpf_exc_check_prog() runs after check_attach_btf_id(), since a struct_ops program's gen_epilogue is only known then, and marks a program that passes it jit_required. bpf_jit_supports_cleanup_pads() says no until the arch patches.

---
 include/linux/filter.h |  1 +
 kernel/bpf/core.c      |  5 ++++
 kernel/bpf/exception.c | 64 ++++++++++++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h |  2 ++
 kernel/bpf/verifier.c  |  9 ++++++
 5 files changed, 81 insertions(+)
include/linux/filter.h +1 -0
index 9339c6131f8f..c8ca863e352a 100644
@@ -1248,6 +1248,7 @@ bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_arena_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
+bool bpf_jit_supports_cleanup_pads(void);
 bool bpf_jit_supports_ptr_xchg(void);
 bool bpf_jit_supports_arena(void);
 bool bpf_jit_supports_arena_scalar(void);
kernel/bpf/core.c +5 -0
index 642f0ab8eb2d..a5a5463449c1 100644
@@ -3524,6 +3524,11 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp,
 {
 }
 
+bool __weak bpf_jit_supports_cleanup_pads(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_timed_may_goto(void)
 {
 	return false;
kernel/bpf/exception.c +64 -0

70 lines not on this page · raw message

kernel/bpf/exception.h +2 -0
index d5c6ac459870..30918714a6e8 100644
@@ -12,6 +12,8 @@ struct bpf_insn;
 
 int bpf_exc_check_info(struct bpf_verifier_env *env, const union bpf_attr *attr,
 		       bpfptr_t uattr);
+void bpf_exc_mark_call_sites(struct bpf_verifier_env *env);
+int bpf_exc_check_prog(struct bpf_verifier_env *env);
 int bpf_exc_pad_of_call(struct bpf_verifier_env *env, u32 idx);
 bool bpf_is_unwind_kfunc(const struct bpf_insn *insn);
 bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn);
kernel/bpf/verifier.c +9 -0
index 1facec5485b3..eb60311b7d1f 100644
@@ -23473,6 +23473,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/* The CFG needs an edge from a call in a cleanup range to its pad. */
+	bpf_exc_mark_call_sites(env);
+
 	/* Validate instructions and resolve the program's referenced resources. */
 	ret = check_and_resolve_insns(env);
 	if (ret < 0)
@@ -23510,6 +23513,12 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret)
 		goto skip_full_check;
 
+	if (env->cleanup_info_cnt) {
+		ret = bpf_exc_check_prog(env);
+		if (ret)
+			goto skip_full_check;
+	}
+
 	ret = bpf_compute_const_regs(env);
 	if (ret < 0)
 		goto skip_full_check;
Signature · 1 line
2.53.0-Meta
author reviewer
#5 lore raw

PATCH v12 5/25 bpf: Make exception landing pads reachable in the CFG

In reply to Yonghong Song

A call that a cleanup record covers can also reach the record's landing pad. Add that edge to the CFG walk, to bpf_insn_successors(), and to stack liveness, where a slot is alive across a call when the call's pad reads it.

The CFG walk pushes the pad in a visit of its own, and the fall-through on the next visit. bpf_check_cfg() re-visits the top of its stack until DONE_EXPLORING, so a visit pushing both would leave the pad DISCOVERED off the current path, and push_insn() would take a branch from one pad into another -- how a frame with two regions chains them -- for a back-edge.

Without the liveness change, a slot only the pad reads looked dead, and clean_verifier_state() poisoned it while the callee ran.

---
 kernel/bpf/cfg.c      | 38 ++++++++++++++++++++++++++++++++++++++
 kernel/bpf/liveness.c | 24 +++++++++++++++++++++++-
 2 files changed, 61 insertions(+), 1 deletion(-)
kernel/bpf/cfg.c +38 -0
index bce7b5abcfda..ad7ea08f3b15 100644
@@ -6,6 +6,7 @@
 #include <linux/sort.h>
 
 #include "diagnostics.h"
+#include "exception.h"
 
 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
 
@@ -160,6 +161,38 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
 	return DONE_EXPLORING;
 }
 
+static int visit_cleanup_pad_edge(int t, struct bpf_verifier_env *env)
+{
+	int *insn_stack = env->cfg.insn_stack;
+	int *insn_state = env->cfg.insn_state;
+	int w;
+
+	if (!env->cleanup_info_cnt)
+		return DONE_EXPLORING;
+	w = bpf_exc_pad_of_call(env, t);
+	if (w < 0)
+		return DONE_EXPLORING;
+
+	/*
+	 * @t is a call that may branch here, and @w is the target of that
+	 * branch, so both are prune points. @w especially: every covered call
+	 * site in a region unwinds to the same pad, and without a prune point
+	 * at its head the verifier walks the pad again for each of them.
+	 */
+	mark_prune_point(env, t);
+	mark_prune_point(env, w);
+	mark_jmp_point(env, w);
+	mark_jump_target(env, w);
+
+	if (insn_state[w])
+		return DONE_EXPLORING;
+	if (env->cfg.cur_stack >= env->prog->len)
+		return -E2BIG;
+	insn_stack[env->cfg.cur_stack++] = w;
+	insn_state[w] |= DISCOVERED;
+	return KEEP_EXPLORING;
+}
+
 static int visit_func_call_insn(int t, struct bpf_insn *insns,
 				struct bpf_verifier_env *env,
 				bool visit_callee)
@@ -167,6 +200,11 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,
 	int ret, insn_sz;
 	int w;
 
+	/* One push per visit: @t is revisited once the pad is explored. */
+	ret = visit_cleanup_pad_edge(t, env);
+	if (ret != DONE_EXPLORING)
+		return ret;
+
 	insn_sz = bpf_is_ldimm64(&insns[t]) ? 2 : 1;
 	ret = push_insn(t, t + insn_sz, FALLTHROUGH, env);
 	if (ret)
kernel/bpf/liveness.c +23 -1
index e657e6d95408..8b7ee551ff6f 100644
@@ -8,6 +8,8 @@
 #include <linux/slab.h>
 #include <linux/sort.h>
 
+#include "exception.h"
+
 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
 
 /*
@@ -404,6 +406,18 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 			succ->items[succ->cnt++] = exit_idx;
 	}
 
+	/*
+	 * A covered call can also leave through its landing pad. Only a call to
+	 * a subprog or to bpf_unwind() is covered, so the two entries of
+	 * env->succ are enough.
+	 */
+	if (unlikely(env->cleanup_info_cnt)) {
+		int pad = bpf_exc_pad_of_call(env, idx);
+
+		if (pad >= 0)
+			succ->items[succ->cnt++] = pad;
+	}
+
 	return succ;
 }
 
@@ -530,7 +544,8 @@ bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp
 	 * Slot is alive if it is read before q->insn_idx in current func instance,
 	 * or if for some outer func instance:
 	 * - alive before callsite if callsite calls callback or is callx, otherwise
-	 * - alive after callsite
+	 * - alive after callsite,
+	 * - or alive at the landing pad a cleanup record gives the callsite
 	 */
 	struct live_stack_query *q = &env->liveness->live_stack_query;
 	struct func_instance *instance, *curframe_instance;
@@ -565,6 +580,13 @@ bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp
 		alive = callee_stack_access_at_callsite(env, callsite)
 			? is_live_before(instance, callsite, rel, half_spi)
 			: is_live_before(instance, callsite + 1, rel, half_spi);
+
+		if (!alive && unlikely(env->cleanup_info_cnt)) {
+			int pad = bpf_exc_pad_of_call(env, callsite);
+
+			if (pad >= 0)
+				alive = is_live_before(instance, pad, rel, half_spi);
+		}
 		if (alive)
 			return true;
 	}
Signature · 1 line
2.53.0-Meta
author reviewer
#6 lore raw

PATCH v12 6/25 bpf: Verify an unwind through landing pads and epilogues

In reply to Yonghong Song

Once a later patch makes bpf_unwind() dispatch pads, an unwind rewrites the return address of every frame it passes: to the pad where a record covers the frame's call, else to the frame's epilogue. Each frame then returns normally, a pad through its resume, which is lowered to 'r0 = 0; exit'. The verifier follows the unwind the same way: into the pad of each frame that has one, out of each frame that does not, and out of main as a program exit.

  instruction                          goes on at
  -----------------------------------  ---------------------------------
  bpf_unwind(), record over it         its own frame's pad
  bpf_unwind(), no record over it      unwind_frames()
  bpf_unwind_resume()                  unwind_frames()
  call to a global subprog that can    next insn, and the unwind from the
  unwind                               returned state: to the call's pad,

or on through unwind_frames()

Whether a subprog can unwind is worked out on the CFG, as might_throw
is: a call to bpf_unwind() marks its subprog might_unwind, and merge_callee_effects() carries the mark to its callers. A callx goes around merge_callee_effects(), so if any subprog whose address is taken can unwind, mark_callx_might_unwind() marks every subprog holding a callx, and its callers. That is conservative, as only the main walk knows which subprog a callx calls.

unwind_frames(), for main -> A -> B -> C where only A's call is covered:

  program                              run time: bpf_unwind() in C
  -----------------------------------  ---------------------------------
  main:     call A                     main returns via its epilogue
  A:     1: call B     [1, 2) -> P     A resumes at P
         2: ...
         P: <drop A's resources>
            call bpf_unwind_resume
  B:        call C     no record       B returns via its epilogue
  C:        call bpf_unwind            C returns via 'r0 = 0; exit'
  frame   at C's bpf_unwind()   unwind_frames()     at P's resume
  -----   -------------------   -----------------   -----------------
    3     C  <- curframe        popped
    2     B                     popped
    1     A                     A  <- curframe, P   popped
    0     main                  main                exit with r0 = 0

P gets A's frame as B and C left it, with r0 unknown and r1-r5 cleared. With no pad left, the program ends through process_bpf_exit_full(). Only then are the unwind's resources checked, and a leak is reported against bpf_unwind at the insn that unwound: a frame it pops may leave something for a caller's pad to drop.

The pad's state comes from the unwind, not from a snapshot at the call:
before unwinding, the callee may have written the caller's stack through a pointer, overwritten a spilled pointer, reinitialised a dynptr or an iterator, or changed packet data, none of which its epilogue restores. A global subprog is verified on its own, so the unwind out of a call to one starts from the state the call returns in, after check_func_call().

Precision backtracking gets three new edges:

  edge                     frame
  -----------------------  ---------------------------------------------
  global call <- its pad   stays in the caller (subseq_idx is the pad)
  unwind <- pad            moves to the frame the unwind left, recorded

in its history entry under INSN_F_UNWIND
unwind <- main's exit the same, after a jump from the unwinding insn
to where the path ends

Where no frame has a pad, the path ends at E, the insn it exits from in place of a BPF_EXIT: main's call into the popped frames, else the unwinding insn U itself. Backtracking for the return check starts at E without processing it, while r0 is set at U, Q being U's predecessor:

frames popped none popped

main: E: call A main: Q: r6 = 0
A: call B E = U: call bpf_unwind
B: Q: r6 = 0
U: call bpf_unwind

walk: E -> U -> Q -> ... walk: E -> U -> Q -> ...
^ skipped ^ skipped

The jump from U to E takes the walk to U rather than to the insn before E; where E is U, it is what gets U processed at all. U's INSN_F_UNWIND mark matters there only for an unwinding global call, which backtracking would otherwise take for one that returned.

Unlike a BPF_EXIT in main, the end of such a path can be inside a loop, where maybe_exit_scc() finds no visit info for it. The new unwind_exit in the state tells it that the path ended there.

---
 include/linux/bpf_verifier.h |  11 ++-
 kernel/bpf/backtrack.c       |  40 +++++++++-
 kernel/bpf/cfg.c             |  88 +++++++++++++++++++++
 kernel/bpf/states.c          |   8 +-
 kernel/bpf/verifier.c        | 145 ++++++++++++++++++++++++++++++++++-
 5 files changed, 281 insertions(+), 11 deletions(-)
include/linux/bpf_verifier.h +8 -3
index 89a714057820..cc95d2ad015c 100644
@@ -429,6 +429,8 @@ enum {
 	INSN_F_SRC_REG_STACK = BIT(2), /* src_reg is PTR_TO_STACK */
 
 	INSN_F_STACK_ARG_ACCESS = BIT(3),
+
+	INSN_F_UNWIND = BIT(4),
 };
 
 /* Registers linked to one jump condition that a history entry can record */
@@ -437,10 +439,10 @@ enum {
 struct bpf_jmp_history_entry {
 	/* insn idx can't be bigger than 1 million */
 	u32 idx : 20;
-	u32 frame : 4;	/* stack access frame number */
+	u32 frame : 4;	/* stack access frame number, or the frame an unwind ran in */
 	/* special INSN_F_xxx flags */
-	u32 flags : 4;
-	u32 : 4;
+	u32 flags : 5;
+	u32 : 3;
 	u32 prev_idx : 20;
 	u32 spi : 12;	/* stack slot index */
 	/*
@@ -527,6 +529,8 @@ struct bpf_verifier_state {
 
 	bool speculative;
 	bool in_sleepable;
+	/* the path ends with an unwind returning from frame 0 */
+	bool unwind_exit;
 
 	/*
 	 * First and last insn idx of this verifier state.
@@ -949,6 +953,7 @@ struct bpf_subprog_info {
 	s16 fastcall_stack_off;
 	bool has_tail_call: 1;
 	bool might_throw: 1;
+	bool might_unwind: 1;
 	bool tail_call_reachable: 1;
 	bool has_ld_abs: 1;
 	bool is_cb: 1;
kernel/bpf/backtrack.c +36 -4
index 43d5740034c0..16f6ef59ddf7 100644
@@ -4,6 +4,7 @@
 #include <linux/bpf_verifier.h>
 #include <linux/filter.h>
 #include <linux/bitmap.h>
+#include "exception.h"
 
 #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
 
@@ -424,7 +425,31 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 		if (class == BPF_STX)
 			bt_set_reg(bt, sreg);
 	} else if (class == BPF_JMP || class == BPF_JMP32) {
-		if (bpf_pseudo_call(insn) || bpf_is_callx(insn)) {
+		if (hist && (hist->flags & INSN_F_UNWIND)) {
+			/*
+			 * A bpf_unwind(), a resume or an unwinding global call
+			 * left frame hist->frame here, for a landing pad in
+			 * this one, or for the main frame's exit, which is
+			 * this frame itself when it is the main one. The walk
+			 * crosses back into that frame, past any frames
+			 * between, which were entered and never returned
+			 * from. The pad found r0 unknown and r1-r5 clobbered;
+			 * r6-r9 and the stack are this frame's own and stay
+			 * marked in its masks until the walk comes back out.
+			 */
+			bt_clear_reg(bt, BPF_REG_0);
+			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
+				verifier_bug(env, "backtracking unwind unexpected regs %x",
+					     bt_reg_mask(bt));
+				return -EFAULT;
+			}
+			if (verifier_bug_if(hist->frame < bt->frame, env,
+					    "unwind from frame %d to frame %d",
+					    hist->frame, bt->frame))
+				return -EFAULT;
+			bt->frame = hist->frame;
+			return 0;
+		} else if (bpf_pseudo_call(insn) || bpf_is_callx(insn)) {
 			int subprog_insn_idx, subprog = -1;
 
 			if (bpf_pseudo_call(insn)) {
@@ -440,10 +465,17 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				 * extra instructions from subprog; the next
 				 * instruction after call to global subprog
 				 * should be literally next instruction in
-				 * caller program
+				 * caller program, or the call's landing pad,
+				 * branched to from the state the call returns
+				 * in when the callee can unwind. A pad may be
+				 * the next instruction too, so the two edges
+				 * cannot be told apart, and need not be: the
+				 * pad finds r0 unknown and r1-r5 cleared, so
+				 * both clear the same registers below.
 				 */
-				verifier_bug_if(idx + 1 != subseq_idx, env,
-						"extra insn from subprog");
+				verifier_bug_if(idx + 1 != subseq_idx &&
+						bpf_exc_pad_of_call(env, idx) != subseq_idx,
+						env, "extra insn from subprog");
 				/* global subprog always sets R0 */
 				bt_clear_reg(bt, BPF_REG_0);
 				/* and if it does not set R2, main pass would catch it */
kernel/bpf/cfg.c +88 -0

118 lines not on this page · raw message

kernel/bpf/states.c +6 -2
index 953700822fa7..a3dd2f5712e7 100644
@@ -201,9 +201,13 @@ static int maybe_exit_scc(struct bpf_verifier_env *env, struct bpf_verifier_stat
 		 * c. A checkpoint is reached and matched. Checkpoints are created by
 		 *    is_state_visited(), which calls maybe_enter_scc(), which allocates
 		 *    bpf_scc_visit instances for checkpoints within SCCs.
-		 * (c) is the only case that can reach this point.
+		 * d. An unwind returns from frame 0, ending the path at a call or at
+		 *    the unwinding insn, which may be in an SCC as a top-level
+		 *    BPF_EXIT is not. Like (b), it leaves nothing to exit.
+		 * (c) and (d), and a speculative path, are the only ways to reach
+		 * this point.
 		 */
-		if (!st->speculative) {
+		if (!st->speculative && !st->unwind_exit) {
 			verifier_bug(env, "scc exit: no visit info for call chain %s",
 				     format_callchain(env, callchain));
 			return -EFAULT;
kernel/bpf/verifier.c +143 -2

175 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#7 lore raw

Re: [PATCH bpf-next v12 06/25] bpf: Verify an unwind through landing pads and epilogues

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] bpf: missing might_unwind propagation in mark_callx_might_unwind() --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: missing might_unwind propagation in mark_callx_might_unwind() --

commit c4002615f0bb137c4f685915b07888fb1dc5bbea
Author: Yonghong Song <[email protected]>

bpf: Verify an unwind through landing pads and epilogues

This commit teaches the BPF verifier to follow stack unwinds from bpf_unwind(), bpf_unwind_resume(), and unwinding global subprograms through caller landing pads and epilogues, including CFG effect propagation and precision backtracking.

Quoted 4 lines
kernel/bpf/cfg.c +0 -0
index ad7ea08f3b15e..9608293bdc069 100644

[ ... ]

Quoted 8 lines
kernel/bpf/cfg.c +1 -0
@@ -91,6 +99,7 @@ static void merge_callee_effects(struct bpf_verifier_env *env, int t, int w)
 	caller->changes_pkt_data |= callee->changes_pkt_data;
 	caller->might_sleep |= callee->might_sleep;
 	caller->might_throw |= callee->might_throw;
+	caller->might_unwind |= callee->might_unwind;
 }
 
 enum {

[ ... ]

Quoted 39 lines
kernel/bpf/cfg.c +35 -0
@@ -705,6 +716,82 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
 	}
 }
 
+static bool addr_taken_subprog_might_unwind(struct bpf_verifier_env *env)
+{
+	struct bpf_insn *insns = env->prog->insnsi;
+	struct bpf_subprog_info *callee;
+	struct bpf_func_ptr *ptrs;
+	u32 cnt, j;
+	int i;
+
+	for (i = 0; i < env->prog->len; i++) {
+		if (bpf_pseudo_func(&insns[i])) {
+			callee = bpf_find_containing_subprog(env, i + insns[i].imm + 1);
+			if (callee->might_unwind)
+				return true;
+			continue;
+		}
+		ptrs = insn_func_ptrs(env, i, &cnt);
+		for (j = 0; j < cnt; j++) {
+			callee = bpf_find_containing_subprog(env, ptrs[j].xlated_off);
+			if (callee->might_unwind)
+				return true;
+		}
+	}
+	return false;
+}
+
+/*
+ * merge_callee_effects() does not carry might_unwind to a subprog that calls
+ * through a pointer it was handed. So if any subprog a callx can reach may
+ * unwind, mark every subprog with a callx, and its callers.
+ *
+ * This is conservative: a subprog with a callx is marked even when nothing it
+ * calls unwinds. The main walk knows each callx's target and could tell those
+ * apart.
+ */
+static void mark_callx_might_unwind(struct bpf_verifier_env *env)

kernel/bpf/cfg.c static int visit_insn(int t, struct bpf_verifier_env *env) · line 763

Quoted 10 lines
kernel/bpf/cfg.c +10 -0
+{
+	struct bpf_insn *insns = env->prog->insnsi;
+	struct bpf_subprog_info *caller, *callee;
+	int i, j, len = env->prog->len;
+	struct bpf_func_ptr *ptrs;
+	bool changed;
+	u32 cnt;
+
+	if (!addr_taken_subprog_might_unwind(env))
+		return;

[Severity: High]
Can addr_taken_subprog_might_unwind() return false prematurely here when an address-taken subprogram calls back into an unwinding subprogram whose entry instruction is still in the DISCOVERED state during the CFG walk?

In bpf_check_cfg(), visit_insn() traverses insn_func_ptrs() (for example, an ld_imm64 referencing a read-only map such as .rodata that holds function pointers) and bpf_pseudo_func() as CFG edges. Because neither sort_subprogs_topo() nor check_max_stack_depth_subprog() rejects cycles through insn_func_ptrs() or async callback bpf_pseudo_func() edges, a subprogram cb whose address is in .rodata or taken via bpf_pseudo_func() can call a subprogram G via BPF_PSEUDO_CALL while G references .rodata or &cb on one branch and calls bpf_unwind() on a second branch.

If bpf_check_cfg() explores G before cb:

1. G's entry instruction is marked DISCOVERED and pushes branch 1 onto insn_stack while G->might_unwind is still false.
2. Branch 1's insn_func_ptrs() or bpf_pseudo_func() edge pushes cb onto insn_stack.
3. cb's BPF_PSEUDO_CALL to G runs merge_callee_effects() while G->might_unwind is still false, and push_insn() sees G's entry is already DISCOVERED and returns DONE_EXPLORING, so cb finishes exploration with cb->might_unwind == false.
4. bpf_check_cfg() later explores branch 2 of G and sets G->might_unwind = true.

When mark_callx_might_unwind() then runs, addr_taken_subprog_might_unwind() sees cb->might_unwind == false and returns early before the fixed-point loop below can propagate G->might_unwind to cb and its callers.

As a result, a global subprogram calling cb (via callx or BPF_PSEUDO_CALL) will have might_unwind == false, causing process_global_call_unwind() to skip verifying the unwind path at the caller and allowing acquired references or locks to leak on unwind.

Quoted 30 lines
kernel/bpf/cfg.c +30 -0
+
+	for (i = 0; i < len; i++)
+		if (bpf_is_callx(&insns[i]))
+			bpf_find_containing_subprog(env, i)->might_unwind = true;
+
+	do {
+		changed = false;
+		for (i = 0; i < len; i++) {
+			caller = bpf_find_containing_subprog(env, i);
+			if (caller->might_unwind)
+				continue;
+			if (bpf_pseudo_call(&insns[i]) || bpf_pseudo_func(&insns[i])) {
+				callee = bpf_find_containing_subprog(env, i + insns[i].imm + 1);
+				if (!callee->might_unwind)
+					continue;
+				caller->might_unwind = true;
+				changed = true;
+				continue;
+			}
+			ptrs = insn_func_ptrs(env, i, &cnt);
+			for (j = 0; j < cnt; j++) {
+				callee = bpf_find_containing_subprog(env, ptrs[j].xlated_off);
+				if (!callee->might_unwind)
+					continue;
+				caller->might_unwind = true;
+				changed = true;
+			}
+		}
+	} while (changed);
+}

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
author reviewer
#8 lore raw

Re: [PATCH bpf-next v12 06/25] bpf: Verify an unwind through landing pads and epilogues

Quoted 106 lines · On 10/16/26 8:27 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: missing might_unwind propagation in mark_callx_might_unwind() --

commit c4002615f0bb137c4f685915b07888fb1dc5bbea
Author: Yonghong Song <[email protected]>

bpf: Verify an unwind through landing pads and epilogues

This commit teaches the BPF verifier to follow stack unwinds from bpf_unwind(), bpf_unwind_resume(), and unwinding global subprograms through caller landing pads and epilogues, including CFG effect propagation and precision backtracking.

Quoted 4 lines
kernel/bpf/cfg.c +0 -0
index ad7ea08f3b15e..9608293bdc069 100644

[ ... ]

Quoted 8 lines
kernel/bpf/cfg.c +1 -0
@@ -91,6 +99,7 @@ static void merge_callee_effects(struct bpf_verifier_env *env, int t, int w)
  	caller->changes_pkt_data |= callee->changes_pkt_data;
  	caller->might_sleep |= callee->might_sleep;
  	caller->might_throw |= callee->might_throw;
+	caller->might_unwind |= callee->might_unwind;
  }
 
  enum {

[ ... ]

Quoted 39 lines
kernel/bpf/cfg.c +35 -0
@@ -705,6 +716,82 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
  	}
  }
 
+static bool addr_taken_subprog_might_unwind(struct bpf_verifier_env *env)
+{
+	struct bpf_insn *insns = env->prog->insnsi;
+	struct bpf_subprog_info *callee;
+	struct bpf_func_ptr *ptrs;
+	u32 cnt, j;
+	int i;
+
+	for (i = 0; i < env->prog->len; i++) {
+		if (bpf_pseudo_func(&insns[i])) {
+			callee = bpf_find_containing_subprog(env, i + insns[i].imm + 1);
+			if (callee->might_unwind)
+				return true;
+			continue;
+		}
+		ptrs = insn_func_ptrs(env, i, &cnt);
+		for (j = 0; j < cnt; j++) {
+			callee = bpf_find_containing_subprog(env, ptrs[j].xlated_off);
+			if (callee->might_unwind)
+				return true;
+		}
+	}
+	return false;
+}
+
+/*
+ * merge_callee_effects() does not carry might_unwind to a subprog that calls
+ * through a pointer it was handed. So if any subprog a callx can reach may
+ * unwind, mark every subprog with a callx, and its callers.
+ *
+ * This is conservative: a subprog with a callx is marked even when nothing it
+ * calls unwinds. The main walk knows each callx's target and could tell those
+ * apart.
+ */
+static void mark_callx_might_unwind(struct bpf_verifier_env *env)

kernel/bpf/cfg.c static int visit_insn(int t, struct bpf_verifier_env *env) · line 763

Quoted 10 lines
kernel/bpf/cfg.c +10 -0
+{
+	struct bpf_insn *insns = env->prog->insnsi;
+	struct bpf_subprog_info *caller, *callee;
+	int i, j, len = env->prog->len;
+	struct bpf_func_ptr *ptrs;
+	bool changed;
+	u32 cnt;
+
+	if (!addr_taken_subprog_might_unwind(env))
+		return;

[Severity: High]
Can addr_taken_subprog_might_unwind() return false prematurely here when an address-taken subprogram calls back into an unwinding subprogram whose entry instruction is still in the DISCOVERED state during the CFG walk?

In bpf_check_cfg(), visit_insn() traverses insn_func_ptrs() (for example, an ld_imm64 referencing a read-only map such as .rodata that holds function pointers) and bpf_pseudo_func() as CFG edges. Because neither sort_subprogs_topo() nor check_max_stack_depth_subprog() rejects cycles through insn_func_ptrs() or async callback bpf_pseudo_func() edges, a subprogram cb whose address is in .rodata or taken via bpf_pseudo_func() can call a subprogram G via BPF_PSEUDO_CALL while G references .rodata or &cb on one branch and calls bpf_unwind() on a second branch.

If bpf_check_cfg() explores G before cb:

1. G's entry instruction is marked DISCOVERED and pushes branch 1 onto

    insn_stack while G->might_unwind is still false.
2. Branch 1's insn_func_ptrs() or bpf_pseudo_func() edge pushes cb onto
    insn_stack.
3. cb's BPF_PSEUDO_CALL to G runs merge_callee_effects() while
    G->might_unwind is still false, and push_insn() sees G's entry is
    already DISCOVERED and returns DONE_EXPLORING, so cb finishes exploration
    with cb->might_unwind == false.
4. bpf_check_cfg() later explores branch 2 of G and sets
    G->might_unwind = true.
Quoted 8 lines

When mark_callx_might_unwind() then runs, addr_taken_subprog_might_unwind() sees cb->might_unwind == false and returns early before the fixed-point loop below can propagate G->might_unwind to cb and its callers.

As a result, a global subprogram calling cb (via callx or BPF_PSEUDO_CALL) will have might_unwind == false, causing process_global_call_unwind() to skip verifying the unwind path at the caller and allowing acquired references or locks to leak on unwind.

I think you are right, inside mark_callx_might_unwind(), before addr_taken_subprog_might_unwind(), we indeed need to do one pass for the below do loop as it may actually mark might_unwind as true.

Quoted 33 lines
Quoted 31 lines
kernel/bpf/cfg.c +30 -0
+
+	for (i = 0; i < len; i++)
+		if (bpf_is_callx(&insns[i]))
+			bpf_find_containing_subprog(env, i)->might_unwind = true;
+
+	do {
+		changed = false;
+		for (i = 0; i < len; i++) {
+			caller = bpf_find_containing_subprog(env, i);
+			if (caller->might_unwind)
+				continue;
+			if (bpf_pseudo_call(&insns[i]) || bpf_pseudo_func(&insns[i])) {
+				callee = bpf_find_containing_subprog(env, i + insns[i].imm + 1);
+				if (!callee->might_unwind)
+					continue;
+				caller->might_unwind = true;
+				changed = true;
+				continue;
+			}
+			ptrs = insn_func_ptrs(env, i, &cnt);
+			for (j = 0; j < cnt; j++) {
+				callee = bpf_find_containing_subprog(env, ptrs[j].xlated_off);
+				if (!callee->might_unwind)
+					continue;
+				caller->might_unwind = true;
+				changed = true;
+			}
+		}
+	} while (changed);
+}

[ ... ]

author reviewer
#9 lore raw

PATCH v12 7/25 bpf: Refuse a callback that can unwind

In reply to Yonghong Song

A callback is not called by the program but by kernel code, a helper or a kfunc such as bpf_loop(), which the callback returns to and which then returns to the program:

main -> A -> bpf_loop() -> cb -> C C calls bpf_unwind()

Once a later patch makes bpf_unwind() dispatch pads, it rewrites the return address of each of the program's frames and stops at the first frame that is not one, here bpf_loop(): kernel code with its own frame and state, which the walk cannot send cb past. So cb returns into bpf_loop(), which goes on looping and then returns to A after its call, while the verifier follows the unwind from cb straight into A.

Refuse a callback subprog marked might_unwind. The check does not depend on a cleanup table: without one, bpf_unwind() still sends every frame it walks through its epilogue and still stops at bpf_loop(), so the mismatch is the same.

The check does not tell the kinds of callback apart, so an async one, a timer callback for example, is refused as well, although nothing of the program sits above it for the walk to skip. And as mark_callx_might_unwind() marks a subprog with a callx conservatively, a callback with a callx is refused even when nothing it calls unwinds.

---
 kernel/bpf/exception.c | 9 +++++++++
 kernel/bpf/exception.h | 1 +
 kernel/bpf/verifier.c  | 4 ++++
 3 files changed, 14 insertions(+)
kernel/bpf/exception.c +9 -0
index 80d1edc19441..316ae9e4d7c7 100644
@@ -141,6 +141,15 @@ int bpf_exc_check_info(struct bpf_verifier_env *env, const union bpf_attr *attr,
 BTF_ID_LIST_SINGLE(bpf_unwind_id, func, bpf_unwind)
 BTF_ID_LIST_SINGLE(bpf_unwind_resume_id, func, bpf_unwind_resume)
 
+int bpf_exc_check_callback(struct bpf_verifier_env *env, int subprog)
+{
+	if (!env->subprog_info[subprog].might_unwind)
+		return 0;
+
+	verbose(env, "subprog %d may unwind and is used as a callback\n", subprog);
+	return -EINVAL;
+}
+
 static int reject_throw(struct bpf_verifier_env *env)
 {
 	u32 i;
kernel/bpf/exception.h +1 -0
index 30918714a6e8..49d4685c297a 100644
@@ -17,5 +17,6 @@ int bpf_exc_check_prog(struct bpf_verifier_env *env);
 int bpf_exc_pad_of_call(struct bpf_verifier_env *env, u32 idx);
 bool bpf_is_unwind_kfunc(const struct bpf_insn *insn);
 bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn);
+int bpf_exc_check_callback(struct bpf_verifier_env *env, int subprog);
 
 #endif /* __BPF_EXCEPTION_H */
kernel/bpf/verifier.c +4 -0
index fce2baa6272d..fcf0d0fe0f2f 100644
@@ -11191,6 +11191,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
 	 * callbacks
 	 */
 	env->subprog_info[subprog].is_cb = true;
+	err = bpf_exc_check_callback(env, subprog);
+	if (err)
+		return err;
+
 	if (bpf_pseudo_kfunc_call(insn) &&
 	    !is_callback_calling_kfunc(insn->imm)) {
 		verifier_bug(env, "kfunc %s#%d not marked as callback-calling",
Signature · 1 line
2.53.0-Meta
#10 lore raw

Re: [PATCH bpf-next v12 07/25] bpf: Refuse a callback that can unwind

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] bpf: missing might_unwind propagation to callback in bpf_check_cfg() --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: missing might_unwind propagation to callback in bpf_check_cfg() --

commit 1785732c5b5ae17b86f08f6a94449da36fb9c958
Author: Yonghong Song <[email protected]>

bpf: Refuse a callback that can unwind

This commit adds bpf_exc_check_callback() and calls it in push_callback_call() to reject callback subprograms that have might_unwind set.

Quoted 5 lines
kernel/bpf/verifier.c +0 -0
index fce2baa6272d1..fcf0d0fe0f2ff 100644

kernel/bpf/verifier.c static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins · line 11200

Quoted 10 lines
kernel/bpf/verifier.c +4 -0
@@ -11191,6 +11191,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
 	 * callbacks
 	 */
 	env->subprog_info[subprog].is_cb = true;
+	err = bpf_exc_check_callback(env, subprog);
+	if (err)
+		return err;
+
 	if (bpf_pseudo_kfunc_call(insn) &&
 	    !is_callback_calling_kfunc(insn->imm)) {
 		verifier_bug(env, "kfunc %s#%d not marked as callback-calling",

[Severity: High]
Can a callback subprogram that transitively calls bpf_unwind() bypass bpf_exc_check_callback() with env->subprog_info[subprog].might_unwind still false?

In bpf_check_cfg(), direct calls (BPF_PSEUDO_CALL), function address loads (BPF_PSEUDO_FUNC), and read-only map function pointer loads (insn_func_ptrs()) are explored in a single DFS sharing env->cfg.insn_state[]:

kernel/bpf/cfg.c:merge_callee_effects() {
	caller = bpf_find_containing_subprog(env, t);
	callee = bpf_find_containing_subprog(env, w);
	caller->changes_pkt_data |= callee->changes_pkt_data;
	caller->might_sleep |= callee->might_sleep;
	caller->might_throw |= callee->might_throw;
	caller->might_unwind |= callee->might_unwind;
}

merge_callee_effects() relies on callee w already being EXPLORED when returning to caller t. However, if subprogram H has a conditional branch whose fallthrough loads a read-only map containing the address of callback cb via insn_func_ptrs() (or loads cb via BPF_PSEUDO_FUNC when H is explored from a cleanup pad before main's fallthrough) and whose jump target calls bpf_unwind(), while cb calls H via BPF_PSEUDO_CALL:

1. bpf_check_cfg() enters H (marking insn_state[w_H] = DISCOVERED) and explores H's fallthrough first, before visiting H's bpf_unwind() branch, so H->might_unwind is still false.

2. H's fallthrough visits cb via visit_func_ptrs_insn() or visit_func_call_insn().

3. When cb visits its BPF_PSEUDO_CALL to H, merge_callee_effects() sees H->might_unwind == false, and push_insn() sees (insn_state[w_H] & 0xF0) == DISCOVERED and returns DONE_EXPLORING. Thus cb finishes CFG exploration with cb->might_unwind == false before H explores its bpf_unwind() branch and sets H->might_unwind = true.

4. At the end of bpf_check_cfg(), mark_callx_might_unwind() checks addr_taken_subprog_might_unwind():

kernel/bpf/cfg.c:mark_callx_might_unwind() {
	if (!addr_taken_subprog_might_unwind(env))
		return;
    ...
}

Because the address-taken subprogram cb still has might_unwind == false, addr_taken_subprog_might_unwind() returns false and skips the fixed-point propagation loop.

5. Neither sort_subprogs_topo() nor check_max_stack_depth() rejects the program because insn_func_ptrs() edges are not call edges and async BPF_PSEUDO_FUNC callbacks are skipped in stack depth checks.

When push_callback_call() then calls bpf_exc_check_callback(env, cb), env->subprog_info[cb].might_unwind is false, allowing the unwinding callback to pass verification and cause a verifier/runtime control-flow mismatch or jump to a zero epilogue_ip when bpf_unwind() is invoked at runtime.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
author reviewer
#11 lore raw

Re: [PATCH bpf-next v12 07/25] bpf: Refuse a callback that can unwind

Quoted 77 lines · On 10/16/26 8:19 PM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: missing might_unwind propagation to callback in bpf_check_cfg() --

commit 1785732c5b5ae17b86f08f6a94449da36fb9c958
Author: Yonghong Song <[email protected]>

bpf: Refuse a callback that can unwind

This commit adds bpf_exc_check_callback() and calls it in push_callback_call() to reject callback subprograms that have might_unwind set.

Quoted 5 lines
kernel/bpf/verifier.c +0 -0
index fce2baa6272d1..fcf0d0fe0f2ff 100644

kernel/bpf/verifier.c static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins · line 11200

Quoted 10 lines
kernel/bpf/verifier.c +4 -0
@@ -11191,6 +11191,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
  	 * callbacks
  	 */
  	env->subprog_info[subprog].is_cb = true;
+	err = bpf_exc_check_callback(env, subprog);
+	if (err)
+		return err;
+
  	if (bpf_pseudo_kfunc_call(insn) &&
  	    !is_callback_calling_kfunc(insn->imm)) {
  		verifier_bug(env, "kfunc %s#%d not marked as callback-calling",

[Severity: High]
Can a callback subprogram that transitively calls bpf_unwind() bypass bpf_exc_check_callback() with env->subprog_info[subprog].might_unwind still false?

In bpf_check_cfg(), direct calls (BPF_PSEUDO_CALL), function address loads (BPF_PSEUDO_FUNC), and read-only map function pointer loads (insn_func_ptrs()) are explored in a single DFS sharing env->cfg.insn_state[]:

kernel/bpf/cfg.c:merge_callee_effects() {
	caller = bpf_find_containing_subprog(env, t);
	callee = bpf_find_containing_subprog(env, w);
	caller->changes_pkt_data |= callee->changes_pkt_data;
	caller->might_sleep |= callee->might_sleep;
	caller->might_throw |= callee->might_throw;
	caller->might_unwind |= callee->might_unwind;
}

merge_callee_effects() relies on callee w already being EXPLORED when returning to caller t. However, if subprogram H has a conditional branch whose fallthrough loads a read-only map containing the address of callback cb via insn_func_ptrs() (or loads cb via BPF_PSEUDO_FUNC when H is explored from a cleanup pad before main's fallthrough) and whose jump target calls bpf_unwind(), while cb calls H via BPF_PSEUDO_CALL:

1. bpf_check_cfg() enters H (marking insn_state[w_H] = DISCOVERED) and explores H's fallthrough first, before visiting H's bpf_unwind() branch, so H->might_unwind is still false.

2. H's fallthrough visits cb via visit_func_ptrs_insn() or

    visit_func_call_insn().

3. When cb visits its BPF_PSEUDO_CALL to H, merge_callee_effects() sees

    H->might_unwind == false, and push_insn() sees
    (insn_state[w_H] & 0xF0) == DISCOVERED and returns DONE_EXPLORING.
    Thus cb finishes CFG exploration with cb->might_unwind == false
    before H explores its bpf_unwind() branch and sets
    H->might_unwind = true.

4. At the end of bpf_check_cfg(), mark_callx_might_unwind() checks addr_taken_subprog_might_unwind():

kernel/bpf/cfg.c:mark_callx_might_unwind() {
	if (!addr_taken_subprog_might_unwind(env))
		return;
     ...
}
Quoted 3 lines

Because the address-taken subprogram cb still has might_unwind == false, addr_taken_subprog_might_unwind() returns false and skips the fixed-point propagation loop.

The previous patch 6 indeed has a bug and this may cause addr_taken_subprog_might)unwind() return false and this will cause issues. Fixing patch 6 issue should fix the issue.

Quoted 11 lines

5. Neither sort_subprogs_topo() nor check_max_stack_depth() rejects the program because insn_func_ptrs() edges are not call edges and async BPF_PSEUDO_FUNC callbacks are skipped in stack depth checks.

When push_callback_call() then calls bpf_exc_check_callback(env, cb), env->subprog_info[cb].might_unwind is false, allowing the unwinding callback to pass verification and cause a verifier/runtime control-flow mismatch or jump to a zero epilogue_ip when bpf_unwind() is invoked at runtime.

author reviewer
#12 lore raw

PATCH v12 8/25 bpf: Refuse a landing pad that does not resume

In reply to Yonghong Song

A cleanup pad runs drop glue and calls bpf_unwind_resume(), so its frame returns and the unwind goes on. A catch pad carries on in its frame instead. Only the first is supported: bpf_unwind() rewrites every frame's return address in one pass, so a catch pad's caller would still resume at a pad.

Nothing in the record says which kind a pad is, but the code does: a cleanup pad reaches _Unwind_Resume, a catch pad a return. So the unwind marks the frame whose pad it enters, and refuses in the marked code:

  refused                                  where
  ---------------------------------------  ----------------------------
  an exit: how a catch pad ends            the pad's own frame
  a tail call: it replaces the frame       the pad's own frame
  a BPF_LD_[ABS|IND]: it leaves through    the pad's own frame

an exit on a failed load
an indirect jump: nothing a compiler the pad's own frame
frontend emits in a pad needs one
a bpf_unwind(), a call to a global the pad and its callees
subprog that might_unwind
a bpf_unwind_resume() outside a pad any program

The first three are refused only in the pad's own frame: a subprog the pad calls may do them and still come back.

---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/exception.c       | 58 ++++++++++++++++++++++++++++++++++++
 kernel/bpf/exception.h       |  1 +
 kernel/bpf/states.c          |  3 ++
 kernel/bpf/verifier.c        | 16 +++++++++-
 5 files changed, 78 insertions(+), 1 deletion(-)
include/linux/bpf_verifier.h +1 -0
index cc95d2ad015c..4ca943c37e3c 100644
@@ -379,6 +379,7 @@ struct bpf_func_state {
 	bool in_async_callback_fn;
 	bool in_exception_callback_fn;
 	bool no_stack_arg_load;
+	bool in_pad;
 	/* For callback calling functions that limit number of possible
 	 * callback executions (e.g. bpf_loop) keeps track of current
 	 * simulated iteration number.
kernel/bpf/exception.c +58 -0

64 lines not on this page · raw message

kernel/bpf/exception.h +1 -0
index 49d4685c297a..6d6550daa98e 100644
@@ -18,5 +18,6 @@ int bpf_exc_pad_of_call(struct bpf_verifier_env *env, u32 idx);
 bool bpf_is_unwind_kfunc(const struct bpf_insn *insn);
 bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn);
 int bpf_exc_check_callback(struct bpf_verifier_env *env, int subprog);
+int bpf_exc_check_insn(struct bpf_verifier_env *env, struct bpf_insn *insn);
 
 #endif /* __BPF_EXCEPTION_H */
kernel/bpf/states.c +3 -0
index a3dd2f5712e7..93b5cc3a2f02 100644
@@ -999,6 +999,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
 	if (!old->no_stack_arg_load && cur->no_stack_arg_load)
 		return false;
 
+	if (old->in_pad != cur->in_pad)
+		return false;
+
 	for (i = 0; i < MAX_BPF_REG; i++)
 		if (((1 << i) & live_regs) &&
 		    !regsafe(env, &old->regs[i], &cur->regs[i],
kernel/bpf/verifier.c +15 -1
index fcf0d0fe0f2f..e73492144e29 100644
@@ -15573,8 +15573,13 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 
 	if (bpf_is_unwind_kfunc(insn))
 		return process_bpf_unwind(env, insn_idx_p);
-	if (bpf_is_unwind_resume_kfunc(insn))
+	if (bpf_is_unwind_resume_kfunc(insn)) {
+		if (!cur_func(env)->in_pad) {
+			verbose(env, "resume at insn %d is not in a landing pad\n", insn_idx);
+			return -EINVAL;
+		}
 		return unwind_frames(env, NULL);
+	}
 
 	return 0;
 }
@@ -19721,6 +19726,7 @@ static int unwind_frames(struct bpf_verifier_env *env, bool *do_print_state)
 			return err;
 		clear_caller_saved_regs(env, caller->regs);
 		mark_reg_unknown(env, caller->regs, BPF_REG_0);
+		caller->in_pad = true;
 		env->insn_idx = pad;
 		if (do_print_state)
 			*do_print_state = true;
@@ -19795,6 +19801,7 @@ static int process_global_call_unwind(struct bpf_verifier_env *env,
 	frame = cur_func(env);
 	clear_caller_saved_regs(env, frame->regs);
 	mark_reg_unknown(env, frame->regs, BPF_REG_0);
+	frame->in_pad = true;
 	env->insn_idx = pad;
 	*do_print_state = true;
 	return INSN_IDX_UPDATED;
@@ -19809,6 +19816,7 @@ static int process_bpf_unwind(struct bpf_verifier_env *env, int *insn_idx)
 		return unwind_frames(env, NULL);
 	clear_caller_saved_regs(env, frame->regs);
 	mark_reg_unknown(env, frame->regs, BPF_REG_0);
+	frame->in_pad = true;
 	*insn_idx = pad;
 	return INSN_IDX_UPDATED;
 }
@@ -20004,6 +20012,12 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 	struct bpf_insn *insn = &env->prog->insnsi[env->insn_idx];
 	u8 class = BPF_CLASS(insn->code);
 
+	if (unlikely(env->cleanup_info_cnt)) {
+		err = bpf_exc_check_insn(env, insn);
+		if (err)
+			return err;
+	}
+
 	switch (class) {
 	case BPF_ALU:
 	case BPF_ALU64:
Signature · 1 line
2.53.0-Meta
author reviewer
#13 lore raw

PATCH v12 9/25 bpf: Do not use a private stack for a program that can unwind

In reply to Yonghong Song

A private stack keeps its frame pointer in %r9 on x86-64, saved and restored around every call. An unwind skips the restore: a frame resumed at its pad then addresses its stack through a stale pointer, and one sent to its epilogue pops its callee-saved registers one slot off.

So force NO_PRIV_STACK for a program that has a table or can unwind, on every arch for now.

---
 kernel/bpf/exception.c | 10 ++++++++++
 kernel/bpf/exception.h |  1 +
 kernel/bpf/verifier.c  | 11 +++++++++++
 3 files changed, 22 insertions(+)
kernel/bpf/exception.c +10 -0
index 43e01d0be44f..5945fdf5f75c 100644
@@ -183,6 +183,16 @@ void bpf_exc_mark_call_sites(struct bpf_verifier_env *env)
 	}
 }
 
+bool bpf_prog_may_unwind(const struct bpf_verifier_env *env)
+{
+	u32 i;
+
+	for (i = 0; i < env->subprog_cnt; i++)
+		if (env->subprog_info[i].might_unwind)
+			return true;
+	return false;
+}
+
 int bpf_exc_check_prog(struct bpf_verifier_env *env)
 {
 	int err;
kernel/bpf/exception.h +1 -0
index 6d6550daa98e..ecbca25dabef 100644
@@ -14,6 +14,7 @@ int bpf_exc_check_info(struct bpf_verifier_env *env, const union bpf_attr *attr,
 		       bpfptr_t uattr);
 void bpf_exc_mark_call_sites(struct bpf_verifier_env *env);
 int bpf_exc_check_prog(struct bpf_verifier_env *env);
+bool bpf_prog_may_unwind(const struct bpf_verifier_env *env);
 int bpf_exc_pad_of_call(struct bpf_verifier_env *env, u32 idx);
 bool bpf_is_unwind_kfunc(const struct bpf_insn *insn);
 bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn);
kernel/bpf/verifier.c +11 -0
index e73492144e29..9c512ed5d1dc 100644
@@ -5956,6 +5956,17 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
 		}
 	}
 
+	/*
+	 * A private stack keeps its frame pointer in %r9 on x86-64, restored
+	 * by a pop after the call that an unwind skips. A frame resumed at a
+	 * pad then addresses its stack through a stale pointer, and a frame
+	 * sent to its epilogue instead pops its callee-saved registers one
+	 * slot off. Do not use a private stack for any program that can
+	 * unwind, on every arch for now.
+	 */
+	if (env->cleanup_info_cnt || bpf_prog_may_unwind(env))
+		priv_stack_mode = NO_PRIV_STACK;
+
 	if (priv_stack_mode == PRIV_STACK_UNKNOWN)
 		priv_stack_mode = bpf_enable_priv_stack(env->prog);
 
Signature · 1 line
2.53.0-Meta
author reviewer
#14 lore raw

PATCH v12 10/25 bpf: Prepare JITed programs for dispatching cleanup pads

In reply to Yonghong Song

A following patch makes bpf_unwind() rewrite the saved return address of each frame above its caller: to the pad covering the call, else to the frame's epilogue. Every frame then just returns, its epilogue restoring its caller's r6-r9. This patch adds most of what that relies on:

- Each bpf_unwind() call is followed by 'r0 = 0' and a jump to its pad, or an exit: the walk leaves the calling frame alone.
- A pad's resume lowers to 'r0 = 0; exit'.
- Each JITed function gets a table of its covered calls as native address ranges, looked up by bpf_exc_pad_for_ip(). The main function's table, and its epilogue_ip, go to the outer program.
- arch_bpf_stack_walk_ra() walks the stack handing out each return-address slot. Where an arch has only the weak stub, nothing is dispatched, so any program that can unwind, table or not, is now held to bpf_exc_check_prog().

The next patch makes sure that every function the walk can pass has an epilogue.

---
 include/linux/bpf.h          |  30 ++++++++++
 include/linux/bpf_verifier.h |   1 +
 include/linux/filter.h       |   2 +
 kernel/bpf/core.c            |   7 +++
 kernel/bpf/exception.c       |  77 +++++++++++++++++++++++++
 kernel/bpf/exception.h       |   6 ++
 kernel/bpf/fixups.c          | 106 ++++++++++++++++++++++++++++++++++-
 kernel/bpf/verifier.c        |   5 +-
 8 files changed, 232 insertions(+), 2 deletions(-)
include/linux/bpf.h +30 -0
index 54144372281c..7f23f4efde01 100644
@@ -1841,6 +1841,34 @@ enum bpf_sig_keyring {
 	BPF_SIG_KEYRING_BPF,
 };
 
+/* One cleanup region of a JITed (sub)program. */
+struct bpf_cleanup_range {
+	u64 begin;
+	u64 end;
+	u64 pad;
+};
+
+struct bpf_exception_info {
+	struct bpf_cleanup_info *info;
+	struct bpf_cleanup_range *ranges;
+	u32 nr_info;
+	u32 nr_ranges;
+};
+
+#ifdef CONFIG_BPF_SYSCALL
+void bpf_exc_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image);
+void bpf_exc_free_info(struct bpf_prog_aux *aux);
+#else
+
+static inline void bpf_exc_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image)
+{
+}
+
+static inline void bpf_exc_free_info(struct bpf_prog_aux *aux)
+{
+}
+#endif
+
 struct bpf_prog_aux {
 	atomic64_t refcnt;
 	u32 used_map_cnt;
@@ -1922,6 +1950,8 @@ struct bpf_prog_aux {
 	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
 	u16 stack_arg_sp_adjust;
 	u16 freplace_link_cnt; /* counts freplace links extending this prog */
+	struct bpf_exception_info *exc;
+	u64 epilogue_ip; /* native address of this (sub)program's epilogue */
 #ifdef CONFIG_SECURITY
 	void *security;
 #endif
include/linux/bpf_verifier.h +1 -0
index 4ca943c37e3c..1c33b2cf29cf 100644
@@ -2003,6 +2003,7 @@ int bpf_opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, const union
 int bpf_convert_ctx_accesses(struct bpf_verifier_env *env);
 int bpf_jit_subprogs(struct bpf_verifier_env *env);
 int bpf_fixup_call_args(struct bpf_verifier_env *env);
+int bpf_exc_patch_unwind_calls(struct bpf_verifier_env *env);
 int bpf_do_misc_fixups(struct bpf_verifier_env *env);
 int bpf_insn_def32(struct bpf_prog *prog, struct bpf_insn *insn);
 
include/linux/filter.h +2 -0
index c8ca863e352a..a61a78675d11 100644
@@ -1291,6 +1291,8 @@ u32 bpf_jit_plan_arg_moves(const struct bpf_jit_arg_abi *abi,
 			   struct bpf_jit_arg_move *moves);
 u64 bpf_arch_uaddress_limit(void);
 void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie);
+void arch_bpf_stack_walk_ra(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp, u64 *ra),
+			    void *cookie);
 u64 arch_bpf_timed_may_goto(void);
 u64 bpf_check_timed_may_goto(struct bpf_timed_may_goto *);
 bool bpf_helper_changes_pkt_data(enum bpf_func_id func_id);
kernel/bpf/core.c +7 -0
index a5a5463449c1..3af8e8272691 100644
@@ -292,6 +292,7 @@ void __bpf_prog_free(struct bpf_prog *fp)
 		mutex_destroy(&fp->aux->dst_mutex);
 		mutex_destroy(&fp->aux->st_ops_assoc_mutex);
 		kfree(fp->aux->poke_tab);
+		bpf_exc_free_info(fp->aux);
 		kfree(fp->aux);
 	}
 	free_percpu(fp->stats);
@@ -3524,6 +3525,12 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp,
 {
 }
 
+void __weak arch_bpf_stack_walk_ra(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp,
+						      u64 *ra),
+				   void *cookie)
+{
+}
+
 bool __weak bpf_jit_supports_cleanup_pads(void)
 {
 	return false;
kernel/bpf/exception.c +77 -0

80 lines not on this page · raw message

kernel/bpf/exception.h +6 -0
index ecbca25dabef..09bbcd0a09de 100644
@@ -9,6 +9,10 @@
 union bpf_attr;
 struct bpf_verifier_env;
 struct bpf_insn;
+struct bpf_cleanup_info;
+struct bpf_cleanup_range;
+struct bpf_prog;
+struct bpf_prog_aux;
 
 int bpf_exc_check_info(struct bpf_verifier_env *env, const union bpf_attr *attr,
 		       bpfptr_t uattr);
@@ -20,5 +24,7 @@ bool bpf_is_unwind_kfunc(const struct bpf_insn *insn);
 bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn);
 int bpf_exc_check_callback(struct bpf_verifier_env *env, int subprog);
 int bpf_exc_check_insn(struct bpf_verifier_env *env, struct bpf_insn *insn);
+int bpf_exc_attach_info(struct bpf_prog_aux *aux, struct bpf_cleanup_info *recs, u32 cnt);
+const struct bpf_cleanup_range *bpf_exc_pad_for_ip(const struct bpf_prog *prog, u64 ip);
 
 #endif /* __BPF_EXCEPTION_H */
kernel/bpf/fixups.c +105 -1

148 lines not on this page · raw message

kernel/bpf/verifier.c +4 -1
index 9c512ed5d1dc..85542afcc369 100644
@@ -23683,7 +23683,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret)
 		goto skip_full_check;
 
-	if (env->cleanup_info_cnt) {
+	if (env->cleanup_info_cnt || bpf_prog_may_unwind(env)) {
 		ret = bpf_exc_check_prog(env);
 		if (ret)
 			goto skip_full_check;
@@ -23783,6 +23783,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 		/* program is valid, convert *(u32*)(ctx + off) accesses */
 		ret = bpf_convert_ctx_accesses(env);
 
+	if (ret == 0)
+		ret = bpf_exc_patch_unwind_calls(env);
+
 	if (ret == 0)
 		ret = bpf_do_misc_fixups(env);
 
Signature · 1 line
2.53.0-Meta
author reviewer
#15 lore raw

PATCH v12 11/25 bpf: Keep an epilogue for each subprog an unwind can pass through

In reply to Yonghong Song

A later patch makes bpf_unwind() walk the BPF frames on the stack. A frame whose call site has a landing pad continues at the pad. Any other frame is sent to its epilogue: its saved return address is rewritten to point there, and the frame returns to its caller from it. The x86 JIT emits a function's epilogue at its first exit insn, so every subprog the walk can pass through needs an exit that is still there when it is JITed. There are two ways it can be missing.

First, dead code removal deletes an exit the verifier never reached. In a subprog whose path to its exit always goes through a call that unwinds, that is every exit it has.

Second, a compiler emits no exit for a function that never returns. Built with panic=unwind, rustc compiles

        #[inline(never)]
        fn fail() -> ! { unsafe { bpf_unwind() }; loop {} }

to

fail:
call bpf_unwind
L0: goto L0

bpf_exc_patch_unwind_calls() later puts 'r0 = 0; exit' after the bpf_unwind() call, and fail() returns through it, since the walk leaves the return address of the frame that called bpf_unwind() alone. A function that calls fail() and then loops the same way, though, has no exit at all for the walk to send it to.

Add keep_subprog_exits(), which dead code removal calls before removing anything. For each subprog that can unwind and that the verifier reached, it marks the last exit as seen, so that the exit is kept. If the subprog has no exit, it puts one after the subprog's last insn. Only the walk ever gets there. keep_subprog_exits() runs before the exits after bpf_unwind() and bpf_unwind_resume() calls are added, so a frame like fail() gets one too, which is never reached.

The exit is put in by patching the last insn into itself followed by the exit. The patch leaves most of the last insn's aux data on the exit, so it is put back on the last insn, its nospec mark and kfunc data included.

---
 kernel/bpf/fixups.c | 56 ++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 53 insertions(+), 3 deletions(-)
kernel/bpf/fixups.c +53 -3
index 1970069d6957..a73c52b543e9 100644
@@ -727,13 +727,63 @@ static void keep_funcs_with_addr_taken(struct bpf_verifier_env *env)
 	}
 }
 
+/*
+ * The walk sends a frame that a callee unwinds through to its epilogue, which
+ * the x86 JIT emits at the first exit, so keep one. A frame that only unwinds
+ * may have none: put one after its last insn.
+ */
+static int keep_subprog_exits(struct bpf_verifier_env *env)
+{
+	struct bpf_insn patch[2] = { {}, BPF_EXIT_INSN() };
+	struct bpf_prog *new_prog;
+	u32 i, j;
+
+	for (i = 0; i < env->subprog_cnt; i++) {
+		u32 start, end;
+		bool found = false;
+
+		if (!env->subprog_info[i].might_unwind)
+			continue;
+		start = env->subprog_info[i].start;
+		if (!env->insn_aux_data[start].seen)
+			continue;
+
+		end = env->subprog_info[i + 1].start;
+		for (j = end; j-- > start; ) {
+			if (env->prog->insnsi[j].code == (BPF_JMP | BPF_EXIT)) {
+				found = true;
+				break;
+			}
+		}
+		if (!found) {
+			struct bpf_insn_aux_data aux = env->insn_aux_data[end - 1];
+
+			patch[0] = env->prog->insnsi[end - 1];
+			new_prog = bpf_patch_insn_data(env, end - 1, patch, 2);
+			if (!new_prog)
+				return -ENOMEM;
+			env->prog = new_prog;
+			env->insn_aux_data[end - 1] = aux;
+			memset(&env->insn_aux_data[end], 0, sizeof(aux));
+			j = end;
+		}
+		env->insn_aux_data[j].seen = env->pass_cnt;
+	}
+	return 0;
+}
+
 int bpf_opt_remove_dead_code(struct bpf_verifier_env *env)
 {
-	struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
-	int insn_cnt = env->prog->len;
-	int i, err;
+	struct bpf_insn_aux_data *aux_data;
+	int i, insn_cnt, err;
 
 	keep_funcs_with_addr_taken(env);
+	err = keep_subprog_exits(env);
+	if (err)
+		return err;
+
+	aux_data = env->insn_aux_data;
+	insn_cnt = env->prog->len;
 
 	for (i = 0; i < insn_cnt; i++) {
 		int j;
Signature · 1 line
2.53.0-Meta
author reviewer
#16 lore raw

PATCH v12 12/25 bpf: Allow a subprog to end with a call that does not return

In reply to Yonghong Song

A compiler emits nothing after a call it knows does not return: a cleanup pad's call to _Unwind_Resume, which libbpf resolves to bpf_unwind_resume(), or a call to a Rust function that returns ! because it unwinds. Built with panic=unwind, rustc gives for example

        fail:
                call bpf_unwind
        L0:     goto L0
        check:
                if w1 > 0x7 goto L0
                w0 = w1
                exit
        L0:     call fail
        prog:
                ...
                r1 = r6
                call _Unwind_Resume

and check_subprogs() refuses prog() and check(): "last insn is not an exit or jmp".

Let check_subprogs() accept a final call that never returns: to bpf_unwind() or bpf_unwind_resume(), or to a static subprog with neither an exit insn nor the hidden exit of a tail call or of a BPF_LD_[ABS|IND]. Any other final call is still refused, including one to a global subprog, which do_check() takes to return. bpf_unwind_resume() is lowered to 'r0 = 0; exit' as before.

Then, as keep_subprog_exits() does for a subprog with no exit, put an exit after each such call before the CFG walk. From then on the call simply falls through to an exit, so liveness, backtracking, fastcall and prepare_func_exit() need not know about it, and do_check() never reaches the exit. For a subprog that can unwind, keep_subprog_exits() keeps it as the epilogue the walk can send the frame to.

add_exits_after_final_calls() runs after add_kfuncs(), once insn array maps and the function pointers in read-only maps have recorded the insn offsets the patch adjusts, and before an offloaded program's per-insn state is set up. It also adjusts the call's own offset to a callee after it and the exit_idx of each later subprog. explored_states is sized by the program length, so it is now allocated after the exits are put in.

---
 kernel/bpf/verifier.c | 113 +++++++++++++++++++++++++++++++++++-------
 1 file changed, 94 insertions(+), 19 deletions(-)
kernel/bpf/verifier.c +94 -19
index 85542afcc369..804a6835c56f 100644
@@ -3317,6 +3317,54 @@ static void find_subprog_properties(struct bpf_verifier_env *env)
 	}
 }
 
+static int subprog_falls_through(struct bpf_verifier_env *env, int subprog, int idx)
+{
+	verbose(env, "last insn is not an exit or jmp\n");
+	bpf_diag_program_structure(
+		env, idx, "subprogram can fall through",
+		"End each subprogram with an exit or an explicit jump that keeps control flow inside the subprogram.",
+		"Subprogram %d reaches its last instruction %d without an exit or jump, so control could continue into the next subprogram.",
+		subprog, idx);
+	return -EINVAL;
+}
+
+/*
+ * Put an exit after a call that ends a subprog, so that nothing from the CFG
+ * walk on has to know a subprog can end that way. check_subprogs() only lets
+ * such a call be to a callee that never returns, so do_check() never reaches
+ * the exit.
+ */
+static int add_exits_after_final_calls(struct bpf_verifier_env *env)
+{
+	struct bpf_insn patch[2] = { {}, BPF_EXIT_INSN() };
+	struct bpf_prog *new_prog;
+	int i, j;
+
+	for (i = 0; i < env->subprog_cnt; i++) {
+		int last = env->subprog_info[i + 1].start - 1;
+		struct bpf_insn *insn = &env->prog->insnsi[last];
+
+		if (!bpf_pseudo_call(insn) && !bpf_pseudo_kfunc_call(insn))
+			continue;
+		patch[0] = *insn;
+		new_prog = bpf_patch_insn_data(env, last, patch, 2);
+		if (!new_prog)
+			return -ENOMEM;
+		env->prog = new_prog;
+		/* The patch moved the call's aux data, orig_idx included, to the exit. */
+		env->insn_aux_data[last].orig_idx = env->insn_aux_data[last + 1].orig_idx;
+		/* The patch does not adjust the call itself for a callee after it. */
+		insn = &new_prog->insnsi[last];
+		if (bpf_pseudo_call(insn) && insn->imm >= 0)
+			insn->imm++;
+		/* Nor does it move the exits check_subprogs() found after it. */
+		for (j = i + 1; j < env->subprog_cnt; j++)
+			if (env->subprog_info[j].exit_idx != U32_MAX)
+				env->subprog_info[j].exit_idx++;
+	}
+	return 0;
+}
+
 static int check_subprogs(struct bpf_verifier_env *env)
 {
 	int i, subprog_start, subprog_end, off, cur_subprog = 0;
@@ -3356,26 +3404,44 @@ static int check_subprogs(struct bpf_verifier_env *env)
 			/*
 			 * To avoid fall-through from one subprog into another,
 			 * the last insn of the subprog should be either exit
-			 * or unconditional jump back or bpf_throw call.
+			 * or unconditional jump back or a call that does not
+			 * return, checked below.
 			 */
 			if (code != (BPF_JMP | BPF_EXIT) &&
 			    code != (BPF_JMP32 | BPF_JA) &&
 			    code != (BPF_JMP | BPF_JA) &&
-			    !insn_is_gotox(&insn[i])) {
-				verbose(env, "last insn is not an exit or jmp\n");
-				bpf_diag_program_structure(
-					env, i, "subprogram can fall through",
-					"End each subprogram with an exit or an explicit jump that keeps control flow inside the subprogram.",
-					"Subprogram %d reaches its last instruction %d without an exit or jump, so control could continue into the next subprogram.",
-					cur_subprog, i);
-				return -EINVAL;
-			}
+			    !insn_is_gotox(&insn[i]) &&
+			    !bpf_pseudo_call(&insn[i]) &&
+			    !bpf_pseudo_kfunc_call(&insn[i]))
+				return subprog_falls_through(env, cur_subprog, i);
 			subprog_start = subprog_end;
 			cur_subprog++;
 			if (cur_subprog < env->subprog_cnt)
 				subprog_end = subprog[cur_subprog + 1].start;
 		}
 	}
+
+	/*
+	 * A compiler ends a subprog with a call only if the callee never
+	 * returns. Refuse any other final call.
+	 */
+	for (i = 0; i < env->subprog_cnt; i++) {
+		int last = subprog[i + 1].start - 1, callee;
+
+		if (bpf_pseudo_kfunc_call(&insn[last])) {
+			if (!bpf_is_unwind_kfunc(&insn[last]) &&
+			    !bpf_is_unwind_resume_kfunc(&insn[last]))
+				return subprog_falls_through(env, i, last);
+			continue;
+		}
+		if (!bpf_pseudo_call(&insn[last]))
+			continue;
+		callee = bpf_find_subprog(env, last + insn[last].imm + 1);
+		if (bpf_subprog_is_global(env, callee) ||
+		    subprog[callee].exit_idx != U32_MAX ||
+		    subprog[callee].has_tail_call || subprog[callee].has_ld_abs)
+			return subprog_falls_through(env, i, last);
+	}
 	return 0;
 }
 
@@ -23602,15 +23668,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	env->test_reg_invariants = attr->prog_flags & BPF_F_TEST_REG_INVARIANTS;
 	env->arena_scalar = attr->prog_flags & BPF_F_ARENA_SCALAR;
 
-	env->explored_states = kvzalloc_objs(struct list_head,
-					     state_htab_size(env),
-					     GFP_KERNEL_ACCOUNT);
-	ret = -ENOMEM;
-	if (!env->explored_states)
-		goto skip_full_check;
-
-	for (i = 0; i < state_htab_size(env); i++)
-		INIT_LIST_HEAD(&env->explored_states[i]);
 	INIT_LIST_HEAD(&env->free_list);
 
 	/* Prepare BTF and func_info needed to discover all subprograms. */
@@ -23661,6 +23718,24 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/*
+	 * After everything above has recorded insn offsets, so the patch
+	 * adjusts them, and before an offloaded program's per-insn state.
+	 */
+	ret = add_exits_after_final_calls(env);
+	if (ret < 0)
+		goto skip_full_check;
+
+	env->explored_states = kvzalloc_objs(struct list_head,
+					     state_htab_size(env),
+					     GFP_KERNEL_ACCOUNT);
+	ret = -ENOMEM;
+	if (!env->explored_states)
+		goto skip_full_check;
+
+	for (i = 0; i < state_htab_size(env); i++)
+		INIT_LIST_HEAD(&env->explored_states[i]);
+
 	if (bpf_prog_is_offloaded(env->prog->aux)) {
 		ret = bpf_prog_offload_verifier_prep(env->prog);
 		if (ret)
Signature · 1 line
2.53.0-Meta
#17 lore raw

Re: [PATCH bpf-next v12 12/25] bpf: Allow a subprog to end with a call that does not return

TL;DR: Inserting exits after final calls may shift verifier log instruction indices for newly accepted programs, which could make libbpf's CO-RE relocation lookup miss or name the wrong field. Was…
Quoted 1 line

bpf: Allow a subprog to end with a call that does not return

TL;DR: Inserting exits after final calls may shift verifier log instruction indices for newly accepted programs, which could make libbpf's CO-RE relocation lookup miss or name the wrong field. Was this intended?

Quoted 3 lines
kernel/bpf/verifier.c +0 -0

[ ... ]

Quoted 12 lines
kernel/bpf/verifier.c +8 -0
@@ -3317,6 +3317,54 @@ static void find_subprog_properties(struct bpf_verifier_env *env)
 	}
 }
 
+/*
+ * Put an exit after a call that ends a subprog, so that nothing from the CFG
+ * walk on has to know a subprog can end that way. check_subprogs() only lets
+ * such a call be to a callee that never returns, so do_check() never reaches
+ * the exit.
+ */
+static int add_exits_after_final_calls(struct bpf_verifier_env *env)
+{

[ ... ]

Quoted 7 lines
kernel/bpf/verifier.c +7 -0
+		patch[0] = *insn;
+		new_prog = bpf_patch_insn_data(env, last, patch, 2);
+		if (!new_prog)
+			return -ENOMEM;
+		env->prog = new_prog;
+		/* The patch moved the call's aux data, orig_idx included, to the exit. */
+		env->insn_aux_data[last].orig_idx = env->insn_aux_data[last + 1].orig_idx;

[ ... ]

Quoted 3 lines
kernel/bpf/verifier.c +3 -0
+	}
+	return 0;
+}

[ ... ]

kernel/bpf/verifier.c int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, · line 23728

Quoted 12 lines
kernel/bpf/verifier.c +8 -0
@@ -23661,6 +23718,24 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/*
+	 * After everything above has recorded insn offsets, so the patch
+	 * adjusts them, and before an offloaded program's per-insn state.
+	 */
+	ret = add_exits_after_final_calls(env);
+	if (ret < 0)
+		goto skip_full_check;
+

Does inserting these exits before bpf_check_cfg() and do_check() shift the instruction indices that the verifier log reports?

Until now no pass changed the instruction count before verification, so the indices in the log matched the instructions user space loaded. With this patch, every insn behind a subprog that ends with a call moves down by one for each exit added in front of it. The CFG walk and do_check() print env->insn_idx, so every message about those insns now carries an index that is not in the loaded program. insn_aux_data[].orig_idx exists for this kind of mismatch, but only a few messages use it.

This has a concrete effect on libbpf. fixup_log_failed_core_relo() in tools/lib/bpf/libbpf.c reads the index from the log and uses it to find the relocation:

    if (sscanf(line1, "%d: (%*d) call unknown#195896080\n", &insn_idx) != 1)
        return;
    relo = find_relo_core(prog, insn_idx);

find_relo_core() needs an exact relo->insn_idx match. Take a program whose main subprog ends with "call _Unwind_Resume" (the prog() example in the commit message) and that has a poisoned CO-RE relocation at index N in a later subprog. The kernel now logs that instruction as N+1, so libbpf either finds no relocation and leaves the raw "unknown#195896080" message, or picks up a different CO-RE relocation sitting at N+1 and names the wrong field.

The cleanup-record messages are affected as well. "resume at insn %d is not in a landing pad" and "insn %u ... is in a landing pad" now print shifted indices, while bpf_exc_check_info() reports the user's own offsets.

Only programs that this patch newly accepts are affected, since add_exits_after_final_calls() patches nothing otherwise. The commit message says the exits are put in "before the CFG walk" but does not mention the effect on the log indices.

Was this intended? Would it make sense to report orig_idx in the do_check() messages, or to handle a final call without inserting instructions until after verification?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/38997507726

author reviewer
#18 lore raw

Re: [PATCH bpf-next v12 12/25] bpf: Allow a subprog to end with a call that does not return

Quoted 42 lines · On 10/16/26 8:35 PM, [email protected] wrote:
Quoted 1 line

bpf: Allow a subprog to end with a call that does not return

TL;DR: Inserting exits after final calls may shift verifier log instruction indices for newly accepted programs, which could make libbpf's CO-RE relocation lookup miss or name the wrong field. Was this intended?

Quoted 3 lines
kernel/bpf/verifier.c +0 -0

[ ... ]

Quoted 12 lines
kernel/bpf/verifier.c +8 -0
@@ -3317,6 +3317,54 @@ static void find_subprog_properties(struct bpf_verifier_env *env)
  	}
  }
 
+/*
+ * Put an exit after a call that ends a subprog, so that nothing from the CFG
+ * walk on has to know a subprog can end that way. check_subprogs() only lets
+ * such a call be to a callee that never returns, so do_check() never reaches
+ * the exit.
+ */
+static int add_exits_after_final_calls(struct bpf_verifier_env *env)
+{

[ ... ]

Quoted 7 lines
kernel/bpf/verifier.c +7 -0
+		patch[0] = *insn;
+		new_prog = bpf_patch_insn_data(env, last, patch, 2);
+		if (!new_prog)
+			return -ENOMEM;
+		env->prog = new_prog;
+		/* The patch moved the call's aux data, orig_idx included, to the exit. */
+		env->insn_aux_data[last].orig_idx = env->insn_aux_data[last + 1].orig_idx;

[ ... ]

Quoted 3 lines
kernel/bpf/verifier.c +3 -0
+	}
+	return 0;
+}

[ ... ]

Quoted 4 lines
kernel/bpf/verifier.c +0 -0
@@ -23661,6 +23718,24 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
  	if (ret < 0)
  		goto skip_full_check;
 
Quoted 10 lines

kernel/bpf/verifier.c int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, · line 23728

Quoted 8 lines
kernel/bpf/verifier.c +8 -0
+	/*
+	 * After everything above has recorded insn offsets, so the patch
+	 * adjusts them, and before an offloaded program's per-insn state.
+	 */
+	ret = add_exits_after_final_calls(env);
+	if (ret < 0)
+		goto skip_full_check;
+

Does inserting these exits before bpf_check_cfg() and do_check() shift the instruction indices that the verifier log reports?

I think this is indeed an issue, esp. for verifier log reporting.

For example, the original function:
0: call foo
1: call bpf_unwind
2: call bpf_unwind_resume

After add_exits_after_final_calls, we will have

   idx  orig_idx
   0    0         call foo
   1    1         bpf_unwind
   2    1         exit
   3    2         call bpf_unwind_resume
   4    2         exit

When verifier trying to report error message, we cannot use 'idx' any more, we have to use orig_idx. For example, we have
verbose_linfo(env, env->insn_idx, "; ");
...
verbose(env, "%d: ", env->insn_idx);

here env->insn_idx refers to 'idx' in the above,
but they do not correspond to the original idx.

The actual one may be replacing env->insn_idx to corresponding orig_idx.

We could have a helper to map idx->orig_idx. This should work. Do we have a better solution for this?

Quoted 44 lines

Until now no pass changed the instruction count before verification, so the indices in the log matched the instructions user space loaded. With this patch, every insn behind a subprog that ends with a call moves down by one for each exit added in front of it. The CFG walk and do_check() print env->insn_idx, so every message about those insns now carries an index that is not in the loaded program. insn_aux_data[].orig_idx exists for this kind of mismatch, but only a few messages use it.

This has a concrete effect on libbpf. fixup_log_failed_core_relo() in tools/lib/bpf/libbpf.c reads the index from the log and uses it to find the relocation:

     if (sscanf(line1, "%d: (%*d) call unknown#195896080\n", &insn_idx) != 1)
         return;
     relo = find_relo_core(prog, insn_idx);

find_relo_core() needs an exact relo->insn_idx match. Take a program whose main subprog ends with "call _Unwind_Resume" (the prog() example in the commit message) and that has a poisoned CO-RE relocation at index N in a later subprog. The kernel now logs that instruction as N+1, so libbpf either finds no relocation and leaves the raw "unknown#195896080" message, or picks up a different CO-RE relocation sitting at N+1 and names the wrong field.

The cleanup-record messages are affected as well. "resume at insn %d is not in a landing pad" and "insn %u ... is in a landing pad" now print shifted indices, while bpf_exc_check_info() reports the user's own offsets.

Only programs that this patch newly accepts are affected, since add_exits_after_final_calls() patches nothing otherwise. The commit message says the exits are put in "before the CFG walk" but does not mention the effect on the log indices.

Was this intended? Would it make sense to report orig_idx in the do_check() messages, or to handle a final call without inserting instructions until after verification?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/38997507726

author reviewer
#19 lore raw

PATCH v12 13/25 bpf: Dispatch cleanup pads by rewriting return addresses

In reply to Yonghong Song

bpf_unwind() now walks the BPF frames through arch_bpf_stack_walk_ra() and rewrites the saved return address of each frame above the one that called it: to the pad where a record covers the call, else to the frame's epilogue. The walk stops after the main function. For the verifier patch's example, main -> A -> B -> C with only A's call to B covered:

  slot                    points into               rewritten to
  ----------------------  ------------------------  ---------------------
  bpf_unwind()'s return   C, after its unwind call  left alone
  C's return              B, after 'call C'         B's epilogue
  B's return              A, after 'call B'         P, A's pad
  A's return              main, after 'call A'      main's epilogue
  main's return           the kernel                left alone

Each frame then just returns: C through the 'r0 = 0; exit' after its bpf_unwind(), B and main through their epilogues, A through its pad P and P's resume.

The walk finds the frames in the calling program, not with bpf_prog_ksym_find(). A running instance holds no reference to its program, so user space can drop the last one, by closing the program's fds and links, while the instance still runs: the kallsyms entries go at once, and only freeing the program waits for the RCU or RCU Tasks Trace grace period. An unwind in between, say in a sleepable program blocked in bpf_copy_from_user(), would find none of its frames, so no pad would run. So bpf_unwind() takes the program's aux as a KF_IMPLICIT_ARGS argument, which keeps its BTF prototype void(void), and matches each return address against aux->func[], or the program itself. To get the implicit argument loaded, adjust_insn_aux_data() moves arg_prog, as it moves cleanup_pad, to the slot the original instruction kept: the unwind call stays first in its patch.

Both kfuncs become callable here. bpf_unwind() is notrace and NOKPROBE, so that only plain kernel frames sit between it and the frame that called it, and the walk does not have to get through a trampoline or a kretprobe return on its way there.

---
 kernel/bpf/fixups.c  |  2 ++
 kernel/bpf/helpers.c | 67 +++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 68 insertions(+), 1 deletion(-)
kernel/bpf/fixups.c +2 -0
index a73c52b543e9..6c2e5e544124 100644
@@ -247,6 +247,8 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 			data[off + cnt - 1].non_stack_access = false;
 			data[i].cleanup_pad = data[off + cnt - 1].cleanup_pad;
 			data[off + cnt - 1].cleanup_pad = 0;
+			data[i].arg_prog = data[off + cnt - 1].arg_prog;
+			data[off + cnt - 1].arg_prog = 0;
 		} else if (bpf_is_mem_insn(insn + i)) {
 			data[i].non_stack_access = true;
 		}
kernel/bpf/helpers.c +66 -1

89 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
author reviewer
#20 lore raw

PATCH v12 14/25 bpf: Refuse a trampoline that calls a subprog that can unwind

In reply to Yonghong Song

bpf_unwind() walks up the stack rewriting return addresses. It expects every frame up to the main function to be one of the program's own BPF functions, and stops at the first frame that is not. A trampoline attached to a subprog with fexit, fmod_ret or fsession is such a frame:
it calls the subprog, so it sits between the subprog and its caller:

main -> A -> trampoline -> B -> C C calls bpf_unwind()

The walk rewrites the return into B but stops at the trampoline, so B returns through it to A after its call, instead of to A's pad or
epilogue: a path the verifier never walked for an unwind.

Refuse such an attachment to a subprog marked might_unwind, now copied into each function's aux. Nothing else puts a frame between two of a program's frames: fentry leaves none, a trampoline on main is above where the walk stops, freplace is a program of its own whose unwind ends as a normal return to its caller, a callback that can unwind is refused, kprobes and fgraph cannot hook JIT code, and a tail call replaces a frame.

---
 include/linux/bpf.h   |  1 +
 kernel/bpf/fixups.c   |  1 +
 kernel/bpf/verifier.c | 17 +++++++++++++++++
 3 files changed, 19 insertions(+)
include/linux/bpf.h +1 -0
index 7f23f4efde01..1b3b6ee05c09 100644
@@ -1910,6 +1910,7 @@ struct bpf_prog_aux {
 	bool priv_stack_requested;
 	bool changes_pkt_data;
 	bool might_sleep;
+	bool might_unwind;
 	bool kprobe_write_ctx;
 	struct {
 		s32 keyring_serial;
kernel/bpf/fixups.c +1 -0
index 6c2e5e544124..dd29ead32cf3 100644
@@ -1494,6 +1494,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 		func[i]->aux->exception_cb = env->subprog_info[i].is_exception_cb;
 		func[i]->aux->changes_pkt_data = env->subprog_info[i].changes_pkt_data;
 		func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
+		func[i]->aux->might_unwind = env->subprog_info[i].might_unwind;
 		func[i]->aux->token = prog->aux->token;
 		if (!i)
 			func[i]->aux->exception_boundary = env->seen_exception;
kernel/bpf/verifier.c +17 -0
index 804a6835c56f..505ecb8de221 100644
@@ -22371,6 +22371,23 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
 				prog_extension ? "Extension" : "Tracing");
 			return -EINVAL;
 		}
+		/*
+		 * A trampoline that calls its target stays as a frame between a
+		 * subprog and its caller, and bpf_unwind() cannot walk past it:
+		 * the frames above it would carry on after their calls rather
+		 * than at their landing pads or epilogues. fentry leaves no
+		 * frame, and the main program's caller is above where the walk
+		 * stops.
+		 */
+		if (aux->func && subprog && aux->func[subprog]->aux->might_unwind &&
+		    (prog->expected_attach_type == BPF_TRACE_FEXIT ||
+		     prog->expected_attach_type == BPF_MODIFY_RETURN ||
+		     prog->expected_attach_type == BPF_TRACE_FSESSION)) {
+			bpf_log(log,
+				"Cannot attach fexit, fmod_ret or fsession to %s, which can unwind\n",
+				tname);
+			return -EINVAL;
+		}
 		conservative = aux->func_info_aux[subprog].unreliable;
 		if (prog_extension) {
 			if (conservative) {
Signature · 1 line
2.53.0-Meta
author reviewer
#21 lore raw

PATCH v12 15/25 bpf, x86: Dispatch exception cleanup pads at run time

In reply to Yonghong Song

Add what bpf_unwind() relies on to send each frame of a program to its landing pad or its epilogue on x86-64:

- arch_bpf_stack_walk_ra(): the ORC walk, handing out each frame's return address, state.ip, and the slot it was read from, unwind_get_return_address_ptr(). A BPF frame is unwound through its frame pointer, which the JIT always sets up.
- aux->epilogue_ip: the address of the one epilogue the JIT emits per function, at the offset it keeps as ctx->cleanup_addr.
- Once the image is final, the JIT fills in the native cleanup table.
- bpf_jit_supports_cleanup_pads() says yes with CONFIG_UNWINDER_ORC, as bpf_jit_supports_exceptions() does.

A pad needs no ENDBR: it is only reached as a return address.

---
 arch/x86/net/bpf_jit_comp.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
arch/x86/net/bpf_jit_comp.c +36 -0
index 11a4e4f87aac..e0e39661bce8 100644
@@ -15,6 +15,7 @@
 #include <linux/memory.h>
 #include <linux/sort.h>
 #include <linux/execmem.h>
+#include <linux/kprobes.h>
 #include <asm/extable.h>
 #include <asm/ftrace.h>
 #include <asm/set_memory.h>
@@ -3779,6 +3780,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			seen_exit = true;
 			/* Update cleanup_addr */
 			ctx->cleanup_addr = proglen;
+			/* Where an unwind sends a frame with no pad. */
+			bpf_prog->aux->epilogue_ip = (u64)image + proglen;
 			if (bpf_prog_was_classic(bpf_prog) &&
 			    !ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN)) {
 				if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
@@ -4976,6 +4979,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
 		 */
 		bpf_prog_update_insn_ptrs(prog, addrs, image);
 
+		/*
+		 * Same mapping, consumed by the bpf_unwind() walk:
+		 * turn the cleanup records into native address ranges now
+		 * that the image is final.
+		 */
+		bpf_exc_fill_native_ranges(prog, addrs, image);
+
 		/*
 		 * ctx.prog_offset is used when CFI preambles put code *before*
 		 * the function. See emit_cfi(). For FineIBT specifically this code
@@ -5114,6 +5124,11 @@ bool bpf_jit_supports_exceptions(void)
 	return IS_ENABLED(CONFIG_UNWINDER_ORC);
 }
 
+bool bpf_jit_supports_cleanup_pads(void)
+{
+	return IS_ENABLED(CONFIG_UNWINDER_ORC);
+}
+
 bool bpf_jit_supports_private_stack(void)
 {
 	return true;
@@ -5135,6 +5150,27 @@ void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp
 #endif
 }
 
+notrace void arch_bpf_stack_walk_ra(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp,
+							u64 *ra),
+				    void *cookie)
+{
+#if defined(CONFIG_UNWINDER_ORC)
+	struct unwind_state state;
+	unsigned long addr, *ra;
+
+	for (unwind_start(&state, current, NULL, NULL); !unwind_done(&state);
+	     unwind_next_frame(&state)) {
+		addr = state.ip;
+		ra = unwind_get_return_address_ptr(&state);
+		if (!ra)
+			break;
+		if (!consume_fn(cookie, (u64)addr, (u64)state.sp, (u64)state.bp, (u64 *)ra))
+			break;
+	}
+#endif
+}
+NOKPROBE_SYMBOL(arch_bpf_stack_walk_ra);
+
 void bpf_arch_poke_desc_update(struct bpf_jit_poke_descriptor *poke,
 			       struct bpf_prog *new, struct bpf_prog *old)
 {
Signature · 1 line
2.53.0-Meta
author reviewer
#22 lore raw

PATCH v12 16/25 bpf, arm64: Dispatch exception cleanup pads at run time

In reply to Yonghong Song

Add what bpf_unwind() relies on to send each frame of a program to its landing pad or its epilogue on arm64, as the previous patch does for x86-64:

- arch_bpf_stack_walk_ra(): walks the frame records with the kernel unwinder, handing each return address to the consumer and storing a new one back into the record it was read from. The first frame, whose return into bpf_unwind() comes from the walk's own record, is skipped.
- With CONFIG_ARM64_PTR_AUTH_KERNEL and a CPU that supports address authentication, the new address is signed the way the BPF prologue's PACIASP signs the link register, key IA with the record + 16 as modifier, using PACIA1716 so that no assembler support for pointer authentication is needed.
- Once the image is final, the JIT fills in the native cleanup table and records each function's epilogue address in aux->epilogue_ip, where an unwind sends a frame with no pad.
- bpf_jit_supports_cleanup_pads() says yes, also with a shadow call
stack: only JITed frames' records are written, and JITed code keeps no x18 copy of its return address.

A pad needs no BTI: it is only reached as a return address.

---
 arch/arm64/kernel/stacktrace.c | 77 ++++++++++++++++++++++++++++++++++
 arch/arm64/net/bpf_jit_comp.c  | 16 +++++++
 2 files changed, 93 insertions(+)
arch/arm64/kernel/stacktrace.c +77 -0
index 3ebcf8c53fb0..6ff485280da5 100644
@@ -445,6 +445,83 @@ noinline noinstr void arch_bpf_stack_walk(bool (*consume_entry)(void *cookie, u6
 	kunwind_stack_walk(arch_bpf_unwind_consume_entry, &data, current, NULL);
 }
 
+struct bpf_unwind_ra_consume_entry_data {
+	bool (*consume_entry)(void *cookie, u64 ip, u64 sp, u64 fp, u64 *ra);
+	void *cookie;
+	unsigned long record;
+	bool seen_first;
+};
+
+/* PACIA1716 (hint #8): sign x17 with the modifier in x16, under key IA. */
+static u64 bpf_unwind_sign_ra(u64 ra, u64 modifier)
+{
+	register u64 x17 asm("x17") = ra;
+	register u64 x16 asm("x16") = modifier;
+
+	asm volatile(ARM64_ASM_PREAMBLE
+		     "	hint	#8\n"
+		     : "+r" (x17) : "r" (x16));
+	return x17;
+}
+
+/* PACIASP's modifier is the entry sp, record + 16 for a BPF prologue. */
+static void bpf_unwind_store_ra(unsigned long record, u64 ra)
+{
+	struct frame_record *rec = (struct frame_record *)record;
+
+	/*
+	 * Whether the slot holds a signed address is a property of the build,
+	 * not one to be read off the value: a PAC can come out equal to the
+	 * bits stripping puts back, and a signed address would then be taken
+	 * for an unsigned one. What signs is CONFIG_ARM64_PTR_AUTH_KERNEL --
+	 * the prologue here, and -mbranch-protection for everything the
+	 * compiler emits.
+	 */
+	if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL) &&
+	    system_supports_address_auth())
+		ra = bpf_unwind_sign_ra(ra, record + sizeof(struct frame_record));
+	WRITE_ONCE(rec->lr, ra);
+}
+
+static bool
+arch_bpf_unwind_ra_consume_entry(const struct kunwind_state *state, void *cookie)
+{
+	struct bpf_unwind_ra_consume_entry_data *data = cookie;
+	unsigned long record = data->record;
+	bool seen_first = data->seen_first;
+	u64 ra = state->common.pc;
+	bool cont;
+
+	/* The record this frame's return address will have come out of. */
+	data->record = state->common.fp;
+	data->seen_first = true;
+
+	/*
+	 * The first pc returns into bpf_unwind(), from this walk's own frame
+	 * record: not a BPF frame, and not one to redirect.
+	 */
+	if (!seen_first)
+		return true;
+	/* A consumer that stops still gets to redirect the frame it stopped on. */
+	cont = data->consume_entry(data->cookie, state->common.pc, 0,
+				   state->common.fp, &ra);
+	if (ra != state->common.pc)
+		bpf_unwind_store_ra(record, ra);
+	return cont;
+}
+
+noinline noinstr void arch_bpf_stack_walk_ra(bool (*consume_entry)(void *cookie, u64 ip, u64 sp,
+								   u64 fp, u64 *ra),
+					     void *cookie)
+{
+	struct bpf_unwind_ra_consume_entry_data data = {
+		.consume_entry = consume_entry,
+		.cookie = cookie,
+	};
+
+	kunwind_stack_walk(arch_bpf_unwind_ra_consume_entry, &data, current, NULL);
+}
+
 static const char *state_source_string(const struct kunwind_state *state)
 {
 	switch (state->source) {
arch/arm64/net/bpf_jit_comp.c +16 -0
index 506c4201fec8..c8c0328b3219 100644
@@ -2422,6 +2422,17 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
 		 * reasons, expects to point to the next instruction)
 		 */
 		bpf_prog_update_insn_ptrs(prog, ctx.offset, ctx.ro_image);
+
+		/*
+		 * Same byte offsets, consumed by the bpf_unwind() walk:
+		 * turn the cleanup records into native address ranges now that
+		 * the image is final.
+		 */
+		bpf_exc_fill_native_ranges(prog, ctx.offset, ctx.ro_image);
+
+		/* Where an unwind sends a frame with no pad. */
+		prog->aux->epilogue_ip = (u64)ctx.ro_image +
+					 ctx.epilogue_offset * AARCH64_INSN_SIZE;
 out_off:
 		if (!ro_header && priv_stack_ptr) {
 			free_percpu(priv_stack_ptr);
@@ -3415,6 +3426,11 @@ bool bpf_jit_supports_exceptions(void)
 	return true;
 }
 
+bool bpf_jit_supports_cleanup_pads(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_arena(void)
 {
 	return true;
Signature · 1 line
2.53.0-Meta
author reviewer
#23 lore raw

PATCH v12 17/25 libbpf: Resolve the compiler's _Unwind_Resume to the kernel's kfunc

In reply to Yonghong Song

LLVM ends a cleanup landing pad with a call to _Unwind_Resume, the unwind ABI's name for carrying an unwind on. The kernel's kfunc is
bpf_unwind_resume: _Unwind_Resume in the kernel's own symbol table, for a function whose body never runs, would only confuse.

The compiler emits the call but no declaration, and libbpf refuses an extern it has no BTF for, so the program declares it itself (extern void _Unwind_Resume(void *) __ksym;), as a language runtime has to. libbpf only translates the name, both where it resolves the kfunc itself and where it records the name for a light skeleton's loader to resolve.

---
 tools/lib/bpf/libbpf.c | 56 ++++++++++++++++++++++++++++++++----------
 1 file changed, 43 insertions(+), 13 deletions(-)
tools/lib/bpf/libbpf.c +43 -13
index cb09ded90773..b3cfd7c58776 100644
@@ -9187,6 +9187,23 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s
 	}
 }
 
+/*
+ * LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the
+ * base unwind ABI's entry point for carrying an unwind on once a frame's
+ * cleanups have run. The kernel knows it as bpf_unwind_resume. Any other
+ * extern, and a variable of that name, is looked up by @name unchanged.
+ */
+static const char *kern_extern_name(const struct bpf_object *obj,
+				    const struct extern_desc *ext, const char *name)
+{
+	const char *essent = ext->essent_name ?: ext->name;
+
+	if (!btf_is_func(btf__type_by_id(obj->btf, ext->btf_id)) ||
+	    strcmp(essent, "_Unwind_Resume"))
+		return name;
+	return "bpf_unwind_resume";
+}
+
 static int bpf_program_record_relos(struct bpf_program *prog)
 {
 	struct bpf_object *obj = prog->obj;
@@ -9195,6 +9212,7 @@ static int bpf_program_record_relos(struct bpf_program *prog)
 	for (i = 0; i < prog->nr_reloc; i++) {
 		struct reloc_desc *relo = &prog->reloc_desc[i];
 		struct extern_desc *ext = &obj->externs[relo->ext_idx];
+		const char *name;
 		int kind;
 
 		switch (relo->type) {
@@ -9203,14 +9221,14 @@ static int bpf_program_record_relos(struct bpf_program *prog)
 				continue;
 			kind = btf_is_var(btf__type_by_id(obj->btf, ext->btf_id)) ?
 				BTF_KIND_VAR : BTF_KIND_FUNC;
-			bpf_gen__record_extern(obj->gen_loader, ext->name,
-					       ext->is_weak, !ext->ksym.type_id,
-					       true, kind, relo->insn_idx);
+			name = kern_extern_name(obj, ext, ext->name);
+			bpf_gen__record_extern(obj->gen_loader, name, ext->is_weak,
+					       !ext->ksym.type_id, true, kind, relo->insn_idx);
 			break;
 		case RELO_EXTERN_CALL:
-			bpf_gen__record_extern(obj->gen_loader, ext->name,
-					       ext->is_weak, false, false, BTF_KIND_FUNC,
-					       relo->insn_idx);
+			name = kern_extern_name(obj, ext, ext->name);
+			bpf_gen__record_extern(obj->gen_loader, name, ext->is_weak, false,
+					       false, BTF_KIND_FUNC, relo->insn_idx);
 			break;
 		case RELO_CORE: {
 			struct bpf_core_relo cr = {
@@ -9684,17 +9702,24 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,
 	struct module_btf *mod_btf = NULL;
 	const struct btf_type *kern_func;
 	struct btf *kern_btf = NULL;
+	const char *local_name, *kern_name;
 	int ret;
 
 	local_func_proto_id = ext->ksym.type_id;
 
-	kfunc_id = find_ksym_btf_id(obj, ext->essent_name ?: ext->name, BTF_KIND_FUNC, &kern_btf,
-				    &mod_btf);
+	local_name = ext->essent_name ?: ext->name;
+	kern_name = kern_extern_name(obj, ext, local_name);
+
+	kfunc_id = find_ksym_btf_id(obj, kern_name, BTF_KIND_FUNC, &kern_btf, &mod_btf);
 	if (kfunc_id < 0) {
 		if (kfunc_id == -ESRCH && ext->is_weak)
 			return 0;
-		pr_warn("extern (func ksym) '%s': not found in kernel or module BTFs\n",
-			ext->name);
+		if (kern_name != local_name)
+			pr_warn("extern (func ksym) '%s' ('%s' in the kernel): not found in kernel or module BTFs\n",
+				ext->name, kern_name);
+		else
+			pr_warn("extern (func ksym) '%s': not found in kernel or module BTFs\n",
+				ext->name);
 		return kfunc_id;
 	}
 
@@ -9707,9 +9732,14 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,
 		if (ext->is_weak)
 			return 0;
 
-		pr_warn("extern (func ksym) '%s': func_proto [%d] incompatible with %s [%d]\n",
-			ext->name, local_func_proto_id,
-			mod_btf ? mod_btf->name : "vmlinux", kfunc_proto_id);
+		if (kern_name != local_name)
+			pr_warn("extern (func ksym) '%s': func_proto [%d] incompatible with '%s' in %s [%d]\n",
+				ext->name, local_func_proto_id, kern_name,
+				mod_btf ? mod_btf->name : "vmlinux", kfunc_proto_id);
+		else
+			pr_warn("extern (func ksym) '%s': func_proto [%d] incompatible with %s [%d]\n",
+				ext->name, local_func_proto_id,
+				mod_btf ? mod_btf->name : "vmlinux", kfunc_proto_id);
 		return -EINVAL;
 	}
 
Signature · 1 line
2.53.0-Meta
author reviewer
#24 lore raw

PATCH v12 18/25 libbpf: Add cleanup_info to bpf_prog_load_opts

In reply to Yonghong Song

Let a caller hand the kernel an exception cleanup table:
bpf_prog_load_opts grows cleanup_info_cnt, cleanup_info and cleanup_info_rec_size, which bpf_prog_load() passes on to BPF_PROG_LOAD. cleanup_info_cnt comes first to fill the hole after fd_array_cnt.

---
 tools/lib/bpf/bpf.c |  6 +++++-
 tools/lib/bpf/bpf.h | 10 +++++++++-
 2 files changed, 14 insertions(+), 2 deletions(-)
tools/lib/bpf/bpf.c +5 -1
index b49822d212ae..b4031f62bdee 100644
@@ -295,7 +295,7 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
 		  const struct bpf_insn *insns, size_t insn_cnt,
 		  struct bpf_prog_load_opts *opts)
 {
-	const size_t attr_sz = offsetofend(union bpf_attr, keyring_id);
+	const size_t attr_sz = offsetofend(union bpf_attr, cleanup_info_cnt);
 	void *finfo = NULL, *linfo = NULL;
 	const char *func_info, *line_info;
 	__u32 log_size, log_level, attach_prog_fd, attach_btf_obj_fd;
@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
 	attr.fd_array = ptr_to_u64(OPTS_GET(opts, fd_array, NULL));
 	attr.fd_array_cnt = OPTS_GET(opts, fd_array_cnt, 0);
 
+	attr.cleanup_info = ptr_to_u64(OPTS_GET(opts, cleanup_info, NULL));
+	attr.cleanup_info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);
+	attr.cleanup_info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+
 	if (log_level) {
 		attr.log_buf = ptr_to_u64(log_buf);
 		attr.log_size = log_size;
tools/lib/bpf/bpf.h +9 -1
index 826d9cc9ab65..64699e0e4e1a 100644
@@ -128,9 +128,17 @@ struct bpf_prog_load_opts {
 
 	/* if set, provides the length of fd_array */
 	__u32 fd_array_cnt;
+
+	/* exception cleanup table: cleanup_info_cnt records of
+	 * cleanup_info_rec_size bytes each (struct bpf_cleanup_info),
+	 * holding instruction indices into insns
+	 */
+	__u32 cleanup_info_cnt;
+	const void *cleanup_info;
+	__u32 cleanup_info_rec_size;
 	size_t :0;
 };
-#define bpf_prog_load_opts__last_field fd_array_cnt
+#define bpf_prog_load_opts__last_field cleanup_info_rec_size
 
 LIBBPF_API int bpf_prog_load(enum bpf_prog_type prog_type,
 			     const char *prog_name, const char *license,
Signature · 1 line
2.53.0-Meta
#25 lore raw

Re: [PATCH bpf-next v12 18/25] libbpf: Add cleanup_info to bpf_prog_load_opts

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] libbpf: multi-line comment in bpf_prog_load_opts violates BPF comment style --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] libbpf: multi-line comment in bpf_prog_load_opts violates BPF comment style --

commit a97157c7935f8f71aa63285b4b87c0084fc0c397
Author: Yonghong Song <[email protected]>

libbpf: Add cleanup_info to bpf_prog_load_opts

This commit extends struct bpf_prog_load_opts and bpf_prog_load() with cleanup_info_cnt, cleanup_info, and cleanup_info_rec_size so callers can pass an exception cleanup table to BPF_PROG_LOAD.

tools/lib/bpf/bpf.h struct bpf_prog_load_opts { · line 135

Quoted 13 lines
tools/lib/bpf/bpf.h +5 -0
index 826d9cc9ab65d..64699e0e4e1a8 100644
@@ -128,9 +128,17 @@ struct bpf_prog_load_opts {
 
 	/* if set, provides the length of fd_array */
 	__u32 fd_array_cnt;
+
+	/* exception cleanup table: cleanup_info_cnt records of
+	 * cleanup_info_rec_size bytes each (struct bpf_cleanup_info),
+	 * holding instruction indices into insns
+	 */

[Severity: Low]
This isn't a bug, but in struct bpf_prog_load_opts in tools/lib/bpf/bpf.h, should this multi-line comment place the opening /* on its own line to match BPF comment style:

/*
* exception cleanup table: cleanup_info_cnt records of
* cleanup_info_rec_size bytes each (struct bpf_cleanup_info),
* holding instruction indices into insns
*/

Quoted 10 lines
tools/lib/bpf/bpf.h +4 -1
+	__u32 cleanup_info_cnt;
+	const void *cleanup_info;
+	__u32 cleanup_info_rec_size;
 	size_t :0;
 };
-#define bpf_prog_load_opts__last_field fd_array_cnt
+#define bpf_prog_load_opts__last_field cleanup_info_rec_size
 
 LIBBPF_API int bpf_prog_load(enum bpf_prog_type prog_type,
 			     const char *prog_name, const char *license,
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
author reviewer
#26 lore raw

PATCH v12 19/25 libbpf: Collect .bpf_cleanup records and pass them to the kernel

In reply to Yonghong Song

Parse the compiler-emitted .bpf_cleanup section and hand the table to BPF_PROG_LOAD:

- At open, each record field, a byte offset into a code section, is resolved through its .rel.bpf_cleanup relocation: R_BPF_64_NODYLD32 from LLVM or R_BPF_64_ABS32 from GNU as.
- At relocation, once a main program's subprogs are appended, the records in the main program and in those subprogs get final instruction indices and are sorted by begin_off, as the kernel wants. A record in code no function claims belongs to a weak function the static linker kept after overriding it, and is skipped, as that code's relocations are.
- At load, the table goes through the bpf_prog_load() options, from bpf_object_load_prog() and from bpf_program__clone(), the path veristat uses.

---
 tools/lib/bpf/libbpf.c          | 287 +++++++++++++++++++++++++++++++-
 tools/lib/bpf/libbpf_internal.h |   3 +
 2 files changed, 288 insertions(+), 2 deletions(-)
tools/lib/bpf/libbpf.c +285 -2
index b3cfd7c58776..d227dcc7ef0b 100644
@@ -516,6 +516,11 @@ struct bpf_program {
 	void *line_info;
 	__u32 line_info_rec_size;
 	__u32 line_info_cnt;
+
+	struct bpf_cleanup_info *cleanup_info;
+	__u32 cleanup_info_rec_size;
+	__u32 cleanup_info_cnt;
+
 	__u32 prog_flags;
 	__u8  hash[SHA256_DIGEST_LENGTH];
 
@@ -572,6 +577,7 @@ struct bpf_struct_ops {
  * are loaded with BPF_F_ARENA_SCALAR.
  */
 #define ARENA_DATA_SEC ".arena.data"
+#define CLEANUP_SEC ".bpf_cleanup"
 
 enum libbpf_map_type {
 	LIBBPF_MAP_UNSPEC,
@@ -710,6 +716,25 @@ struct elf_sec_desc {
 	Elf_Data *data;
 };
 
+#define CLEANUP_REC_FIELDS	(sizeof(struct bpf_cleanup_info) / sizeof(__u32))
+
+/* Index of each field of struct bpf_cleanup_info, read as an array of __u32. */
+enum {
+	CLEANUP_REC_BEGIN,
+	CLEANUP_REC_END,
+	CLEANUP_REC_PAD,
+};
+
+/*
+ * One (begin, end, landing_pad) triple from .bpf_cleanup, each field resolved
+ * from its relocation to a section and an instruction index within it. Final
+ * indices wait for subprogram placement, which differs per main program.
+ */
+struct cleanup_raw_rec {
+	int sec_idx[CLEANUP_REC_FIELDS];
+	size_t insn_idx[CLEANUP_REC_FIELDS];
+};
+
 struct elf_state {
 	int fd;
 	const void *obj_buf;
@@ -729,6 +754,8 @@ struct elf_state {
 	bool has_st_ops;
 	int arena_data_shndx;
 	int jumptables_data_shndx;
+	Elf_Data *cleanup_data;
+	int cleanup_shndx;
 };
 
 struct usdt_manager;
@@ -817,6 +844,9 @@ struct bpf_object {
 	void *jumptables_data;
 	size_t jumptables_data_sz;
 
+	struct cleanup_raw_rec *cleanup_recs;
+	size_t cleanup_rec_cnt;
+
 	struct {
 		struct bpf_program *prog;
 		unsigned int sym_off;
@@ -874,6 +904,7 @@ void bpf_program__unload(struct bpf_program *prog)
 	zfree(&prog->func_info);
 	zfree(&prog->line_info);
 	zfree(&prog->subprogs);
+	zfree(&prog->cleanup_info);
 }
 
 static void bpf_program__exit(struct bpf_program *prog)
@@ -1639,6 +1670,7 @@ static struct bpf_object *bpf_object__new(const char *path,
 	obj->efile.obj_buf_sz = obj_buf_sz;
 	obj->efile.btf_maps_shndx = -1;
 	obj->efile.arena_data_shndx = -1;
+	obj->efile.cleanup_shndx = -1;
 	obj->kconfig_map_idx = -1;
 	obj->arena_map_idx = -1;
 
@@ -1658,6 +1690,7 @@ static void bpf_object__elf_finish(struct bpf_object *obj)
 	obj->efile.ehdr = NULL;
 	obj->efile.symbols = NULL;
 	obj->efile.arena_data = NULL;
+	obj->efile.cleanup_data = NULL;
 
 	zfree(&obj->efile.secs);
 	obj->efile.sec_cnt = 0;
@@ -4275,6 +4308,9 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 				sec_desc->shdr = sh;
 				sec_desc->data = data;
 				obj->efile.has_st_ops = true;
+			} else if (strcmp(name, CLEANUP_SEC) == 0) {
+				obj->efile.cleanup_data = data;
+				obj->efile.cleanup_shndx = idx;
 			} else if (strcmp(name, ARENA_SEC) == 0) {
 				obj->efile.arena_data = data;
 				obj->efile.arena_data_shndx = idx;
@@ -4300,8 +4336,9 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 
 			/*
 			 * Only do relo for section with exec instructions,
-			 * struct_ops, maps, and read-only data that might
-			 * have pointers to functions.
+			 * struct_ops, maps, read-only data that might have
+			 * pointers to functions, and the exception cleanup
+			 * table.
 			 */
 			if (!section_have_execinstr(obj, targ_sec_idx) &&
 			    !(obj->data_in_arena &&
@@ -4315,6 +4352,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 			    strcmp(name, ".rel" STRUCT_OPS_LINK_SEC) &&
 			    strcmp(name, ".rel?" STRUCT_OPS_SEC) &&
 			    strcmp(name, ".rel?" STRUCT_OPS_LINK_SEC) &&
+			    strcmp(name, ".rel" CLEANUP_SEC) &&
 			    strcmp(name, ".rel" MAPS_ELF_SEC)) {
 				pr_info("elf: skipping relo section(%d) %s for section(%d) %s\n",
 					idx, name, targ_sec_idx,
@@ -5099,6 +5137,219 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,
 	return NULL;
 }
 
+static int bpf_object_init_cleanup_info(struct bpf_object *obj)
+{
+	Elf_Data *data = obj->efile.cleanup_data;
+	size_t i, nrels, nslots, nrecs;
+	struct cleanup_raw_rec *recs;
+	Elf_Data *relo = NULL;
+	const char *vals;
+	int ret = 0;
+	bool native;
+
+	if (!data || obj->efile.cleanup_shndx < 0 || !data->d_size)
+		return 0;
+
+	native = is_native_endianness(obj);
+
+	for (i = 0; i < obj->efile.sec_cnt; i++) {
+		struct elf_sec_desc *sd = &obj->efile.secs[i];
+
+		if (sd->sec_type == SEC_RELO && sd->shdr &&
+		    sd->shdr->sh_info == (Elf64_Word)obj->efile.cleanup_shndx) {
+			relo = sd->data;
+			break;
+		}
+	}
+	if (!relo) {
+		pr_warn("%s present without relocations\n", CLEANUP_SEC);
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+	if (data->d_size % sizeof(struct bpf_cleanup_info)) {
+		pr_warn("%s size %zu is not a multiple of the record size %zu\n",
+			CLEANUP_SEC, data->d_size, sizeof(struct bpf_cleanup_info));
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+
+	vals = data->d_buf;
+	nslots = data->d_size / sizeof(__u32);
+	nrecs = data->d_size / sizeof(struct bpf_cleanup_info);
+
+	recs = calloc(nrecs, sizeof(*recs));
+	if (!recs)
+		return -ENOMEM;
+	for (i = 0; i < nslots; i++)
+		recs[i / CLEANUP_REC_FIELDS].sec_idx[i % CLEANUP_REC_FIELDS] = -1;
+
+	/* One relocation per 4-byte field, naming the section it points into. */
+	nrels = relo->d_size / sizeof(Elf64_Rel);
+	for (i = 0; i < nrels; i++) {
+		Elf64_Rel *rel = elf_rel_by_idx(relo, i);
+		Elf64_Sym *sym = elf_sym_by_idx(obj, ELF64_R_SYM(rel->r_info));
+		size_t type = ELF64_R_TYPE(rel->r_info);
+		size_t slot = rel->r_offset / sizeof(__u32);
+		struct cleanup_raw_rec *rec;
+		size_t off;
+		__u32 val;
+
+		if (type != R_BPF_64_NODYLD32 && type != R_BPF_64_ABS32) {
+			pr_warn("%s: relocation %zu has unexpected type %zu\n",
+				CLEANUP_SEC, i, type);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		if (!sym || slot >= nslots || rel->r_offset % sizeof(__u32)) {
+			pr_warn("%s: bad relocation %zu\n", CLEANUP_SEC, i);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		/*
+		 * The addend lives in the section data, which libelf leaves in
+		 * the object's byte order; a non-section symbol additionally
+		 * contributes its own value. The section is only byte aligned.
+		 */
+		memcpy(&val, vals + slot * sizeof(val), sizeof(val));
+		off = (native ? val : bswap_32(val)) + sym->st_value;
+		if (off % BPF_INSN_SZ) {
+			pr_warn("%s: field %zu offset %zu is not instruction aligned\n",
+				CLEANUP_SEC, slot, off);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		rec = &recs[slot / CLEANUP_REC_FIELDS];
+		if (rec->sec_idx[slot % CLEANUP_REC_FIELDS] >= 0) {
+			pr_warn("%s: field %zu has more than one relocation\n",
+				CLEANUP_SEC, slot);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+		rec->sec_idx[slot % CLEANUP_REC_FIELDS] = sym->st_shndx;
+		rec->insn_idx[slot % CLEANUP_REC_FIELDS] = off / BPF_INSN_SZ;
+	}
+
+	for (i = 0; i < nslots; i++) {
+		if (recs[i / CLEANUP_REC_FIELDS].sec_idx[i % CLEANUP_REC_FIELDS] < 0) {
+			pr_warn("%s: field %zu has no relocation\n", CLEANUP_SEC, i);
+			ret = -LIBBPF_ERRNO__FORMAT;
+			goto out;
+		}
+	}
+
+	obj->cleanup_recs = recs;
+	obj->cleanup_rec_cnt = nrecs;
+	return 0;
+out:
+	free(recs);
+	return ret;
+}
+
+static int cmp_cleanup_info(const void *a, const void *b)
+{
+	const struct bpf_cleanup_info *x = a, *y = b;
+
+	if (x->begin_off == y->begin_off)
+		return 0;
+	return x->begin_off < y->begin_off ? -1 : 1;
+}
+
+static int bpf_prog_collect_cleanup_info(struct bpf_object *obj,
+					 struct bpf_program *prog)
+{
+	size_t i;
+	int j;
+
+	for (i = 0; i < obj->cleanup_rec_cnt; i++) {
+		struct cleanup_raw_rec *raw = &obj->cleanup_recs[i];
+		struct bpf_program *owner = NULL;
+		struct bpf_cleanup_info ci = {};
+		__u32 fields[CLEANUP_REC_FIELDS];
+		void *tmp;
+
+		for (j = 0; j < CLEANUP_REC_FIELDS; j++) {
+			size_t idx = raw->insn_idx[j], final;
+			struct bpf_program *p;
+
+			/*
+			 * An exclusive end may name the instruction past the
+			 * last of a function, so ask about the last one the
+			 * range covers, the way the kernel does.
+			 */
+			if (j == CLEANUP_REC_END) {
+				if (!idx) {
+					pr_warn("%s: record %zu ends at instruction 0\n",
+						CLEANUP_SEC, i);
+					return -LIBBPF_ERRNO__FORMAT;
+				}
+				idx--;
+			}
+
+			p = find_prog_by_sec_insn(obj, raw->sec_idx[j], idx);
+			if (!p && j == CLEANUP_REC_BEGIN) {
+				/*
+				 * Code no function claims is an overridden weak
+				 * function the linker kept: skip its records,
+				 * as its relocations are.
+				 */
+				pr_debug("%s: record %zu is in no function, probably an overridden weak function, skipping\n",
+					 CLEANUP_SEC, i);
+				break;
+			}
+			if (!p) {
+				pr_warn("%s: record %zu field %d is not inside a function\n",
+					CLEANUP_SEC, i, j);
+				return -LIBBPF_ERRNO__FORMAT;
+			}
+			if (!owner) {
+				owner = p;
+			} else if (owner != p) {
+				pr_warn("%s: record %zu spans functions '%s' and '%s'\n",
+					CLEANUP_SEC, i, owner->name, p->name);
+				return -LIBBPF_ERRNO__FORMAT;
+			}
+
+			if (owner == prog) {
+				final = raw->insn_idx[j] - prog->sec_insn_off;
+			} else if (prog_is_subprog(obj, owner) && owner->sub_insn_off) {
+				/*
+				 * sub_insn_off is where this subprogram was
+				 * appended to the main program being relocated;
+				 * zero means it is not part of it.
+				 */
+				final = owner->sub_insn_off +
+					raw->insn_idx[j] - owner->sec_insn_off;
+			} else {
+				owner = NULL;
+				break;
+			}
+			fields[j] = final;
+		}
+		if (!owner)
+			continue;
+
+		ci.begin_off = fields[CLEANUP_REC_BEGIN];
+		ci.end_off = fields[CLEANUP_REC_END];
+		ci.landing_pad_off = fields[CLEANUP_REC_PAD];
+
+		tmp = libbpf_reallocarray(prog->cleanup_info, prog->cleanup_info_cnt + 1,
+					  sizeof(*prog->cleanup_info));
+		if (!tmp)
+			return -ENOMEM;
+		prog->cleanup_info = tmp;
+		prog->cleanup_info_rec_size = sizeof(struct bpf_cleanup_info);
+		prog->cleanup_info[prog->cleanup_info_cnt++] = ci;
+
+		pr_debug("prog '%s': cleanup region [%u,%u) -> landing pad %u\n",
+			 prog->name, ci.begin_off, ci.end_off, ci.landing_pad_off);
+	}
+
+	if (!prog->cleanup_info_cnt)
+		return 0;
+
+	qsort(prog->cleanup_info, prog->cleanup_info_cnt,
+	      sizeof(*prog->cleanup_info), cmp_cleanup_info);
+	return 0;
+}
+
 static int
 bpf_object__collect_prog_relos(struct bpf_object *obj, Elf64_Shdr *shdr, Elf_Data *data)
 {
@@ -8197,6 +8448,13 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_pat
 					return err;
 			}
 		}
+
+		err = bpf_prog_collect_cleanup_info(obj, prog);
+		if (err) {
+			pr_warn("prog '%s': failed to collect cleanup info: %s\n",
+				prog->name, errstr(err));
+			return err;
+		}
 	}
 	for (i = 0; i < obj->nr_programs; i++) {
 		prog = &obj->programs[i];
@@ -8561,6 +8819,9 @@ static int bpf_object__collect_relos(struct bpf_object *obj)
 			return -LIBBPF_ERRNO__INTERNAL;
 		}
 
+		if (idx == obj->efile.cleanup_shndx)
+			continue;
+
 		if (obj->efile.secs[idx].sec_type == SEC_RODATA)
 			err = bpf_object__collect_rodata_relos(obj, shdr, data);
 		else if (obj->efile.secs[idx].sec_type == SEC_DATA)
@@ -8856,6 +9117,11 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
 		load_attr.line_info_rec_size = prog->line_info_rec_size;
 		load_attr.line_info_cnt = prog->line_info_cnt;
 	}
+	if (prog->cleanup_info_cnt) {
+		load_attr.cleanup_info = prog->cleanup_info;
+		load_attr.cleanup_info_cnt = prog->cleanup_info_cnt;
+		load_attr.cleanup_info_rec_size = prog->cleanup_info_rec_size;
+	}
 	load_attr.log_level = log_level;
 	load_attr.prog_flags = prog->prog_flags;
 	/* the program accesses its data in arena through plain numbers */
@@ -9460,6 +9726,7 @@ static struct bpf_object *bpf_object_open(const char *path, const void *obj_buf,
 	err = err ? : bpf_object__init_maps(obj, opts);
 	err = err ? : bpf_object_init_progs(obj, opts);
 	err = err ? : bpf_object__collect_relos(obj);
+	err = err ? : bpf_object_init_cleanup_info(obj);
 	if (err)
 		goto out;
 
@@ -10602,6 +10869,9 @@ void bpf_object__close(struct bpf_object *obj)
 	zfree(&obj->jumptables_data);
 	obj->jumptables_data_sz = 0;
 
+	zfree(&obj->cleanup_recs);
+	obj->cleanup_rec_cnt = 0;
+
 	for (i = 0; i < obj->jumptable_map_cnt; i++)
 		close(obj->jumptable_maps[i].fd);
 	zfree(&obj->jumptable_maps);
@@ -11011,6 +11281,19 @@ int bpf_program__clone(struct bpf_program *prog, const struct bpf_prog_load_opts
 		attr.line_info_rec_size = info ? info_rec_size : prog->line_info_rec_size;
 	}
 
+	/* exception cleanup table */
+	info = OPTS_GET(opts, cleanup_info, NULL);
+	info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+	info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);
+	if (!!info != !!info_cnt || !!info != !!info_rec_size) {
+		pr_warn("prog '%s': cleanup_info, cleanup_info_cnt, and cleanup_info_rec_size must all be specified or all omitted\n",
+			prog->name);
+		return libbpf_err(-EINVAL);
+	}
+	attr.cleanup_info = info ?: prog->cleanup_info;
+	attr.cleanup_info_cnt = info ? info_cnt : prog->cleanup_info_cnt;
+	attr.cleanup_info_rec_size = info ? info_rec_size : prog->cleanup_info_rec_size;
+
 	/* Logging is caller-controlled; no fallback to prog/obj log settings */
 	attr.log_buf = OPTS_GET(opts, log_buf, NULL);
 	attr.log_size = OPTS_GET(opts, log_size, 0);
tools/lib/bpf/libbpf_internal.h +3 -0
index 4e45dfacb548..ecbdb0d7dee1 100644
@@ -56,6 +56,9 @@
 #ifndef R_BPF_64_ABS32
 #define R_BPF_64_ABS32 3
 #endif
+#ifndef R_BPF_64_NODYLD32
+#define R_BPF_64_NODYLD32 4
+#endif
 #ifndef R_BPF_64_32
 #define R_BPF_64_32 10
 #endif
Signature · 1 line
2.53.0-Meta
author reviewer
#27 lore raw

PATCH v12 20/25 libbpf: Carry the exception cleanup table through the light skeleton

In reply to Yonghong Song

A light skeleton does not call bpf_prog_load(): its loader program builds the attr itself, so a program loaded this way reached the kernel without its table and was refused with "unreachable insn". Carry the table the way func_info and line_info are carried: the records in the loader's blob, the count and record size in the attr, and a relocation for the blob's address.

Only a program with records grows the attr to cleanup_info_cnt, so light skeletons without them are unchanged.

---
 tools/lib/bpf/gen_loader.c      | 29 ++++++++++++++++++++++++++---
 tools/lib/bpf/libbpf_internal.h |  7 +++++++
 2 files changed, 33 insertions(+), 3 deletions(-)
tools/lib/bpf/gen_loader.c +26 -3
index 251392aa8b41..f07727c5b5fb 100644
@@ -994,13 +994,15 @@ static void cleanup_relos(struct bpf_gen *gen, int insns)
 	cleanup_core_relo(gen);
 }
 
-/* Convert func, line, and core relo info blobs to target endianness */
+/* Convert func, line, core relo and cleanup info blobs to target endianness */
 static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
-			    int core_relos, struct bpf_prog_load_opts *load_attr)
+			    int core_relos, int cleanup_info,
+			    struct bpf_prog_load_opts *load_attr)
 {
 	struct bpf_func_info *fi = gen->data_start + func_info;
 	struct bpf_line_info *li = gen->data_start + line_info;
 	struct bpf_core_relo *cr = gen->data_start + core_relos;
+	struct bpf_cleanup_info *ci = gen->data_start + cleanup_info;
 	int i;
 
 	for (i = 0; i < load_attr->func_info_cnt; i++)
@@ -1011,6 +1013,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
 
 	for (i = 0; i < gen->core_relo_cnt; i++)
 		bpf_core_relo_bswap(cr++);
+
+	for (i = 0; i < load_attr->cleanup_info_cnt; i++)
+		bpf_cleanup_info_bswap(ci++);
 }
 
 void bpf_gen__prog_load(struct bpf_gen *gen,
@@ -1024,10 +1029,15 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 			       load_attr->line_info_rec_size;
 	int core_relo_tot_sz = gen->core_relo_cnt *
 			       sizeof(struct bpf_core_relo);
+	int cleanup_info_tot_sz = load_attr->cleanup_info_cnt *
+				  load_attr->cleanup_info_rec_size;
 	int prog_load_attr, license_off, insns_off, func_info, line_info, core_relos;
 	int attr_size = offsetofend(union bpf_attr, core_relo_rec_size);
+	int cleanup_info;
 	union bpf_attr attr;
 
+	if (load_attr->cleanup_info_cnt)
+		attr_size = offsetofend(union bpf_attr, cleanup_info_cnt);
 	memset(&attr, 0, attr_size);
 	/* add license string to blob of bytes */
 	license_off = add_data(gen, license, strlen(license) + 1);
@@ -1074,9 +1084,17 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 		 core_relos, gen->core_relo_cnt,
 		 sizeof(struct bpf_core_relo));
 
+	attr.cleanup_info_rec_size = tgt_endian(load_attr->cleanup_info_rec_size);
+	attr.cleanup_info_cnt = tgt_endian(load_attr->cleanup_info_cnt);
+	cleanup_info = add_data(gen, load_attr->cleanup_info, cleanup_info_tot_sz);
+	pr_debug("gen: prog_load: cleanup_info: off %d cnt %u rec size %u\n",
+		 cleanup_info, load_attr->cleanup_info_cnt,
+		 load_attr->cleanup_info_rec_size);
+
 	/* convert all info blobs to target endianness */
 	if (gen->swapped_endian && !gen->error)
-		info_blob_bswap(gen, func_info, line_info, core_relos, load_attr);
+		info_blob_bswap(gen, func_info, line_info, core_relos, cleanup_info,
+				load_attr);
 
 	libbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name));
 	prog_load_attr = add_data(gen, &attr, attr_size);
@@ -1098,6 +1116,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 	/* populate union bpf_attr with a pointer to core_relos */
 	emit_rel_store(gen, attr_field(prog_load_attr, core_relos), core_relos);
 
+	/* with no records, attr ends before cleanup_info: nothing to store */
+	if (load_attr->cleanup_info_cnt)
+		emit_rel_store(gen, attr_field(prog_load_attr, cleanup_info),
+			       cleanup_info);
+
 	/* populate union bpf_attr fd_array with a pointer to data where map_fds are saved */
 	emit_rel_store(gen, attr_field(prog_load_attr, fd_array), gen->fd_array);
 
tools/lib/bpf/libbpf_internal.h +7 -0
index ecbdb0d7dee1..e17fc8f60199 100644
@@ -573,6 +573,13 @@ static inline void bpf_core_relo_bswap(struct bpf_core_relo *i)
 	i->kind = bswap_32(i->kind);
 }
 
+static inline void bpf_cleanup_info_bswap(struct bpf_cleanup_info *i)
+{
+	i->begin_off = bswap_32(i->begin_off);
+	i->end_off = bswap_32(i->end_off);
+	i->landing_pad_off = bswap_32(i->landing_pad_off);
+}
+
 enum btf_field_iter_kind {
 	BTF_FIELD_ITER_IDS,
 	BTF_FIELD_ITER_STRS,
Signature · 1 line
2.53.0-Meta
author reviewer
#28 lore raw

PATCH v12 21/25 libbpf: Let the static linker carry .bpf_cleanup relocations

In reply to Yonghong Song

Each field of a .bpf_cleanup record is a byte offset into a code
section: a 32-bit relocation against that section's symbol, with the offset as the implicit addend. The linker refuses it: it does not know LLVM's R_BPF_64_NODYLD32, and from a non-executable section it takes a relocation against a section symbol only as a 64-bit pointer to code.

Take it in both spellings, R_BPF_64_NODYLD32 from LLVM and R_BPF_64_ABS32 from GNU as, adding the code section's new start to the stored byte offset. Since any non-executable section can now reach that path, it checks itself what the old refusal covered: that the target has data, and that r_offset is aligned and in bounds.

---
 tools/lib/bpf/linker.c | 39 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)
tools/lib/bpf/linker.c +38 -1
index f3f71c452f00..911a649219a9 100644
@@ -1036,7 +1036,8 @@ static int linker_sanity_check_elf_relos(struct src_obj *obj, struct src_sec *se
 		size_t sym_type = ELF64_R_TYPE(relo->r_info);
 
 		if (sym_type != R_BPF_64_64 && sym_type != R_BPF_64_32 &&
-		    sym_type != R_BPF_64_ABS64 && sym_type != R_BPF_64_ABS32) {
+		    sym_type != R_BPF_64_ABS64 && sym_type != R_BPF_64_ABS32 &&
+		    sym_type != R_BPF_64_NODYLD32) {
 			pr_warn("ELF relo #%d in section #%zu has unexpected type %zu in %s\n",
 				i, sec->sec_idx, sym_type, obj->filename);
 			return -EINVAL;
@@ -2263,6 +2264,7 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
 			if (ELF64_ST_TYPE(src_sym->st_info) == STT_SECTION) {
 				struct src_sec *sec = &obj->secs[src_sym->st_shndx];
 				struct bpf_insn *insn;
+				__u32 *val;
 
 				if (src_linked_sec->shdr->sh_flags & SHF_EXECINSTR) {
 					/* calls to the very first static function inside
@@ -2297,6 +2299,41 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
 					if (linker->swapped_endian)
 						off = bswap_64(off);
 					memcpy(ptr, &off, sizeof(off));
+				} else if ((sym_type == R_BPF_64_NODYLD32 ||
+					    sym_type == R_BPF_64_ABS32) &&
+					   (sec->shdr->sh_flags & SHF_EXECINSTR)) {
+					/*
+					 * A byte offset into a code section,
+					 * stored in place. LLVM spells this
+					 * relocation NODYLD32 and GNU as
+					 * spells it ABS32; being bytes, the
+					 * section's new start goes in as it
+					 * is, not scaled the way a call's
+					 * instruction index is above.
+					 *
+					 * The sanity check only checks the
+					 * alignment of r_offset, and only for
+					 * an executable section, and SHT_NOBITS
+					 * has no raw_data, so check both here,
+					 * bounding by subtraction so it cannot
+					 * wrap.
+					 */
+					if (!dst_linked_sec->raw_data ||
+					    dst_linked_sec->sec_sz < (int)sizeof(*val) ||
+					    dst_rel->r_offset % sizeof(*val) ||
+					    dst_rel->r_offset >
+					    (size_t)dst_linked_sec->sec_sz - sizeof(*val)) {
+						pr_warn("ELF relo #%d in section #%zu points outside the data of section '%s' in %s\n",
+							j, src_sec->sec_idx,
+							dst_linked_sec->sec_name,
+							obj->filename);
+						return -EINVAL;
+					}
+					val = dst_linked_sec->raw_data + dst_rel->r_offset;
+					if (linker->swapped_endian)
+						*val = bswap_32(bswap_32(*val) + sec->dst_off);
+					else
+						*val += sec->dst_off;
 				} else {
 					pr_warn("relocation against STT_SECTION in non-exec section is not supported!\n");
 					return -EINVAL;
Signature · 1 line
2.53.0-Meta
author reviewer
#29 lore raw

PATCH v12 22/25 selftests/bpf: Add end-to-end and negative .bpf_cleanup exception tests

In reply to Yonghong Song

C has no unwinding, so the frames that own a resource are written in inline assembly, spelling out what a frontend emits: a call site between two labels, a landing pad, and a .bpf_cleanup record tying them together. Like a frontend, they mostly put nothing, or only a jump to itself, after a direct call to bpf_unwind() or bpf_unwind_resume(); a call to a subprog that always unwinds is still followed by the frame's normal path.

The end-to-end test runs a five-frame call chain whose frames hold an RCU read lock or have preemption disabled, unwinds it from different frames, and checks which landing pads ran. It also checks that an fentry program can attach to a subprog an unwind passes through and an fexit program cannot.

The negative tests cover the shapes the kernel refuses: catch pads, unsupported instructions or a second unwind in a pad, resumes outside a pad, callbacks that can unwind, a table combined with bpf_throw() or an exception callback, misplaced records, stale stack slots trusted by a pad, and resources an unwind leaves held or drops twice.

The test skips where the JIT cannot dispatch landing pads.

---
 .../selftests/bpf/exceptions_cleanup.h        |  27 +
 .../bpf/prog_tests/exceptions_cleanup.c       | 140 ++++
 .../selftests/bpf/progs/exceptions_cleanup.c  | 157 ++++
 .../bpf/progs/exceptions_cleanup_fail.c       | 792 ++++++++++++++++++
 .../bpf/progs/exceptions_cleanup_tracing.c    |  26 +
 5 files changed, 1142 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/exceptions_cleanup.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_tracing.c
tools/testing/selftests/bpf/exceptions_cleanup.h new file +27 -0
new file mode 100644
index 000000000000..96effd2c1361
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef __EXCEPTIONS_CLEANUP_H__
+#define __EXCEPTIONS_CLEANUP_H__
+
+/* progs/exceptions_cleanup.c: one bit per function that reports it ran. */
+#define RAN_FOO3_PREEMPT	0x1
+#define RAN_FOO2_RCU		0x2
+#define RAN_FOO1V_PREEMPT	0x4
+#define RAN_FOO2_DROP		0x8
+#define RAN_BUMP		0x10
+
+#define CLEANUP_REC(begin, end, landing_pad)			\
+	".pushsection .bpf_cleanup,\"a\",@progbits;"		\
+	".long " begin ";"					\
+	".long " end ";"					\
+	".long " landing_pad ";"				\
+	".popsection;"
+
+/* Set a bit in @pads_ran. */
+#define PAD_RAN(bit)						\
+	"r1 = %[pads_ran] ll;"					\
+	"r2 = *(u64 *)(r1 + 0);"				\
+	"r2 |= " bit ";"					\
+	"*(u64 *)(r1 + 0) = r2;"
+
+#endif /* __EXCEPTIONS_CLEANUP_H__ */
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file +140 -0

140 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup.c new file +157 -0

157 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file +792 -0

792 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_cleanup_tracing.c new file +26 -0
new file mode 100644
index 000000000000..9fbc06d0bec4
@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+/*
+ * Tracing programs aimed at foo1() of progs/exceptions_cleanup.c, a subprog
+ * an unwind passes through. The attach target is set at load time.
+ */
+SEC("?fexit")
+int BPF_PROG(fexit_unwinding_subprog)
+{
+	return 0;
+}
+
+int fentry_hits;
+
+SEC("?fentry")
+int BPF_PROG(fentry_unwinding_subprog)
+{
+	fentry_hits++;
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
Signature · 1 line
2.53.0-Meta
author reviewer
#30 lore raw

PATCH v12 23/25 selftests/bpf: Add __set_global() and __ret_global() test tags

In reply to Yonghong Song

A test that looks at a program's state after it ran needs a driver of its own, though the loader already runs programs for __retval(). What is missing is reaching their globals:

- __set_global(var, value) writes one before the run, __ret_global(var, value) checks one after it. Both make the loader run the program; without __retval() it must return 0. Each may be given several times, up to MAX_GLOBAL_VARS.
- The variable is looked up by name in ".bss" or ".data", and must be a four or eight byte int or enum.
- A value is held to its variable's range and narrowed to its width, signedness taken from BTF as in veristat, so a negative literal means the same to both tags.
- A value may be a '|' separated list, so a bitmask reads as it is written in the program.

---
 tools/testing/selftests/bpf/progs/bpf_misc.h |  12 +
 tools/testing/selftests/bpf/test_loader.c    | 319 ++++++++++++++++++-
 2 files changed, 330 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/progs/bpf_misc.h +12 -0
index d7f706c3bc4a..f5baeb979cc2 100644
@@ -107,6 +107,16 @@
  *                   - POINTER_VALUE
  *                   - TEST_DATA_LEN
  * __retval_unpriv   Same, but load program in unprivileged mode.
+ * __set_global      Set a global variable of the program to a value before
+ *                   executing it. The variable has to be an int or an enum,
+ *                   live in .bss or .data and be four or eight bytes wide,
+ *                   and the value is a number or several joined by '|',
+ *                   without parentheses.
+ * __ret_global      Execute the program and check that a global variable
+ *                   holds the given value afterwards, under the same rules.
+ *                   __set_global and __ret_global both make the loader run
+ *                   the program and check its return value; without
+ *                   __retval, the program must return 0.
  *
  * __description     Text to be used for display and as an additional filter
  *                   alias, while the original program name stays matchable.
@@ -167,6 +177,8 @@
 #define __flag(flag)		__test_tag("test_prog_flags=" #flag)
 #define __retval(val)		__test_tag("test_retval=" XSTR(val))
 #define __retval_unpriv(val)	__test_tag("test_retval_unpriv=" XSTR(val))
+#define __set_global(var, val)	__test_tag("test_global_set=" #var ":" XSTR(val))
+#define __ret_global(var, val)	__test_tag("test_global_ret=" #var ":" XSTR(val))
 #define __auxiliary		__test_tag("test_auxiliary")
 #define __auxiliary_unpriv	__test_tag("test_auxiliary_unpriv")
 #define __btf_path(path)	__test_tag("test_btf_path=" path)
tools/testing/selftests/bpf/test_loader.c +318 -1

379 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
author reviewer
#31 lore raw

PATCH v12 24/25 selftests/bpf: Cover more accepted .bpf_cleanup exception shapes

In reply to Yonghong Song

Add the shapes the end-to-end test does not reach:

- a pad reading its frame's callee-saved registers
- a region ending on a 16-byte instruction
- a pad ending in _Unwind_Resume rather than bpf_unwind_resume
- a pad whose first instruction is a nop
- a pad indexing its frame by a register set before the call
- two pads with an uncovered frame between them
- precision chains from a pad back across the resume that led to it, and back into the frame the unwind left
- a frame above an unwind that never returns, whose kept exit is not its last instruction
- frames with no exit at all, ending in the loop a compiler puts after a call that does not return, one of them sent to the epilogue the verifier gives it
- a frame with no exit whose last instruction is a covered call to a subprog after it, which keeps its record when the verifier puts an exit after the call
- nothing after a call that does not return: a bpf_unwind() ending its function, a call to a subprog that only unwinds ending one that otherwise returns, and a pad ending in its resume, run both ways
- a region covering an indirect call, and a subprog calling an unwinding one through a pointer it was handed
- a pad reading a slot its static or global callee wrote before unwinding
- unwinds out of a global subprog called with no record, one returning in R0:R2 among them
- a reference moved into a callee, as rustc does for a by-value argument
- an unwind from inside a loop, with the loop in a subprog and in main
- an unwind reaching main's exit in a program type that checks r0, out of a static callee and out of a global call
- calls to unwinding subprogs on a path never taken, which dead code removal drops whole, subprogs and all
- a pad touching a global of each width and sign a tag can name

Most pads end in their resume and most unwinders in their bpf_unwind(), as a frontend emits them, and prec_inner_frame() follows its covered bpf_unwind() with a jump to itself. always_unwind() keeps the dead code after its bpf_unwind(), and a call to a subprog that always unwinds is still followed by the frame's normal path.

Each shape that runs is driven by __set_global(), __retval() and __ret_global() through RUN_TESTS().

---
 .../selftests/bpf/exceptions_cleanup.h        |   21 +
 .../bpf/prog_tests/exceptions_cleanup.c       |    2 +
 .../bpf/progs/exceptions_cleanup_shapes.c     | 1134 +++++++++++++++++
 3 files changed, 1157 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
tools/testing/selftests/bpf/exceptions_cleanup.h +21 -0
index 96effd2c1361..84f21ecc630a 100644
@@ -10,6 +10,27 @@
 #define RAN_FOO2_DROP		0x8
 #define RAN_BUMP		0x10
 
+/* progs/exceptions_cleanup_shapes.c: one bit per landing pad. */
+#define RAN_REGS		0x1
+#define RAN_WIDE_REC		0x2
+#define RAN_RESUME_ALIAS	0x4
+#define RAN_NOP_PAD		0x8
+#define RAN_VAR_STACK		0x10
+#define RAN_GAP_INNER		0x20
+#define RAN_GAP_OUTER		0x40
+#define RAN_PREC_RESUME		0x80
+#define RAN_NO_EXIT_JA		0x100
+#define RAN_CALLX		0x200
+#define RAN_CALLEE_WRITE	0x400
+#define RAN_CALLEE_OFFSET	0x800
+#define RAN_GLOBAL_WRITE	0x1000
+#define RAN_THROUGH_GLOBAL	0x2000
+#define RAN_PAIR_GLOBAL		0x4000
+#define RAN_MOVE		0x8000
+#define RAN_NO_EXIT_LOOP	0x10000
+#define RAN_LAST_CALL		0x20000
+#define RAN_NO_TAIL		0x40000
+
 #define CLEANUP_REC(begin, end, landing_pad)			\
 	".pushsection .bpf_cleanup,\"a\",@progbits;"		\
 	".long " begin ";"					\
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +2 -0
index d8320ed3294f..76fdb3355420 100644
@@ -4,6 +4,7 @@
 #include "exceptions_cleanup.h"
 #include "exceptions_cleanup.skel.h"
 #include "exceptions_cleanup_fail.skel.h"
+#include "exceptions_cleanup_shapes.skel.h"
 #include "exceptions_cleanup_tracing.skel.h"
 
 /* foo3 unwound: every frame that has a pad ran it. */
@@ -137,4 +138,5 @@ void test_exceptions_cleanup(void)
 	exceptions_cleanup__destroy(skel);
 
 	RUN_TESTS(exceptions_cleanup_fail);
+	RUN_TESTS(exceptions_cleanup_shapes);
 }
tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c new file +1134 -0

1,134 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
author reviewer
#32 lore raw

PATCH v12 25/25 selftests/bpf: Load an exception cleanup program from a light skeleton

In reply to Yonghong Song

A light skeleton loads a program by a path of its own: its loader program builds the attr and resolves kfunc names when it runs. Build progs/exceptions_cleanup.c as a light skeleton too, and run the case in which every pad runs: a table that lost records or came out at the wrong offsets would fail to load, or run the wrong pads.

---
 tools/testing/selftests/bpf/Makefile.skel     |  2 +-
 .../bpf/prog_tests/exceptions_cleanup.c       | 31 +++++++++++++++++++
 2 files changed, 32 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/Makefile.skel +1 -1
index 06e297a17156..d9cbe8ac85b7 100644
@@ -40,7 +40,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
 
 # Generate both light skeleton and libbpf skeleton for these
 LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \
-	kfunc_call_test_subprog.c test_global_percpu_data.c
+	kfunc_call_test_subprog.c test_global_percpu_data.c exceptions_cleanup.c
 SKEL_BLACKLIST += $(LSKELS) $(LSKELS_SIGNED)
 
 test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o
tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c +31 -0
index 76fdb3355420..b8581716bbd4 100644
@@ -6,6 +6,7 @@
 #include "exceptions_cleanup_fail.skel.h"
 #include "exceptions_cleanup_shapes.skel.h"
 #include "exceptions_cleanup_tracing.skel.h"
+#include "exceptions_cleanup.lskel.h"
 
 /* foo3 unwound: every frame that has a pad ran it. */
 #define PADS_FOO3_UNWOUND \
@@ -89,6 +90,33 @@ static void test_tracing(struct exceptions_cleanup *tgt)
 	exceptions_cleanup_tracing__destroy(skel);
 }
 
+static void test_light_skeleton(void)
+{
+	struct exceptions_cleanup_lskel *skel;
+	__u64 ctx = 0;
+	int err;
+
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		    .ctx_in = &ctx,
+		    .ctx_size_in = sizeof(ctx),
+	);
+
+	skel = exceptions_cleanup_lskel__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "light open_and_load"))
+		return;
+
+	/* foo3 unwinds, so every pad runs: the whole table has to arrive. */
+	skel->bss->input = 101;
+	err = bpf_prog_test_run_opts(skel->progs.entry.prog_fd, &topts);
+	if (!ASSERT_OK(err, "run"))
+		goto out;
+	ASSERT_EQ(topts.retval, 0, "retval");
+	ASSERT_EQ(skel->bss->pads_ran, PADS_FOO3_UNWOUND | RAN_BUMP,
+		  "pads_ran");
+out:
+	exceptions_cleanup_lskel__destroy(skel);
+}
+
 void test_exceptions_cleanup(void)
 {
 	char log[8192] = {};
@@ -137,6 +165,9 @@ void test_exceptions_cleanup(void)
 
 	exceptions_cleanup__destroy(skel);
 
+	if (test__start_subtest("light_skeleton"))
+		test_light_skeleton();
+
 	RUN_TESTS(exceptions_cleanup_fail);
 	RUN_TESTS(exceptions_cleanup_shapes);
 }
Signature · 1 line
2.53.0-Meta
#33 lore raw
In reply to Yonghong Song
Quoted 2 lines · Yonghong Song

Each frame therefore runs its own pad, on its own stack, and leaves through its own epilogue -- which is what puts its caller's r6-r9 back.

it's v12 and bots keep finding real bugs, I think it's a sign that we have to pivot.

The idea is to sacrifce run-time and make unwind logic a nop for the verifier:
callee will use r3 to tell caller whether it needs to unwind. while all callers do
'if r3 != 0 goto landing_pad'.
Then unwind is plain insns for the verifier, for every pass
in front of it and for JITs.
in_pad, unwind_exit, INSN_F_UNWIND, cleanup_pad in aux data,
pad edges in cfg.c, liveness.c, backtrack.c, unwind_frames(), keep_subprog_exits(), native tables, epilogue_ip, arch_bpf_stack_walk_ra(), PAC signing, all the complexity go away.

JITs have to keep r3 over the epilogue. x86, arm64, riscv, s390 do. fexit, freplace and tail call don't, so they are refused for a subprog that can unwind.

The callee of callx is a static subprog that the main pass walks, so it returns via prepare_func_exit() like any other. When a subprog with address taken can unwind all of them set r3.

uapi in patch 1, kfuncs in patch 2 and libbpf patches 17-21 stay as-is. Something like this instead of patches 3-16
(tested on x86 only, in qemu: rustc output from v10 thread, catch, second unwind, global subprogs, move into callee, callx directly and via a table
in rodata load and run. full test_progs was not run):

include/linux/bpf.h +1 -0
index 54144372281c..f55a14e6c388 100644
@@ -1882,6 +1882,7 @@ struct bpf_prog_aux {
 	bool priv_stack_requested;
 	bool changes_pkt_data;
 	bool might_sleep;
+	bool ret_unwind;
 	bool kprobe_write_ctx;
 	struct {
 		s32 keyring_serial;
include/linux/bpf_verifier.h +6 -0
index 11cd90d85bad..917fe736af8c 100644
@@ -944,6 +944,9 @@ struct bpf_subprog_info {
 	s16 fastcall_stack_off;
 	bool has_tail_call: 1;
 	bool might_throw: 1;
+	bool might_unwind: 1;
+	bool ret_unwind: 1; /* says in r3 whether it returns or unwinds */
+	bool unwind_stub: 1;
 	bool tail_call_reachable: 1;
 	bool has_ld_abs: 1;
 	bool is_cb: 1;
@@ -1103,6 +1106,7 @@ struct bpf_verifier_env {
 	struct bpf_func_ptr *func_ptrs;
 	u32 func_ptr_cnt;
 	bool has_callx;
+	bool callx_unwinds;
 	/*
 	 * Call graph edges created by callx instructions. A bitmap of
 	 * subprog_cnt * subprog_cnt bits, where bit (caller * subprog_cnt + callee)
@@ -1214,6 +1218,8 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env
 	return &env->subprog_info[subprog];
 }
 
+#define BPF_REG_UNWIND BPF_REG_3
+
 static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
 {
 	return subprog_info(env, subprog)->ret_reg_pair;
kernel/bpf/backtrack.c +8 -2
index 43d5740034c0..0f8dbacdf9ca 100644
@@ -448,6 +448,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				bt_clear_reg(bt, BPF_REG_0);
 				/* and if it does not set R2, main pass would catch it */
 				bt_clear_reg(bt, BPF_REG_2);
+				bt_clear_reg(bt, BPF_REG_UNWIND);
 				/* r1-r5 are invalidated after subprog call,
 				 * so for global func call it shouldn't be set
 				 * anymore
@@ -548,7 +549,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 					return -EFAULT;
 			}
 		} else if (opcode == BPF_EXIT) {
-			bool from_subprog_call, r0_precise, r2_precise;
+			bool from_subprog_call, r0_precise, r2_precise, r3_precise;
 
 			/* BPF_EXIT in subprog or callback always returns
 			 * right after the call instruction, so by checking
@@ -565,6 +566,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 
 			r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
 			r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2);
+			r3_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_UNWIND);
 
 			/* Backtracking to a nested function call, 'idx' is a part of
 			 * the inner frame 'subseq_idx' is a part of the outer frame.
@@ -578,8 +580,10 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 			 * might be a precise helper argument as well, so leave it set.
 			 */
 			bt_clear_reg(bt, BPF_REG_0);
-			if (from_subprog_call)
+			if (from_subprog_call) {
 				bt_clear_reg(bt, BPF_REG_2);
+				bt_clear_reg(bt, BPF_REG_UNWIND);
+			}
 			if (from_subprog_call && (bt_reg_mask(bt) & BPF_REGMASK_ARGS)) {
 				verifier_bug(env, "backtracking exit unexpected regs %x",
 					     bt_reg_mask(bt));
@@ -593,6 +597,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				bt_set_reg(bt, BPF_REG_0);
 			if (r2_precise)
 				bt_set_reg(bt, BPF_REG_2);
+			if (r3_precise)
+				bt_set_reg(bt, BPF_REG_UNWIND);
 			/* r6-r9 and stack slots will stay set in caller frame
 			 * bitmasks until we return back from callee(s)
 			 */
kernel/bpf/exception.c +240 -0

249 lines not on this page · raw message

kernel/bpf/exception.h +3 -0
index cf099dcc5b74..3ca174f9bfc6 100644
@@ -8,8 +8,11 @@
 
 union bpf_attr;
 struct bpf_verifier_env;
+struct bpf_insn;
 
 int bpf_exc_check_info(struct bpf_verifier_env *env, const union bpf_attr *attr,
 		       bpfptr_t uattr);
+bool bpf_is_unwind_kfunc(const struct bpf_insn *insn);
+int bpf_exc_lower(struct bpf_verifier_env *env);
 
 #endif /* __BPF_EXCEPTION_H */
kernel/bpf/fixups.c +1 -0
index 608621d7f225..8580fa2b634f 100644
@@ -1383,6 +1383,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 		func[i]->aux->exception_cb = env->subprog_info[i].is_exception_cb;
 		func[i]->aux->changes_pkt_data = env->subprog_info[i].changes_pkt_data;
 		func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
+		func[i]->aux->ret_unwind = env->subprog_info[i].ret_unwind;
 		func[i]->aux->token = prog->aux->token;
 		if (!i)
 			func[i]->aux->exception_boundary = env->seen_exception;
kernel/bpf/helpers.c +2 -0
index 8df3f4746b68..ce6ab81c1ec0 100644
@@ -5128,6 +5128,8 @@ BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_throw)
+BTF_ID_FLAGS(func, bpf_unwind)
+BTF_ID_FLAGS(func, bpf_unwind_resume)
 #ifdef CONFIG_BPF_EVENTS
 BTF_ID_FLAGS(func, bpf_send_signal_task)
 #endif
kernel/bpf/liveness.c +5 -2
index e657e6d95408..51ccb8b080a9 100644
@@ -2482,7 +2482,7 @@ static int callee_subprog(struct bpf_verifier_env *env, const struct bpf_insn *i
 static void compute_insn_live_regs(struct bpf_verifier_env *env,
 				   struct bpf_insn *insn,
 				   struct insn_live_regs *info,
-				   bool ret_reg_pair)
+				   bool ret_reg_pair, bool ret_unwind)
 {
 	struct bpf_call_summary cs;
 	const u8 class = BPF_CLASS(insn->code);
@@ -2616,6 +2616,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 		case BPF_EXIT:
 			def = 0;
 			use = ret_reg_pair ? (r0 | r2) : r0;
+			if (ret_unwind)
+				use |= reg64_mask(BPF_REG_UNWIND);
 			break;
 		case BPF_CALL:
 			def = ALL_CALLER_SAVED_REGS;
@@ -2698,7 +2700,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		ret_reg_pair = bpf_ret_reg_pair(env, subprog);
 
 		for (i = start; i < end; ++i)
-			compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
+			compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair,
+					       subprog && env->subprog_info[subprog].ret_unwind);
 	}
 
 	/* Forward pass: resolve stack access through FP-derived pointers */
kernel/bpf/verifier.c +33 -10

85 lines not on this page · raw message

pw-bot: cr

#34 lore raw
Quoted 1 line · Alexei Starovoitov

uapi in patch 1, kfuncs in patch 2 and libbpf patches 17-21 stay as-is.

On a 2nd thought...
The table exists to keep the normal path free of unwind code. Once we pay for the branch the kernel only uses the table to generate insns that llvm could have generated.
BPFAsmPrinter emits one record per invoke: arg setup plus one call. So the table doesn't say more than 'this call -> that pad'.

If llvm emits:

  invoke foo              call foo; if r3 != 0 goto pad
  ret                     r3 = 0; exit
  call _Unwind_Resume     r0 = 0; r3 = 1; exit
  invoke bpf_unwind       r0 = 0; goto pad

then .bpf_cleanup, uapi, both kfuncs, libbpf, light skeleton, static linker changes go away. Other loaders have nothing to do either. No patching before the main pass, so insn numbers in the log don't shift. Functions end with exit, so no need for patch 12. _Unwind_Resume and bpf_unwind don't reach the object file,
so no rename in libbpf and no BTF anchor for them in rust. llvm knows what is nounwind, so no conservative closure for callx. That's close to what Eduard was proposing in v4, but without new insn.

The kernel needs to know which subprogs say something in r3. In the sketch llvm adds BTF decl_tag "unwind" to such FUNCs. The insns come from user space now, so the verifier checks that r3 is a scalar and that a callback exits with r3 == 0.
r3 becomes an ABI between llvm and the kernel.

The kernel part, on top of bpf-next instead of the whole series:

include/linux/bpf.h +1 -0
index 54144372281c..f55a14e6c388 100644
@@ -1882,6 +1882,7 @@ struct bpf_prog_aux {
 	bool priv_stack_requested;
 	bool changes_pkt_data;
 	bool might_sleep;
+	bool ret_unwind;
 	bool kprobe_write_ctx;
 	struct {
 		s32 keyring_serial;
include/linux/bpf_verifier.h +3 -0
index 11cd90d85bad..6501c873f365 100644
@@ -944,6 +944,7 @@ struct bpf_subprog_info {
 	s16 fastcall_stack_off;
 	bool has_tail_call: 1;
 	bool might_throw: 1;
+	bool ret_unwind: 1; /* says in r3 whether it returns or unwinds */
 	bool tail_call_reachable: 1;
 	bool has_ld_abs: 1;
 	bool is_cb: 1;
@@ -1214,6 +1215,8 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env
 	return &env->subprog_info[subprog];
 }
 
+#define BPF_REG_UNWIND BPF_REG_3
+
 static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
 {
 	return subprog_info(env, subprog)->ret_reg_pair;
kernel/bpf/backtrack.c +8 -2
index 43d5740034c0..0f8dbacdf9ca 100644
@@ -448,6 +448,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				bt_clear_reg(bt, BPF_REG_0);
 				/* and if it does not set R2, main pass would catch it */
 				bt_clear_reg(bt, BPF_REG_2);
+				bt_clear_reg(bt, BPF_REG_UNWIND);
 				/* r1-r5 are invalidated after subprog call,
 				 * so for global func call it shouldn't be set
 				 * anymore
@@ -548,7 +549,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 					return -EFAULT;
 			}
 		} else if (opcode == BPF_EXIT) {
-			bool from_subprog_call, r0_precise, r2_precise;
+			bool from_subprog_call, r0_precise, r2_precise, r3_precise;
 
 			/* BPF_EXIT in subprog or callback always returns
 			 * right after the call instruction, so by checking
@@ -565,6 +566,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 
 			r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
 			r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2);
+			r3_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_UNWIND);
 
 			/* Backtracking to a nested function call, 'idx' is a part of
 			 * the inner frame 'subseq_idx' is a part of the outer frame.
@@ -578,8 +580,10 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 			 * might be a precise helper argument as well, so leave it set.
 			 */
 			bt_clear_reg(bt, BPF_REG_0);
-			if (from_subprog_call)
+			if (from_subprog_call) {
 				bt_clear_reg(bt, BPF_REG_2);
+				bt_clear_reg(bt, BPF_REG_UNWIND);
+			}
 			if (from_subprog_call && (bt_reg_mask(bt) & BPF_REGMASK_ARGS)) {
 				verifier_bug(env, "backtracking exit unexpected regs %x",
 					     bt_reg_mask(bt));
@@ -593,6 +597,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				bt_set_reg(bt, BPF_REG_0);
 			if (r2_precise)
 				bt_set_reg(bt, BPF_REG_2);
+			if (r3_precise)
+				bt_set_reg(bt, BPF_REG_UNWIND);
 			/* r6-r9 and stack slots will stay set in caller frame
 			 * bitmasks until we return back from callee(s)
 			 */
kernel/bpf/fixups.c +1 -0
index 608621d7f225..8580fa2b634f 100644
@@ -1383,6 +1383,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 		func[i]->aux->exception_cb = env->subprog_info[i].is_exception_cb;
 		func[i]->aux->changes_pkt_data = env->subprog_info[i].changes_pkt_data;
 		func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
+		func[i]->aux->ret_unwind = env->subprog_info[i].ret_unwind;
 		func[i]->aux->token = prog->aux->token;
 		if (!i)
 			func[i]->aux->exception_boundary = env->seen_exception;
kernel/bpf/liveness.c +5 -2
index e657e6d95408..51ccb8b080a9 100644
@@ -2482,7 +2482,7 @@ static int callee_subprog(struct bpf_verifier_env *env, const struct bpf_insn *i
 static void compute_insn_live_regs(struct bpf_verifier_env *env,
 				   struct bpf_insn *insn,
 				   struct insn_live_regs *info,
-				   bool ret_reg_pair)
+				   bool ret_reg_pair, bool ret_unwind)
 {
 	struct bpf_call_summary cs;
 	const u8 class = BPF_CLASS(insn->code);
@@ -2616,6 +2616,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 		case BPF_EXIT:
 			def = 0;
 			use = ret_reg_pair ? (r0 | r2) : r0;
+			if (ret_unwind)
+				use |= reg64_mask(BPF_REG_UNWIND);
 			break;
 		case BPF_CALL:
 			def = ALL_CALLER_SAVED_REGS;
@@ -2698,7 +2700,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		ret_reg_pair = bpf_ret_reg_pair(env, subprog);
 
 		for (i = start; i < end; ++i)
-			compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
+			compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair,
+					       subprog && env->subprog_info[subprog].ret_unwind);
 	}
 
 	/* Forward pass: resolve stack access through FP-derived pointers */
kernel/bpf/verifier.c +45 -3

103 lines not on this page · raw message

The llvm part:

llvm/lib/Target/BPF/BPF.h +2 -0
diff -ruN a/llvm/lib/Target/BPF/BPF.h b/llvm/lib/Target/BPF/BPF.h
@@ -35,6 +35,8 @@
 };
 
 ModulePass *createBPFCheckAndAdjustIRLegacyPass();
+ModulePass *createBPFUnwindPreparePass();
+FunctionPass *createBPFUnwindLowerPass();
 
 class BPFISelDAGToDAGPass : public SelectionDAGISelPass {
 public:
llvm/lib/Target/BPF/BPFISelLowering.cpp +8 -0
diff -ruN a/llvm/lib/Target/BPF/BPFISelLowering.cpp b/llvm/lib/Target/BPF/BPFISelLowering.cpp
@@ -696,6 +696,14 @@
     RetOps.push_back(DAG.getRegister(VA.getLocReg(), VA.getLocVT()));
   }
 
+  // A function that can unwind says in r3 that it did not.
+  if (!MF.getFunction().doesNotThrow()) {
+    Chain = DAG.getCopyToReg(Chain, DL, BPF::R3,
+                             DAG.getConstant(0, DL, MVT::i64), Glue);
+    Glue = Chain.getValue(1);
+    RetOps.push_back(DAG.getRegister(BPF::R3, MVT::i64));
+  }
+
   RetOps[0] = Chain; // Update chain.
 
   // Add the glue if we have it.
llvm/lib/Target/BPF/BPFTargetMachine.cpp +2 -0
diff -ruN a/llvm/lib/Target/BPF/BPFTargetMachine.cpp b/llvm/lib/Target/BPF/BPFTargetMachine.cpp
@@ -117,6 +117,7 @@
   addPass(createBPFCheckAndAdjustIRLegacyPass());
 
   TargetPassConfig::addIRPasses();
+  addPass(createBPFUnwindPreparePass());
 }
 
 TargetTransformInfo
@@ -147,6 +148,7 @@
 }
 
 void BPFPassConfig::addPreEmitPass() {
+  addPass(createBPFUnwindLowerPass());
   addPass(createBPFMIPreEmitCheckingLegacyPass());
   if (!DisableMIPeephole) {
     addPass(createBPFMIExpandStackArgPseudosLegacyPass());
llvm/lib/Target/BPF/BPFUnwind.cpp +206 -0

206 lines not on this page · raw message

llvm/lib/Target/BPF/BTFDebug.cpp +4 -0
diff -ruN a/llvm/lib/Target/BPF/BTFDebug.cpp b/llvm/lib/Target/BPF/BTFDebug.cpp
@@ -1645,6 +1645,10 @@
     FuncTypeId = processDISubprogram(SP, ProtoTypeId, Scope);
   }
 
+  // Tell the kernel which functions say in r3 how they return.
+  if (!MF->getFunction().doesNotThrow())
+    addType(std::make_unique<BTFTypeDeclTag>(FuncTypeId, -1, "unwind"));
+
   for (const auto &TypeEntry : TypeEntries)
     TypeEntry->completeType(*this);
 
llvm/lib/Target/BPF/CMakeLists.txt +1 -0
diff -ruN a/llvm/lib/Target/BPF/CMakeLists.txt b/llvm/lib/Target/BPF/CMakeLists.txt
@@ -41,6 +41,7 @@
   BPFSubtarget.cpp
   BPFTargetLoweringObjectFile.cpp
   BPFTargetMachine.cpp
+  BPFUnwind.cpp
   BPFMIPeephole.cpp
   BPFMIChecking.cpp
   BPFMISimplifyPatchable.cpp

Tested on x86 only, in qemu, with unmodified libbpf.
C with __attribute__((cleanup)) and C++ with try/catch built with -fexceptions:
guards in three frames, global subprog, callx via a table in rodata, 16 byte return, catch, bpf_loop() callback load and run.
A lock without a guard and a callback that unwinds are rejected. rustc output with panic=unwind (drop, move, a[i], dyn Trait) compiles to the expected asm.
test_progs and llvm tests were not run.

I'm not sure that decl_tag is the best way to tell the kernel.

Thoughts?

Eduard,
pls chime in as well.

#35 lore raw
In reply to Alexei Starovoitov
Quoted 8 lines · Alexei Starovoitov
Quoted 2 lines · Yonghong Song

Each frame therefore runs its own pad, on its own stack, and leaves through its own epilogue -- which is what puts its caller's r6-r9 back.

it's v12 and bots keep finding real bugs, I think it's a sign that we have to pivot.

My 2 cents: That seems a bit drastic to me; realistically, there's only one bug for v12, the other bot concerns are not very critical. I still favor this approach over adding a runtime calling convention that requires checks everywhere in the normal path. Sure, there might be more problems (with this one, or whatever we do to replace it once it is actually implemented and we study all corner cases), but at this point most of the current approach seems well thought out to me.

Quoted 1 line

[...]

#36 lore raw
Quoted 2 lines · Kumar Kartikeya Dwivedi

still favor this approach over adding a runtime calling convention that requires checks everywhere in the normal path.

I suspect you're concerned by run-time overhead.
I was as well until I actually benchmarked it.

JIT can drop the branch and remember it in extable instead.

  bpf                         x86 as-is                   x86 with extable
  call check                  call check                  call check
  if r3 != 0 goto pad         test rdx, rdx               call bpf_rcu_read_unlock
  call bpf_rcu_read_unlock    jne pad

call bpf_rcu_read_unlock

The idea:

at 'call foo; if r3 != 0 goto pad':

	emit(call foo)
	extable[n++] = { ret_addr -> pad }

at 'r3 = 1; exit':
my_ret_addr = bpf_unwind_target(my_ret_addr) epilogue

bpf_unwind_target(ra)

  {
	e = search_extable(ra);
	return e ? e->pad : ra;

}

Every frame fixes only its own return address.
No stack walk, no ORC. catch still works.

When JIT cannot drop the branch (private stack, constant blinding) there is no entry. The callee returns as usual and the branch does the job.

The verifier doesn't know about any of that.

That said, I couldn't measure the branch.
N call sites in a row, user space, Zen 5, cycles per call:

N nothing test+jne

  8192     4.04      4.06
  12288    5.57      4.49
  16384    6.60     10.57

32768 24.89 24.39

Up to 8k call sites it's 4 cycles with or without the branch. Above 16k the calls don't fit into BTB and it's 24 cycles either way. In between it goes both ways.

I also tried to keep the distance to the pad in a nop:
call foo
nopl dist(%rax)
'r3 = 1; exit' reads it via its return address. No lookup. It works, but the numbers are the same as for test+jne. No measurable difference in a microbenchmark.

I don't think we need the below diff from day one. I think we should start with plain branches
and then measure benefits of above diff on real rust prog.

The kernel and llvm diffs are so simple.
It's a huge difference vs everything we discussed before.

On top of the kernel part from the previous email
(x86 only, in qemu, the same tests pass with and without bpf_jit_harden=2):

arch/x86/net/bpf_jit_comp.c +123 -2
index 11a4e4f87aac..abe942631486 100644
@@ -15,6 +15,7 @@
 #include <linux/memory.h>
 #include <linux/sort.h>
 #include <linux/execmem.h>
+#include <linux/extable.h>
 #include <asm/extable.h>
 #include <asm/ftrace.h>
 #include <asm/set_memory.h>
@@ -2511,6 +2512,89 @@ static void adjust_cmp_imm(u32 *op, s32 *imm)
 	}
 }
 
+/*
+ * A subprog that can unwind says so in r3 and its caller does
+ * 'if r3 != 0 goto pad' after the call. Keep that off the normal path:
+ * the branch is not emitted, it becomes an extable entry
+ *   return address of the call -> pad
+ * An exit with r3 != 0 looks its own return address up and returns to the pad.
+ * When there is no entry it returns as usual and the branch is there.
+ */
+static bool unwind_site(struct bpf_verifier_env *env, struct bpf_prog *prog, int i)
+{
+	const struct bpf_insn *insn = &prog->insnsi[i];
+
+	/* 'pop r9' after the call has to run on the way to the pad too */
+	if (!env || i + 1 >= prog->len || prog->aux->priv_stack_ptr)
+		return false;
+	if (!bpf_pseudo_call(insn) && !bpf_is_callx(insn))
+		return false;
+	/*
+	 * No need to know the callee: the verifier lets r3 be read after
+	 * a call only when every callee seen there says something in it.
+	 */
+	return insn[1].code == (BPF_JMP | BPF_JNE | BPF_K) && !insn[1].imm &&
+	       insn[1].dst_reg == BPF_REG_UNWIND &&
+	       !bpf_is_jump_target(env, prog->aux->subprog_start + i + 1);
+}
+
+static u64 bpf_unwind_target(const struct bpf_prog_aux *aux, u64 ra)
+{
+	const struct exception_table_entry *e;
+	u32 i;
+
+	for (i = 0; i < aux->real_func_cnt; i++) {
+		struct bpf_prog *p = aux->func[i];
+		u64 start = (u64)p->bpf_func;
+
+		if (ra <= start || ra > start + p->jited_len)
+			continue;
+		/* - 1: what follows the call can have an entry of its own */
+		e = search_extable(p->aux->extable, p->aux->num_exentries, ra - 1);
+		return e ? ra + e->fixup : ra;
+	}
+	return ra;
+}
+
+enum { EXIT_RETURNS, EXIT_UNWINDS, EXIT_LOOK_AT_R3 };
+
+static int unwind_exit(struct bpf_verifier_env *env, struct bpf_prog *prog, int i)
+{
+	const struct bpf_insn *insn = &prog->insnsi[i];
+
+	if (!env || !prog->aux->ret_unwind)
+		return EXIT_RETURNS;
+	if (!i || bpf_is_jump_target(env, prog->aux->subprog_start + i) ||
+	    insn[-1].code != (BPF_ALU64 | BPF_MOV | BPF_K) ||
+	    insn[-1].dst_reg != BPF_REG_UNWIND)
+		return EXIT_LOOK_AT_R3;
+	return insn[-1].imm ? EXIT_UNWINDS : EXIT_RETURNS;
+}
+
+static int emit_unwind_ret_addr(u8 **pprog, struct bpf_prog *bpf_prog, u8 *ip, bool look_at_r3)
+{
+	u8 *prog = *pprog, *start = prog, *jmp = NULL;
+
+	if (look_at_r3) {
+		EMIT3(0x48, 0x85, 0xD2);	/* test rdx, rdx */
+		EMIT2(X86_JE, 0);
+		jmp = prog;
+	}
+	EMIT3(0x50, 0x56, 0x52);		/* push rax; push rsi; push rdx */
+	emit_mov_imm64(&prog, BPF_REG_1, (long)bpf_prog->aux >> 32, (u32)(long)bpf_prog->aux);
+	EMIT4(0x48, 0x8B, 0x75, 0x08);		/* mov rsi, [rbp + 8] */
+	ip += prog - start;
+	ip += x86_call_depth_emit_accounting(&prog, bpf_unwind_target, ip);
+	if (emit_call(&prog, bpf_unwind_target, ip))
+		return -EINVAL;
+	EMIT4(0x48, 0x89, 0x45, 0x08);		/* mov [rbp + 8], rax */
+	EMIT3(0x5A, 0x5E, 0x58);		/* pop rdx; pop rsi; pop rax */
+	if (jmp)
+		jmp[-1] = prog - jmp;
+	*pprog = prog;
+	return 0;
+}
+
 static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,
 		  u8 *rw_image, int oldproglen, struct jit_context *ctx, bool jmp_padding)
 {
@@ -2518,7 +2602,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 	struct bpf_insn *insn = bpf_prog->insnsi;
 	bool callee_regs_used[4] = {};
 	int insn_cnt = bpf_prog->len;
-	bool seen_exit = false;
+	bool seen_exit = false, skip_branch = false;
 	u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY];
 	void __percpu *priv_frame_ptr = NULL;
 	u16 out_stack_arg_cnt, outgoing_rsp;
@@ -2655,6 +2739,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		int nops;
 		u32 jmp_op = BPF_OP(insn->code);
 		s32 cmp_imm;
+		int unw;
+
+		if (skip_branch) {
+			skip_branch = false;
+			goto insn_done;
+		}
 
 		if (priv_frame_ptr) {
 			if (src_reg == BPF_REG_FP)
@@ -3460,6 +3550,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			ip += x86_call_depth_emit_accounting(&prog, func, ip);
 			if (emit_call(&prog, func, ip))
 				return -EINVAL;
+			if (unwind_site(env, bpf_prog, i - 1))
+				goto unwind_exentry;
 			if (priv_frame_ptr)
 				pop_r9(&prog);
 			/*
@@ -3487,6 +3579,27 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			}
 			if (emit_indirect_call(&prog, insn->dst_reg, ip))
 				return -EINVAL;
+			if (unwind_site(env, bpf_prog, i - 1)) {
+				struct exception_table_entry *ex;
+				int ra;
+unwind_exentry:
+				skip_branch = true;
+				if (!bpf_prog->aux->extable)
+					break;
+				if (excnt >= bpf_prog->aux->num_exentries) {
+					pr_err("unwind extable bug\n");
+					return -EFAULT;
+				}
+				ra = proglen + (prog - temp);
+				ex = &bpf_prog->aux->extable[excnt++];
+				jmp_offset = image + ra - 1 - (u8 *)&ex->insn;
+				/* switch ex to rw buffer for writes */
+				ex = (void *)rw_image + ((void *)ex - (void *)image);
+				ex->insn = jmp_offset;
+				ex->fixup = addrs[i + 1 + insn[1].off] - ra;
+				ex->data = EX_TYPE_BPF;
+				break;
+			}
 			if (priv_frame_ptr)
 				pop_r9(&prog);
 			break;
@@ -3772,13 +3885,17 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			break;
 
 		case BPF_JMP | BPF_EXIT:
+			unw = unwind_exit(env, bpf_prog, i - 1);
+			if (unw != EXIT_RETURNS &&
+			    emit_unwind_ret_addr(&prog, bpf_prog, ip, unw == EXIT_LOOK_AT_R3))
+				return -EINVAL;
 			if (seen_exit) {
 				jmp_offset = ctx->cleanup_addr - addrs[i];
 				goto emit_jmp;
 			}
 			seen_exit = true;
 			/* Update cleanup_addr */
-			ctx->cleanup_addr = proglen;
+			ctx->cleanup_addr = proglen + (prog - temp);
 			if (bpf_prog_was_classic(bpf_prog) &&
 			    !ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN)) {
 				if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
@@ -3811,6 +3928,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			return -EINVAL;
 		}
 
+insn_done:
 		ilen = prog - temp;
 		if (ilen > BPF_MAX_INSN_SIZE) {
 			pr_err("bpf_jit: fatal insn size error\n");
@@ -4864,6 +4982,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
 		    is_atomic_fetch_op(insn))
 			prog->aux->num_exentries++;
 	}
+	for (i = 0; i < prog->len; i++)
+		if (unwind_site(env, prog, i))
+			prog->aux->num_exentries++;
 
 	addrs = kvmalloc_objs(*addrs, prog->len + 1);
 	if (!addrs)