Skip to contentSkip to conversation

BPF Standard Streams

RFC PATCH In review Kumar Kartikeya Dwivedi 6 participants 44 messages Apr 14, 2025 – May 7, 2025

Subsystems (10)
Memory management x86 Kselftest BPF +6 more subsystems
lore
Share & tools
lore

Kumar Kartikeya Dwivedi

author maintainer
#0 lore raw

cover BPF Standard Streams

NOTE: Temporarily based off of bpf-next/net due to Qdisc related fixes.

This set introduces a standard output interface with two streams, namely stdout and stderr, for BPF programs. The idea is that these streams will be written to by BPF programs and the kernel, and serve as standard interfaces for informing user space of any BPF runtime violations. Users can also utilize them for printing normal messages for debugging usage, as is the case with bpf_printk() and trace pipe interface.

BPF programs and the kernel can use these streams to output messages. User space can dump these messages using bpftool.

The stream interface itself is implemented using a lockless list, so that we can queue messages from any context. Every printk statement into the stream leads to memory allocation. Allocation itself relies on the bpf_mem_alloc() API, but in case it fails, we use try_alloc_pages() to construct a bespoke bump allocator to carve out elements. If this fails, we finally give up and drop the message.

Three scenarios (rqspinlock deadlocks, arena page faults, and cond_break timeouts) are covered for reporting errors to user space, the format looks as follows (with file:line info for BPF programs to pin point source of error). The output is as follows:

ERROR: AA or ABBA deadlock detected for bpf_res_spin_lock
Attempted lock = 0xff11000104a875e0
Total held locks = 1
Held lock[ 0] = 0xff11000104a875e0

CPU: 5 UID: 0 PID: 764 Comm: test_progs
Call trace:
 bpf_prog_stderr_dump_stack+0x83/0xc0
 bpf_res_spin_lock+0x8c/0xa0
 bpf_prog_0f369a1fb6eb23d7_foo+0xe5/0x140: [stream_bpftool.c:62]
 bpf_prog_986cfd896eb75e8c_stream_bpftool_dump_prog_stream+0x7e/0x2e1: [stream_bpftool.c:93]
 bpf_prog_test_run_syscall+0x102/0x240
 __sys_bpf+0xd68/0x2bf0
 __x64_sys_bpf+0x1e/0x30
 do_syscall_64+0x68/0x140
 entry_SYSCALL_64_after_hwframe+0x76/0x7e

ERROR: Arena READ access at unmapped address 0xdeadbeef

CPU: 5 UID: 0 PID: 764 Comm: test_progs
Call trace:
 bpf_prog_stderr_dump_stack+0x83/0xc0
 ex_handler_bpf+0x4a/0xa0
 fixup_exception+0xde/0x310
 kernelmode_fixup_or_oops.constprop.0+0x2f/0x70
 exc_page_fault+0xdd/0x1d0
 asm_exc_page_fault+0x26/0x30
 bpf_prog_0f369a1fb6eb23d7_foo+0x10c/0x140: [stream_bpftool.c:67]
 bpf_prog_986cfd896eb75e8c_stream_bpftool_dump_prog_stream+0x7e/0x2e1: [stream_bpftool.c:93]
 bpf_prog_test_run_syscall+0x102/0x240
 __sys_bpf+0xd68/0x2bf0
 __x64_sys_bpf+0x1e/0x30
 do_syscall_64+0x68/0x140
 entry_SYSCALL_64_after_hwframe+0x76/0x7e

ERROR: Arena WRITE access at unmapped address 0xfaceb00c

CPU: 5 UID: 0 PID: 764 Comm: test_progs
Call trace:
 bpf_prog_stderr_dump_stack+0x83/0xc0
 ex_handler_bpf+0x4a/0xa0
 fixup_exception+0xde/0x310
 kernelmode_fixup_or_oops.constprop.0+0x2f/0x70
 exc_page_fault+0xdd/0x1d0
 asm_exc_page_fault+0x26/0x30
 bpf_prog_0f369a1fb6eb23d7_foo+0x111/0x140: [stream_bpftool.c:67]
 bpf_prog_986cfd896eb75e8c_stream_bpftool_dump_prog_stream+0x7e/0x2e1: [stream_bpftool.c:93]
 bpf_prog_test_run_syscall+0x102/0x240
 __sys_bpf+0xd68/0x2bf0
 __x64_sys_bpf+0x1e/0x30
 do_syscall_64+0x68/0x140
 entry_SYSCALL_64_after_hwframe+0x76/0x7e

ERROR: Timeout detected for may_goto instruction

CPU: 5 UID: 0 PID: 764 Comm: test_progs
Call trace:
 bpf_prog_stderr_dump_stack+0x83/0xc0
 bpf_check_timed_may_goto+0x4d/0xa0
 arch_bpf_timed_may_goto+0x21/0x40
 bpf_prog_0f369a1fb6eb23d7_foo+0x12f/0x140: [stream_bpftool.c:70]
 bpf_prog_986cfd896eb75e8c_stream_bpftool_dump_prog_stream+0x7e/0x2e1: [stream_bpftool.c:93]
 bpf_prog_test_run_syscall+0x102/0x240
 __sys_bpf+0xd68/0x2bf0
 __x64_sys_bpf+0x1e/0x30
 do_syscall_64+0x68/0x140
 entry_SYSCALL_64_after_hwframe+0x76/0x7e

There is a lot of minor things to address before this is mergeable, but the main logic and approach should be set in stone.

TODO (in order of importance)
-----------------------------
* Add an API for kernel writers to stderr streams to allow them to stage messages to a llist_head and then atomically splice in a bunch of messages into the stream to avoid interleaving from multiple threads. Each "message" consisisting of multiple printk statements then appears as a contiguous sequence to the user space. This will simply be a stage + commit API on top of an object pinning the llist_head that will be del_all'd and then add_batch'd to the stream->log.
* Add bpf_printk() macros for streams.
* Add kdoc comments for kfuncs.
* Add more corner case tests, and negative cases.
* Enforcing memory consumption limits on the log usage, so that in-flight log memory is bounded at runtime.
* Support bpftool filepath printing and JSON output.

Alexei Starovoitov (1):
locking/local_lock, mm: Replace localtry_ helpers with local_trylock_t
type

Kumar Kartikeya Dwivedi (12):
bpf: Tie dynptrs to referenced source objects
bpf: Compare dynptr_id in regsafe
selftests/bpf: Convert dynptr_fail to use vmlinux.h selftests/bpf: Add tests for dynptr source object interaction
bpf: Introduce bpf_dynptr_from_mem_slice
bpf: Introduce per-prog stdout/stderr streams
bpf: Add dump_stack() analogue to print to BPF stderr
bpf: Report may_goto timeout to BPF stderr
bpf: Report rqspinlock deadlocks/timeout to BPF stderr
bpf: Report arena faults to BPF stderr
bpftool: Add support for dumping streams selftests/bpf: Add tests for prog streams

 arch/x86/net/bpf_jit_comp.c                   |  21 +-
 include/linux/bpf.h                           |  63 +-
 include/linux/local_lock.h                    |  58 +-
 include/linux/local_lock_internal.h           | 207 +++---
 kernel/bpf/Makefile                           |   2 +-
 kernel/bpf/arena.c                            |  14 +
 kernel/bpf/core.c                             |  29 +-
 kernel/bpf/helpers.c                          |  58 +-
 kernel/bpf/rqspinlock.c                       |  22 +
 kernel/bpf/stream.c                           | 595 ++++++++++++++++++
 kernel/bpf/syscall.c                          |   2 +-
 kernel/bpf/verifier.c                         | 104 ++-
 mm/memcontrol.c                               |  39 +-
 tools/bpf/bpftool/Makefile                    |   2 +-
 tools/bpf/bpftool/prog.c                      |  71 ++-
 tools/bpf/bpftool/skeleton/stream.bpf.c       |  96 +++
 .../testing/selftests/bpf/prog_tests/dynptr.c |   2 +
 .../testing/selftests/bpf/prog_tests/stream.c |  57 ++
 .../testing/selftests/bpf/progs/dynptr_fail.c |  43 +-
 .../selftests/bpf/progs/dynptr_fail_qdisc.c   |  38 ++
 tools/testing/selftests/bpf/progs/stream.c    | 150 +++++
 .../selftests/bpf/progs/stream_bpftool.c      | 142 +++++
 .../testing/selftests/bpf/progs/stream_fail.c |  38 ++
 23 files changed, 1597 insertions(+), 256 deletions(-)
 create mode 100644 kernel/bpf/stream.c
 create mode 100644 tools/bpf/bpftool/skeleton/stream.bpf.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stream.c
 create mode 100644 tools/testing/selftests/bpf/progs/dynptr_fail_qdisc.c
 create mode 100644 tools/testing/selftests/bpf/progs/stream.c
 create mode 100644 tools/testing/selftests/bpf/progs/stream_bpftool.c
 create mode 100644 tools/testing/selftests/bpf/progs/stream_fail.c

base-commit: 93cb5375fb16693273144e7627e4fe27b811003a

Signature · 1 line
2.47.1
author maintainer
#1 lore raw

RFC PATCH 1/13 bpf: Tie dynptrs to referenced source objects

Ensure that for dynptr constructors (MEM_UNINIT arg_type) taking a referenced object (ref_obj_id) as their memory source, we set the ref_obj_id of the dynptr appropriately as well. This allows us to tie the lifetime of the dynptr to its source and properly invalidate it when the source object is released.

For helpers, we don't have such cases yet as bpf_dynptr_from_mem is not permitted for anything other than PTR_TO_MAP_VALUE, but still pass meta->ref_obj_id as clone_ref_obj_id in case this is relaxed in future. Since they are ossified we know dynptr_from_mem is the only relevant helper and takes one memory argument, so we know the meta->ref_obj_id if non-zero will belong to it.

For kfuncs, make sure for constructors, only 1 ref_obj_id argument is seen, as more than one can be ambiguous in terms of ref_obj_id transfer. Since more kfuncs can be added with possibly multiple memory arguments, make sure meta->ref_obj_id reassignment won't cause incorrect lifetime analysis in the future using ref_obj_cnt logic. Set this ref_obj_id as the clone_ref_obj_id, so that it is transferred to the spilled_ptr stack slot register state.

Add support to unmark_stack_slots_dynptr to not descend to its child slices (using bool slice parameter) so that we don't have circular calls when invoking release_reference. With this logic in place, we may have the following object relationships:

				     +-- slice 1 (ref=1)
 source (ref=1) --> dynptr (ref=1) --|-- slice 2 (ref=1)
				     +-- slice 3 (ref=1)

Destroying dynptr prunes the dynptr and all its children slices, but does not affect the source. Releasing the source will effectively prune the entire ownership tree. Dynptr clones with same ref=1 will be parallel in the ownership tree.

		  +-- orig  dptr (ref=1) --> slice 1 (ref=1)
 source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1)
		  +-- clone dptr (ref=1) --> slice 3 (ref=1)

In such a case, only child slices of the dynptr clone being destroyed are invalidated. Likewise, if the source object goes away, the whole tree ends up getting pruned.

Cc Amery Hung <[email protected]>
Fixes 81bb1c2c3e8e ("bpf: net_sched: Add basic bpf qdisc kfuncs")
---
 kernel/bpf/verifier.c | 81 ++++++++++++++++++++++++++++---------------
 1 file changed, 54 insertions(+), 27 deletions(-)
kernel/bpf/verifier.c +54 -27
index 54c6953a8b84..a62dfab9aea6 100644
@@ -198,7 +198,7 @@ struct bpf_verifier_stack_elem {
 
 static int acquire_reference(struct bpf_verifier_env *env, int insn_idx);
 static int release_reference_nomark(struct bpf_verifier_state *state, int ref_obj_id);
-static int release_reference(struct bpf_verifier_env *env, int ref_obj_id);
+static int release_reference(struct bpf_verifier_env *env, int ref_obj_id, bool objects);
 static void invalidate_non_owning_refs(struct bpf_verifier_env *env);
 static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env);
 static int ref_set_non_owning(struct bpf_verifier_env *env,
@@ -299,6 +299,7 @@ struct bpf_kfunc_call_arg_meta {
 	const char *func_name;
 	/* Out parameters */
 	u32 ref_obj_id;
+	u32 ref_obj_cnt;
 	u8 release_regno;
 	bool r0_rdonly;
 	u32 ret_btf_id;
@@ -759,7 +760,7 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_
 	mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr,
 			       &state->stack[spi - 1].spilled_ptr, type);
 
-	if (dynptr_type_refcounted(type)) {
+	if (dynptr_type_refcounted(type) || clone_ref_obj_id) {
 		/* The id is used to track proper releasing */
 		int id;
 
@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat
 	state->stack[spi - 1].spilled_ptr.live |= REG_LIVE_WRITTEN;
 }
 
-static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
+static int __unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_func_state *state,
+				       int spi, bool slice)
 {
-	struct bpf_func_state *state = func(env, reg);
-	int spi, ref_obj_id, i;
+	u32 ref_obj_id;
+	int i;
 
-	spi = dynptr_get_spi(env, reg);
-	if (spi < 0)
-		return spi;
+	ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
 
-	if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) {
+	if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {
 		invalidate_dynptr(env, state, spi);
 		return 0;
 	}
 
-	ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
-
 	/* If the dynptr has a ref_obj_id, then we need to invalidate
 	 * two things:
 	 *
@@ -842,7 +840,8 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
 	 */
 
 	/* Invalidate any slices associated with this dynptr */
-	WARN_ON_ONCE(release_reference(env, ref_obj_id));
+	if (slice)
+		WARN_ON_ONCE(release_reference(env, ref_obj_id, false));
 
 	/* Invalidate any dynptr clones */
 	for (i = 1; i < state->allocated_stack / BPF_REG_SIZE; i++) {
@@ -864,6 +863,18 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
 	return 0;
 }
 
+static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg, bool slice)
+{
+	struct bpf_func_state *state = func(env, reg);
+	int spi;
+
+	spi = dynptr_get_spi(env, reg);
+	if (spi < 0)
+		return spi;
+
+	return __unmark_stack_slots_dynptr(env, state, spi, slice);
+}
+
 static void __mark_reg_unknown(const struct bpf_verifier_env *env,
 			       struct bpf_reg_state *reg);
 
@@ -1075,7 +1086,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,
 		struct bpf_reg_state *st = &slot->spilled_ptr;
 
 		if (i == 0)
-			WARN_ON_ONCE(release_reference(env, st->ref_obj_id));
+			WARN_ON_ONCE(release_reference(env, st->ref_obj_id, false));
 
 		__mark_reg_not_init(env, st);
 
@@ -9749,7 +9760,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg,
 					 true, meta);
 		break;
 	case ARG_PTR_TO_DYNPTR:
-		err = process_dynptr_func(env, regno, insn_idx, arg_type, 0);
+		err = process_dynptr_func(env, regno, insn_idx, arg_type, meta->ref_obj_id);
 		if (err)
 			return err;
 		break;
@@ -10220,12 +10231,12 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob
  *
  * This is the release function corresponding to acquire_reference(). Idempotent.
  */
-static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)
+static int release_reference(struct bpf_verifier_env *env, int ref_obj_id, bool objects)
 {
 	struct bpf_verifier_state *vstate = env->cur_state;
 	struct bpf_func_state *state;
 	struct bpf_reg_state *reg;
-	int err;
+	int err, spi;
 
 	err = release_reference_nomark(vstate, ref_obj_id);
 	if (err)
@@ -10236,6 +10247,19 @@ static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)
 			mark_reg_invalid(env, reg);
 	}));
 
+	if (!objects)
+		return 0;
+
+	bpf_for_each_spilled_reg(spi, state, reg, (1 << STACK_DYNPTR)) {
+		if (!reg)
+			continue;
+		if (!reg->dynptr.first_slot || reg->ref_obj_id != ref_obj_id)
+			continue;
+		err = __unmark_stack_slots_dynptr(env, state, spi, false);
+		if (err)
+			return err;
+	}
+
 	return 0;
 }
 
@@ -11357,7 +11381,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 				verbose(env, "verifier internal error: CONST_PTR_TO_DYNPTR cannot be released\n");
 				return -EFAULT;
 			}
-			err = unmark_stack_slots_dynptr(env, &regs[meta.release_regno]);
+			err = unmark_stack_slots_dynptr(env, &regs[meta.release_regno], true);
 		} else if (func_id == BPF_FUNC_kptr_xchg && meta.ref_obj_id) {
 			u32 ref_obj_id = meta.ref_obj_id;
 			bool in_rcu = in_rcu_cs(env);
@@ -11379,7 +11403,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 				}));
 			}
 		} else if (meta.ref_obj_id) {
-			err = release_reference(env, meta.ref_obj_id);
+			err = release_reference(env, meta.ref_obj_id, true);
 		} else if (register_is_null(&regs[meta.release_regno])) {
 			/* meta.ref_obj_id can only be 0 if register that is meant to be
 			 * released is NULL, which must be > R0.
@@ -12974,6 +12998,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 			meta->ref_obj_id = reg->ref_obj_id;
 			if (is_kfunc_release(meta))
 				meta->release_regno = regno;
+			meta->ref_obj_cnt++;
 		}
 
 		ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id);
@@ -13100,13 +13125,19 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 		case KF_ARG_PTR_TO_DYNPTR:
 		{
 			enum bpf_arg_type dynptr_arg_type = ARG_PTR_TO_DYNPTR;
-			int clone_ref_obj_id = 0;
+			int clone_ref_obj_id = meta->ref_obj_id;
 
 			if (reg->type == CONST_PTR_TO_DYNPTR)
 				dynptr_arg_type |= MEM_RDONLY;
 
-			if (is_kfunc_arg_uninit(btf, &args[i]))
+			if (is_kfunc_arg_uninit(btf, &args[i])) {
 				dynptr_arg_type |= MEM_UNINIT;
+				/* It's confusing if dynptr constructor takes multiple referenced arguments. */
+				if (meta->ref_obj_cnt > 1) {
+					verbose(env, "verifier internal error: multiple referenced arguments\n");
+					return -EFAULT;
+				}
+			}
 
 			if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) {
 				dynptr_arg_type |= DYNPTR_TYPE_SKB;
@@ -13582,7 +13613,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	 * PTR_TO_BTF_ID in bpf_kfunc_arg_meta, do the release now.
 	 */
 	if (meta.release_regno) {
-		err = release_reference(env, regs[meta.release_regno].ref_obj_id);
+		err = release_reference(env, regs[meta.release_regno].ref_obj_id, true);
 		if (err) {
 			verbose(env, "kfunc %s#%d reference has not been acquired before\n",
 				func_name, meta.func_id);
@@ -13603,7 +13634,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			return err;
 		}
 
-		err = release_reference(env, release_ref_obj_id);
+		err = release_reference(env, release_ref_obj_id, true);
 		if (err) {
 			verbose(env, "kfunc %s#%d reference has not been acquired before\n",
 				func_name, meta.func_id);
@@ -13803,11 +13834,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 					return -EFAULT;
 				}
 				regs[BPF_REG_0].dynptr_id = meta.initialized_dynptr.id;
-
-				/* we don't need to set BPF_REG_0's ref obj id
-				 * because packet slices are not refcounted (see
-				 * dynptr_type_refcounted)
-				 */
+				regs[BPF_REG_0].ref_obj_id = meta.initialized_dynptr.ref_obj_id;
 			} else {
 				verbose(env, "kernel function %s unhandled dynamic return type\n",
 					meta.func_name);
Signature · 1 line
2.47.1
#2 lore raw

Re: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects

Quoted 110 lines · Kumar Kartikeya Dwivedi

Ensure that for dynptr constructors (MEM_UNINIT arg_type) taking a referenced object (ref_obj_id) as their memory source, we set the ref_obj_id of the dynptr appropriately as well. This allows us to tie the lifetime of the dynptr to its source and properly invalidate it when the source object is released.

For helpers, we don't have such cases yet as bpf_dynptr_from_mem is not permitted for anything other than PTR_TO_MAP_VALUE, but still pass meta->ref_obj_id as clone_ref_obj_id in case this is relaxed in future. Since they are ossified we know dynptr_from_mem is the only relevant helper and takes one memory argument, so we know the meta->ref_obj_id if non-zero will belong to it.

For kfuncs, make sure for constructors, only 1 ref_obj_id argument is seen, as more than one can be ambiguous in terms of ref_obj_id transfer. Since more kfuncs can be added with possibly multiple memory arguments, make sure meta->ref_obj_id reassignment won't cause incorrect lifetime analysis in the future using ref_obj_cnt logic. Set this ref_obj_id as the clone_ref_obj_id, so that it is transferred to the spilled_ptr stack slot register state.

Add support to unmark_stack_slots_dynptr to not descend to its child slices (using bool slice parameter) so that we don't have circular calls when invoking release_reference. With this logic in place, we may have the following object relationships:

                                     +-- slice 1 (ref=1)
 source (ref=1) --> dynptr (ref=1) --|-- slice 2 (ref=1)
                                     +-- slice 3 (ref=1)

Destroying dynptr prunes the dynptr and all its children slices, but does not affect the source. Releasing the source will effectively prune the entire ownership tree. Dynptr clones with same ref=1 will be parallel in the ownership tree.

                  +-- orig  dptr (ref=1) --> slice 1 (ref=1)
 source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1)
                  +-- clone dptr (ref=1) --> slice 3 (ref=1)

In such a case, only child slices of the dynptr clone being destroyed are invalidated. Likewise, if the source object goes away, the whole tree ends up getting pruned.

Cc Amery Hung <[email protected]>
Fixes 81bb1c2c3e8e ("bpf: net_sched: Add basic bpf qdisc kfuncs")
---
 kernel/bpf/verifier.c | 81 ++++++++++++++++++++++++++++---------------
 1 file changed, 54 insertions(+), 27 deletions(-)
kernel/bpf/verifier.c +9 -11
index 54c6953a8b84..a62dfab9aea6 100644
@@ -198,7 +198,7 @@ struct bpf_verifier_stack_elem {
 
 static int acquire_reference(struct bpf_verifier_env *env, int insn_idx);
 static int release_reference_nomark(struct bpf_verifier_state *state, int ref_obj_id);
-static int release_reference(struct bpf_verifier_env *env, int ref_obj_id);
+static int release_reference(struct bpf_verifier_env *env, int ref_obj_id, bool objects);
 static void invalidate_non_owning_refs(struct bpf_verifier_env *env);
 static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env);
 static int ref_set_non_owning(struct bpf_verifier_env *env,
@@ -299,6 +299,7 @@ struct bpf_kfunc_call_arg_meta {
        const char *func_name;
        /* Out parameters */
        u32 ref_obj_id;
+       u32 ref_obj_cnt;
        u8 release_regno;
        bool r0_rdonly;
        u32 ret_btf_id;
@@ -759,7 +760,7 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_
        mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr,
                               &state->stack[spi - 1].spilled_ptr, type);
 
-       if (dynptr_type_refcounted(type)) {
+       if (dynptr_type_refcounted(type) || clone_ref_obj_id) {
                /* The id is used to track proper releasing */
                int id;
 
@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat
        state->stack[spi - 1].spilled_ptr.live |= REG_LIVE_WRITTEN;
 }
 
-static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
+static int __unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_func_state *state,
+                                      int spi, bool slice)
 {
-       struct bpf_func_state *state = func(env, reg);
-       int spi, ref_obj_id, i;
+       u32 ref_obj_id;
+       int i;
 
-       spi = dynptr_get_spi(env, reg);
-       if (spi < 0)
-               return spi;
+       ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
 
-       if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) {
+       if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {
                invalidate_dynptr(env, state, spi);
                return 0;
        }
 
-       ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
-
        /* If the dynptr has a ref_obj_id, then we need to invalidate
         * two things:
         *

kernel/bpf/verifier.c static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re · line 844

Quoted 6 lines
kernel/bpf/verifier.c +2 -1
@@ -842,7 +840,8 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
         */
 
        /* Invalidate any slices associated with this dynptr */
-       WARN_ON_ONCE(release_reference(env, ref_obj_id));
+       if (slice)
+               WARN_ON_ONCE(release_reference(env, ref_obj_id, false));

I think the slice argument might not be needed if we do the following instead:

        if (dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type))
                WARN_ON_ONCE(release_reference(env, ref_obj_id, false));

When bpf_kfree_skb():

    -> release_reference(..., object = true)
      -> __unmark_stack_slots_dynptr()
        -> not calling into release_reference() since

dynptr_type_refcounted() returns false

When bpf_ringbuf_submit():

    -> unmark_stack_slots_dynptr()
      -> release_reference(..., object = false)
        -> not calling __unmark_stack_slots_dynptr() since object == false

Am I missing anything?

Quoted 94 lines
kernel/bpf/verifier.c +31 -6
        /* Invalidate any dynptr clones */
        for (i = 1; i < state->allocated_stack / BPF_REG_SIZE; i++) {
@@ -864,6 +863,18 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
        return 0;
 }
 
+static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg, bool slice)
+{
+       struct bpf_func_state *state = func(env, reg);
+       int spi;
+
+       spi = dynptr_get_spi(env, reg);
+       if (spi < 0)
+               return spi;
+
+       return __unmark_stack_slots_dynptr(env, state, spi, slice);
+}
+
 static void __mark_reg_unknown(const struct bpf_verifier_env *env,
                               struct bpf_reg_state *reg);
 
@@ -1075,7 +1086,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,
                struct bpf_reg_state *st = &slot->spilled_ptr;
 
                if (i == 0)
-                       WARN_ON_ONCE(release_reference(env, st->ref_obj_id));
+                       WARN_ON_ONCE(release_reference(env, st->ref_obj_id, false));
 
                __mark_reg_not_init(env, st);
 
@@ -9749,7 +9760,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg,
                                         true, meta);
                break;
        case ARG_PTR_TO_DYNPTR:
-               err = process_dynptr_func(env, regno, insn_idx, arg_type, 0);
+               err = process_dynptr_func(env, regno, insn_idx, arg_type, meta->ref_obj_id);
                if (err)
                        return err;
                break;
@@ -10220,12 +10231,12 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob
  *
  * This is the release function corresponding to acquire_reference(). Idempotent.
  */
-static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)
+static int release_reference(struct bpf_verifier_env *env, int ref_obj_id, bool objects)
 {
        struct bpf_verifier_state *vstate = env->cur_state;
        struct bpf_func_state *state;
        struct bpf_reg_state *reg;
-       int err;
+       int err, spi;
 
        err = release_reference_nomark(vstate, ref_obj_id);
        if (err)
@@ -10236,6 +10247,19 @@ static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)
                        mark_reg_invalid(env, reg);
        }));
 
+       if (!objects)
+               return 0;
+
+       bpf_for_each_spilled_reg(spi, state, reg, (1 << STACK_DYNPTR)) {
+               if (!reg)
+                       continue;
+               if (!reg->dynptr.first_slot || reg->ref_obj_id != ref_obj_id)
+                       continue;
+               err = __unmark_stack_slots_dynptr(env, state, spi, false);
+               if (err)
+                       return err;
+       }
+
        return 0;
 }
 
@@ -11357,7 +11381,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
                                verbose(env, "verifier internal error: CONST_PTR_TO_DYNPTR cannot be released\n");
                                return -EFAULT;
                        }
-                       err = unmark_stack_slots_dynptr(env, &regs[meta.release_regno]);
+                       err = unmark_stack_slots_dynptr(env, &regs[meta.release_regno], true);
                } else if (func_id == BPF_FUNC_kptr_xchg && meta.ref_obj_id) {
                        u32 ref_obj_id = meta.ref_obj_id;
                        bool in_rcu = in_rcu_cs(env);
@@ -11379,7 +11403,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
                                }));
                        }
                } else if (meta.ref_obj_id) {
-                       err = release_reference(env, meta.ref_obj_id);
+                       err = release_reference(env, meta.ref_obj_id, true);
                } else if (register_is_null(&regs[meta.release_regno])) {
                        /* meta.ref_obj_id can only be 0 if register that is meant to be
                         * released is NULL, which must be > R0.

kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 13001

Quoted 4 lines
kernel/bpf/verifier.c +1 -0
@@ -12974,6 +12998,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
                        meta->ref_obj_id = reg->ref_obj_id;
                        if (is_kfunc_release(meta))
                                meta->release_regno = regno;
+                       meta->ref_obj_cnt++;

Does it make sense to convert other "more than one arg with ref_obj_id" checks to using ref_obj_cnt to make it more consistent?

Thanks for fixing the bug!
Amery

Quoted 59 lines
kernel/bpf/verifier.c +11 -10
                }
 
                ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id);
@@ -13100,13 +13125,19 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
                case KF_ARG_PTR_TO_DYNPTR:
                {
                        enum bpf_arg_type dynptr_arg_type = ARG_PTR_TO_DYNPTR;
-                       int clone_ref_obj_id = 0;
+                       int clone_ref_obj_id = meta->ref_obj_id;
 
                        if (reg->type == CONST_PTR_TO_DYNPTR)
                                dynptr_arg_type |= MEM_RDONLY;
 
-                       if (is_kfunc_arg_uninit(btf, &args[i]))
+                       if (is_kfunc_arg_uninit(btf, &args[i])) {
                                dynptr_arg_type |= MEM_UNINIT;
+                               /* It's confusing if dynptr constructor takes multiple referenced arguments. */
+                               if (meta->ref_obj_cnt > 1) {
+                                       verbose(env, "verifier internal error: multiple referenced arguments\n");
+                                       return -EFAULT;
+                               }
+                       }
 
                        if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) {
                                dynptr_arg_type |= DYNPTR_TYPE_SKB;
@@ -13582,7 +13613,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
         * PTR_TO_BTF_ID in bpf_kfunc_arg_meta, do the release now.
         */
        if (meta.release_regno) {
-               err = release_reference(env, regs[meta.release_regno].ref_obj_id);
+               err = release_reference(env, regs[meta.release_regno].ref_obj_id, true);
                if (err) {
                        verbose(env, "kfunc %s#%d reference has not been acquired before\n",
                                func_name, meta.func_id);
@@ -13603,7 +13634,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                        return err;
                }
 
-               err = release_reference(env, release_ref_obj_id);
+               err = release_reference(env, release_ref_obj_id, true);
                if (err) {
                        verbose(env, "kfunc %s#%d reference has not been acquired before\n",
                                func_name, meta.func_id);
@@ -13803,11 +13834,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                                        return -EFAULT;
                                }
                                regs[BPF_REG_0].dynptr_id = meta.initialized_dynptr.id;
-
-                               /* we don't need to set BPF_REG_0's ref obj id
-                                * because packet slices are not refcounted (see
-                                * dynptr_type_refcounted)
-                                */
+                               regs[BPF_REG_0].ref_obj_id = meta.initialized_dynptr.ref_obj_id;
                        } else {
                                verbose(env, "kernel function %s unhandled dynamic return type\n",
                                        meta.func_name);
--

2.47.1

author maintainer
#3 lore raw

Re: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects

Quoted 128 lines · Amery Hung
Quoted 110 lines · Kumar Kartikeya Dwivedi

Ensure that for dynptr constructors (MEM_UNINIT arg_type) taking a referenced object (ref_obj_id) as their memory source, we set the ref_obj_id of the dynptr appropriately as well. This allows us to tie the lifetime of the dynptr to its source and properly invalidate it when the source object is released.

For helpers, we don't have such cases yet as bpf_dynptr_from_mem is not permitted for anything other than PTR_TO_MAP_VALUE, but still pass meta->ref_obj_id as clone_ref_obj_id in case this is relaxed in future. Since they are ossified we know dynptr_from_mem is the only relevant helper and takes one memory argument, so we know the meta->ref_obj_id if non-zero will belong to it.

For kfuncs, make sure for constructors, only 1 ref_obj_id argument is seen, as more than one can be ambiguous in terms of ref_obj_id transfer. Since more kfuncs can be added with possibly multiple memory arguments, make sure meta->ref_obj_id reassignment won't cause incorrect lifetime analysis in the future using ref_obj_cnt logic. Set this ref_obj_id as the clone_ref_obj_id, so that it is transferred to the spilled_ptr stack slot register state.

Add support to unmark_stack_slots_dynptr to not descend to its child slices (using bool slice parameter) so that we don't have circular calls when invoking release_reference. With this logic in place, we may have the following object relationships:

                                     +-- slice 1 (ref=1)
 source (ref=1) --> dynptr (ref=1) --|-- slice 2 (ref=1)
                                     +-- slice 3 (ref=1)

Destroying dynptr prunes the dynptr and all its children slices, but does not affect the source. Releasing the source will effectively prune the entire ownership tree. Dynptr clones with same ref=1 will be parallel in the ownership tree.

                  +-- orig  dptr (ref=1) --> slice 1 (ref=1)
 source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1)
                  +-- clone dptr (ref=1) --> slice 3 (ref=1)

In such a case, only child slices of the dynptr clone being destroyed are invalidated. Likewise, if the source object goes away, the whole tree ends up getting pruned.

Cc Amery Hung <[email protected]>
Fixes 81bb1c2c3e8e ("bpf: net_sched: Add basic bpf qdisc kfuncs")
---
 kernel/bpf/verifier.c | 81 ++++++++++++++++++++++++++++---------------
 1 file changed, 54 insertions(+), 27 deletions(-)
kernel/bpf/verifier.c +9 -11
index 54c6953a8b84..a62dfab9aea6 100644
@@ -198,7 +198,7 @@ struct bpf_verifier_stack_elem {
 
 static int acquire_reference(struct bpf_verifier_env *env, int insn_idx);
 static int release_reference_nomark(struct bpf_verifier_state *state, int ref_obj_id);
-static int release_reference(struct bpf_verifier_env *env, int ref_obj_id);
+static int release_reference(struct bpf_verifier_env *env, int ref_obj_id, bool objects);
 static void invalidate_non_owning_refs(struct bpf_verifier_env *env);
 static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env);
 static int ref_set_non_owning(struct bpf_verifier_env *env,
@@ -299,6 +299,7 @@ struct bpf_kfunc_call_arg_meta {
        const char *func_name;
        /* Out parameters */
        u32 ref_obj_id;
+       u32 ref_obj_cnt;
        u8 release_regno;
        bool r0_rdonly;
        u32 ret_btf_id;
@@ -759,7 +760,7 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_
        mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr,
                               &state->stack[spi - 1].spilled_ptr, type);
 
-       if (dynptr_type_refcounted(type)) {
+       if (dynptr_type_refcounted(type) || clone_ref_obj_id) {
                /* The id is used to track proper releasing */
                int id;
 
@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat
        state->stack[spi - 1].spilled_ptr.live |= REG_LIVE_WRITTEN;
 }
 
-static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
+static int __unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_func_state *state,
+                                      int spi, bool slice)
 {
-       struct bpf_func_state *state = func(env, reg);
-       int spi, ref_obj_id, i;
+       u32 ref_obj_id;
+       int i;
 
-       spi = dynptr_get_spi(env, reg);
-       if (spi < 0)
-               return spi;
+       ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
 
-       if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) {
+       if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {
                invalidate_dynptr(env, state, spi);
                return 0;
        }
 
-       ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
-
        /* If the dynptr has a ref_obj_id, then we need to invalidate
         * two things:
         *

kernel/bpf/verifier.c static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re · line 844

Quoted 6 lines
kernel/bpf/verifier.c +2 -1
@@ -842,7 +840,8 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
         */
 
        /* Invalidate any slices associated with this dynptr */
-       WARN_ON_ONCE(release_reference(env, ref_obj_id));
+       if (slice)
+               WARN_ON_ONCE(release_reference(env, ref_obj_id, false));

I think the slice argument might not be needed if we do the following instead:

        if (dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type))
                WARN_ON_ONCE(release_reference(env, ref_obj_id, false));

When bpf_kfree_skb():

    -> release_reference(..., object = true)
      -> __unmark_stack_slots_dynptr()
        -> not calling into release_reference() since
Quoted 8 lines

dynptr_type_refcounted() returns false

When bpf_ringbuf_submit():

    -> unmark_stack_slots_dynptr()
      -> release_reference(..., object = false)
        -> not calling __unmark_stack_slots_dynptr() since object == false

Am I missing anything?

That is true, I will probably try to rework it the way Andrii suggested.

Quoted 94 lines
Quoted 94 lines
kernel/bpf/verifier.c +31 -6
        /* Invalidate any dynptr clones */
        for (i = 1; i < state->allocated_stack / BPF_REG_SIZE; i++) {
@@ -864,6 +863,18 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
        return 0;
 }
 
+static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg, bool slice)
+{
+       struct bpf_func_state *state = func(env, reg);
+       int spi;
+
+       spi = dynptr_get_spi(env, reg);
+       if (spi < 0)
+               return spi;
+
+       return __unmark_stack_slots_dynptr(env, state, spi, slice);
+}
+
 static void __mark_reg_unknown(const struct bpf_verifier_env *env,
                               struct bpf_reg_state *reg);
 
@@ -1075,7 +1086,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,
                struct bpf_reg_state *st = &slot->spilled_ptr;
 
                if (i == 0)
-                       WARN_ON_ONCE(release_reference(env, st->ref_obj_id));
+                       WARN_ON_ONCE(release_reference(env, st->ref_obj_id, false));
 
                __mark_reg_not_init(env, st);
 
@@ -9749,7 +9760,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg,
                                         true, meta);
                break;
        case ARG_PTR_TO_DYNPTR:
-               err = process_dynptr_func(env, regno, insn_idx, arg_type, 0);
+               err = process_dynptr_func(env, regno, insn_idx, arg_type, meta->ref_obj_id);
                if (err)
                        return err;
                break;
@@ -10220,12 +10231,12 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob
  *
  * This is the release function corresponding to acquire_reference(). Idempotent.
  */
-static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)
+static int release_reference(struct bpf_verifier_env *env, int ref_obj_id, bool objects)
 {
        struct bpf_verifier_state *vstate = env->cur_state;
        struct bpf_func_state *state;
        struct bpf_reg_state *reg;
-       int err;
+       int err, spi;
 
        err = release_reference_nomark(vstate, ref_obj_id);
        if (err)
@@ -10236,6 +10247,19 @@ static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)
                        mark_reg_invalid(env, reg);
        }));
 
+       if (!objects)
+               return 0;
+
+       bpf_for_each_spilled_reg(spi, state, reg, (1 << STACK_DYNPTR)) {
+               if (!reg)
+                       continue;
+               if (!reg->dynptr.first_slot || reg->ref_obj_id != ref_obj_id)
+                       continue;
+               err = __unmark_stack_slots_dynptr(env, state, spi, false);
+               if (err)
+                       return err;
+       }
+
        return 0;
 }
 
@@ -11357,7 +11381,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
                                verbose(env, "verifier internal error: CONST_PTR_TO_DYNPTR cannot be released\n");
                                return -EFAULT;
                        }
-                       err = unmark_stack_slots_dynptr(env, &regs[meta.release_regno]);
+                       err = unmark_stack_slots_dynptr(env, &regs[meta.release_regno], true);
                } else if (func_id == BPF_FUNC_kptr_xchg && meta.ref_obj_id) {
                        u32 ref_obj_id = meta.ref_obj_id;
                        bool in_rcu = in_rcu_cs(env);
@@ -11379,7 +11403,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
                                }));
                        }
                } else if (meta.ref_obj_id) {
-                       err = release_reference(env, meta.ref_obj_id);
+                       err = release_reference(env, meta.ref_obj_id, true);
                } else if (register_is_null(&regs[meta.release_regno])) {
                        /* meta.ref_obj_id can only be 0 if register that is meant to be
                         * released is NULL, which must be > R0.
Quoted 8 lines

kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 13001

Quoted 5 lines
kernel/bpf/verifier.c +1 -0
@@ -12974,6 +12998,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
                        meta->ref_obj_id = reg->ref_obj_id;
                        if (is_kfunc_release(meta))
                                meta->release_regno = regno;
+                       meta->ref_obj_cnt++;

Does it make sense to convert other "more than one arg with ref_obj_id" checks to using ref_obj_cnt to make it more consistent?

Ack, will change.

Quoted 63 lines

Thanks for fixing the bug!
Amery

Quoted 59 lines
kernel/bpf/verifier.c +11 -10
                }
 
                ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id);
@@ -13100,13 +13125,19 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
                case KF_ARG_PTR_TO_DYNPTR:
                {
                        enum bpf_arg_type dynptr_arg_type = ARG_PTR_TO_DYNPTR;
-                       int clone_ref_obj_id = 0;
+                       int clone_ref_obj_id = meta->ref_obj_id;
 
                        if (reg->type == CONST_PTR_TO_DYNPTR)
                                dynptr_arg_type |= MEM_RDONLY;
 
-                       if (is_kfunc_arg_uninit(btf, &args[i]))
+                       if (is_kfunc_arg_uninit(btf, &args[i])) {
                                dynptr_arg_type |= MEM_UNINIT;
+                               /* It's confusing if dynptr constructor takes multiple referenced arguments. */
+                               if (meta->ref_obj_cnt > 1) {
+                                       verbose(env, "verifier internal error: multiple referenced arguments\n");
+                                       return -EFAULT;
+                               }
+                       }
 
                        if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) {
                                dynptr_arg_type |= DYNPTR_TYPE_SKB;
@@ -13582,7 +13613,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
         * PTR_TO_BTF_ID in bpf_kfunc_arg_meta, do the release now.
         */
        if (meta.release_regno) {
-               err = release_reference(env, regs[meta.release_regno].ref_obj_id);
+               err = release_reference(env, regs[meta.release_regno].ref_obj_id, true);
                if (err) {
                        verbose(env, "kfunc %s#%d reference has not been acquired before\n",
                                func_name, meta.func_id);
@@ -13603,7 +13634,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                        return err;
                }
 
-               err = release_reference(env, release_ref_obj_id);
+               err = release_reference(env, release_ref_obj_id, true);
                if (err) {
                        verbose(env, "kfunc %s#%d reference has not been acquired before\n",
                                func_name, meta.func_id);
@@ -13803,11 +13834,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                                        return -EFAULT;
                                }
                                regs[BPF_REG_0].dynptr_id = meta.initialized_dynptr.id;
-
-                               /* we don't need to set BPF_REG_0's ref obj id
-                                * because packet slices are not refcounted (see
-                                * dynptr_type_refcounted)
-                                */
+                               regs[BPF_REG_0].ref_obj_id = meta.initialized_dynptr.ref_obj_id;
                        } else {
                                verbose(env, "kernel function %s unhandled dynamic return type\n",
                                        meta.func_name);
--

2.47.1

maintainer
#4 lore raw

Re: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects

In reply to Kumar Kartikeya Dwivedi

Kumar Kartikeya Dwivedi <[email protected]> writes:

[...]

Quoted 10 lines
kernel/bpf/verifier.c +2 -3
@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat
 	state->stack[spi - 1].spilled_ptr.live |= REG_LIVE_WRITTEN;
 }
 
-static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
+static int __unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_func_state *state,
+				       int spi, bool slice)
 {
-	struct bpf_func_state *state = func(env, reg);
-	int spi, ref_obj_id, i;

kernel/bpf/verifier.c static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat · line 830

Quoted 10 lines
kernel/bpf/verifier.c +4 -4
+	u32 ref_obj_id;
+	int i;
 
-	spi = dynptr_get_spi(env, reg);
-	if (spi < 0)
-		return spi;
+	ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
 
-	if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) {
+	if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {

If dynptr_type_refcounted is true, does this mean that ref_obj_id is set? If it does, the check could be simplified to just if (!ref_obj_id).

Quoted 9 lines
kernel/bpf/verifier.c +0 -2
 		invalidate_dynptr(env, state, spi);
 		return 0;
 	}
 
-	ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
-
 	/* If the dynptr has a ref_obj_id, then we need to invalidate
 	 * two things:
 	 *

[...]

author maintainer
#5 lore raw

Re: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects

Quoted 23 lines · Eduard Zingerman

Kumar Kartikeya Dwivedi <[email protected]> writes:

[...]

Quoted 18 lines
kernel/bpf/verifier.c +5 -6
@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat
      state->stack[spi - 1].spilled_ptr.live |= REG_LIVE_WRITTEN;
 }
 
-static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
+static int __unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_func_state *state,
+                                    int spi, bool slice)
 {
-     struct bpf_func_state *state = func(env, reg);
-     int spi, ref_obj_id, i;
+     u32 ref_obj_id;
+     int i;
 
-     spi = dynptr_get_spi(env, reg);
-     if (spi < 0)
-             return spi;
+     ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
 
Quoted 5 lines

kernel/bpf/verifier.c static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat · line 830

Quoted 2 lines
kernel/bpf/verifier.c +1 -1
-     if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) {
+     if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {

If dynptr_type_refcounted is true, does this mean that ref_obj_id is set? If it does, the check could be simplified to just if (!ref_obj_id).

Yes, I will change in non-RFC v1.

Quoted 12 lines
Quoted 10 lines
kernel/bpf/verifier.c +0 -2
              invalidate_dynptr(env, state, spi);
              return 0;
      }
 
-     ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
-
      /* If the dynptr has a ref_obj_id, then we need to invalidate
       * two things:
       *

[...]

#6 lore raw

Re: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects

In reply to Kumar Kartikeya Dwivedi
Quoted 72 lines · Kumar Kartikeya Dwivedi

Ensure that for dynptr constructors (MEM_UNINIT arg_type) taking a referenced object (ref_obj_id) as their memory source, we set the ref_obj_id of the dynptr appropriately as well. This allows us to tie the lifetime of the dynptr to its source and properly invalidate it when the source object is released.

For helpers, we don't have such cases yet as bpf_dynptr_from_mem is not permitted for anything other than PTR_TO_MAP_VALUE, but still pass meta->ref_obj_id as clone_ref_obj_id in case this is relaxed in future. Since they are ossified we know dynptr_from_mem is the only relevant helper and takes one memory argument, so we know the meta->ref_obj_id if non-zero will belong to it.

For kfuncs, make sure for constructors, only 1 ref_obj_id argument is seen, as more than one can be ambiguous in terms of ref_obj_id transfer. Since more kfuncs can be added with possibly multiple memory arguments, make sure meta->ref_obj_id reassignment won't cause incorrect lifetime analysis in the future using ref_obj_cnt logic. Set this ref_obj_id as the clone_ref_obj_id, so that it is transferred to the spilled_ptr stack slot register state.

Add support to unmark_stack_slots_dynptr to not descend to its child slices (using bool slice parameter) so that we don't have circular calls when invoking release_reference. With this logic in place, we may have the following object relationships:

                                     +-- slice 1 (ref=1)
 source (ref=1) --> dynptr (ref=1) --|-- slice 2 (ref=1)
                                     +-- slice 3 (ref=1)

Destroying dynptr prunes the dynptr and all its children slices, but does not affect the source. Releasing the source will effectively prune the entire ownership tree. Dynptr clones with same ref=1 will be parallel in the ownership tree.

                  +-- orig  dptr (ref=1) --> slice 1 (ref=1)
 source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1)
                  +-- clone dptr (ref=1) --> slice 3 (ref=1)

In such a case, only child slices of the dynptr clone being destroyed are invalidated. Likewise, if the source object goes away, the whole tree ends up getting pruned.

Cc Amery Hung <[email protected]>
Fixes 81bb1c2c3e8e ("bpf: net_sched: Add basic bpf qdisc kfuncs")
---
 kernel/bpf/verifier.c | 81 ++++++++++++++++++++++++++++---------------
 1 file changed, 54 insertions(+), 27 deletions(-)
kernel/bpf/verifier.c +2 -1
index 54c6953a8b84..a62dfab9aea6 100644
@@ -198,7 +198,7 @@ struct bpf_verifier_stack_elem {
 
 static int acquire_reference(struct bpf_verifier_env *env, int insn_idx);
 static int release_reference_nomark(struct bpf_verifier_state *state, int ref_obj_id);
-static int release_reference(struct bpf_verifier_env *env, int ref_obj_id);
+static int release_reference(struct bpf_verifier_env *env, int ref_obj_id, bool objects);
 static void invalidate_non_owning_refs(struct bpf_verifier_env *env);
 static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env);
 static int ref_set_non_owning(struct bpf_verifier_env *env,
@@ -299,6 +299,7 @@ struct bpf_kfunc_call_arg_meta {
        const char *func_name;
        /* Out parameters */
        u32 ref_obj_id;
+       u32 ref_obj_cnt;
        u8 release_regno;
        bool r0_rdonly;
        u32 ret_btf_id;

kernel/bpf/verifier.c static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_ · line 763

Quoted 5 lines
kernel/bpf/verifier.c +1 -1
@@ -759,7 +760,7 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_
        mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr,
                               &state->stack[spi - 1].spilled_ptr, type);
 
-       if (dynptr_type_refcounted(type)) {
+       if (dynptr_type_refcounted(type) || clone_ref_obj_id) {

it would be a bug to have both conditions true, right? Should we express this a bit more explicitly?
maybe something like

int id = 0;

if (dynptr_type_refcounted(type)) {
    WARN_ON_ONCE(clone_ref_obj_id);
    id = acquire_reference(env, insn_idx);
}
if (clone_ref_obj_id)
    id = clone_ref_obj_id;

state->stack[spi].spilled_ptr.ref_obj_id = id; state->stack[spi - 1].spilled_ptr.ref_obj_id = id; state->stack[spi].spilled_ptr.live |= REG_LIVE_WRITTEN; state->stack[spi - 1].spilled_ptr.live |= REG_LIVE_WRITTEN;

?

Quoted 13 lines
kernel/bpf/verifier.c +2 -3
                /* The id is used to track proper releasing */
                int id;
 
@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat
        state->stack[spi - 1].spilled_ptr.live |= REG_LIVE_WRITTEN;
 }
 
-static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
+static int __unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_func_state *state,
+                                      int spi, bool slice)
 {
-       struct bpf_func_state *state = func(env, reg);
-       int spi, ref_obj_id, i;

kernel/bpf/verifier.c static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat · line 830

Quoted 10 lines
kernel/bpf/verifier.c +4 -4
+       u32 ref_obj_id;
+       int i;
 
-       spi = dynptr_get_spi(env, reg);
-       if (spi < 0)
-               return spi;
+       ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
 
-       if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) {
+       if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {

Eduard already pointed out that !ref_obj_id condition is enough now

Quoted 10 lines
kernel/bpf/verifier.c +0 -2
                invalidate_dynptr(env, state, spi);
                return 0;
        }
 
-       ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
-
        /* If the dynptr has a ref_obj_id, then we need to invalidate
         * two things:
         *

kernel/bpf/verifier.c static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re · line 844

Quoted 6 lines
kernel/bpf/verifier.c +2 -1
@@ -842,7 +840,8 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
         */
 
        /* Invalidate any slices associated with this dynptr */
-       WARN_ON_ONCE(release_reference(env, ref_obj_id));
+       if (slice)
+               WARN_ON_ONCE(release_reference(env, ref_obj_id, false));

tbh, this bool flag to release_reference to prevent cycles feels like a hack and that we are not solving this properly...

also, changes in this patch still won't properly support the case where we create dynptr -> slice -> dynptr -> slice -> ... chains, right? Any dynptr in the chain will cause "release" of any other dynptr, including parents, right?

Quoted 3 lines
kernel/bpf/verifier.c +0 -0
        /* Invalidate any dynptr clones */
        for (i = 1; i < state->allocated_stack / BPF_REG_SIZE; i++) {

[...]

kernel/bpf/verifier.c static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob · line 10234

Quoted 6 lines
kernel/bpf/verifier.c +1 -1
@@ -10220,12 +10231,12 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob
  *
  * This is the release function corresponding to acquire_reference(). Idempotent.
  */
-static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)
+static int release_reference(struct bpf_verifier_env *env, int ref_obj_id, bool objects)

hm, "objects" really doesn't describe anything...

Quoted 10 lines
kernel/bpf/verifier.c +1 -1
 {
        struct bpf_verifier_state *vstate = env->cur_state;
        struct bpf_func_state *state;
        struct bpf_reg_state *reg;
-       int err;
+       int err, spi;
 
        err = release_reference_nomark(vstate, ref_obj_id);
        if (err)

kernel/bpf/verifier.c static int release_reference(struct bpf_verifier_env *env, int ref_obj_id) · line 10251

Quoted 5 lines
kernel/bpf/verifier.c +2 -0
@@ -10236,6 +10247,19 @@ static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)
                        mark_reg_invalid(env, reg);
        }));
 
+       if (!objects)
+               return 0;

instead of a bool flag into release_reference, shouldn't the below logic be just a separate function?

Quoted 14 lines
kernel/bpf/verifier.c +11 -0
+
+       bpf_for_each_spilled_reg(spi, state, reg, (1 << STACK_DYNPTR)) {
+               if (!reg)
+                       continue;
+               if (!reg->dynptr.first_slot || reg->ref_obj_id != ref_obj_id)
+                       continue;
+               err = __unmark_stack_slots_dynptr(env, state, spi, false);
+               if (err)
+                       return err;
+       }
+
        return 0;
 }
 

[...]

author maintainer
#7 lore raw

Re: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects

Quoted 94 lines · Andrii Nakryiko
Quoted 72 lines · Kumar Kartikeya Dwivedi

Ensure that for dynptr constructors (MEM_UNINIT arg_type) taking a referenced object (ref_obj_id) as their memory source, we set the ref_obj_id of the dynptr appropriately as well. This allows us to tie the lifetime of the dynptr to its source and properly invalidate it when the source object is released.

For helpers, we don't have such cases yet as bpf_dynptr_from_mem is not permitted for anything other than PTR_TO_MAP_VALUE, but still pass meta->ref_obj_id as clone_ref_obj_id in case this is relaxed in future. Since they are ossified we know dynptr_from_mem is the only relevant helper and takes one memory argument, so we know the meta->ref_obj_id if non-zero will belong to it.

For kfuncs, make sure for constructors, only 1 ref_obj_id argument is seen, as more than one can be ambiguous in terms of ref_obj_id transfer. Since more kfuncs can be added with possibly multiple memory arguments, make sure meta->ref_obj_id reassignment won't cause incorrect lifetime analysis in the future using ref_obj_cnt logic. Set this ref_obj_id as the clone_ref_obj_id, so that it is transferred to the spilled_ptr stack slot register state.

Add support to unmark_stack_slots_dynptr to not descend to its child slices (using bool slice parameter) so that we don't have circular calls when invoking release_reference. With this logic in place, we may have the following object relationships:

                                     +-- slice 1 (ref=1)
 source (ref=1) --> dynptr (ref=1) --|-- slice 2 (ref=1)
                                     +-- slice 3 (ref=1)

Destroying dynptr prunes the dynptr and all its children slices, but does not affect the source. Releasing the source will effectively prune the entire ownership tree. Dynptr clones with same ref=1 will be parallel in the ownership tree.

                  +-- orig  dptr (ref=1) --> slice 1 (ref=1)
 source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1)
                  +-- clone dptr (ref=1) --> slice 3 (ref=1)

In such a case, only child slices of the dynptr clone being destroyed are invalidated. Likewise, if the source object goes away, the whole tree ends up getting pruned.

Cc Amery Hung <[email protected]>
Fixes 81bb1c2c3e8e ("bpf: net_sched: Add basic bpf qdisc kfuncs")
---
 kernel/bpf/verifier.c | 81 ++++++++++++++++++++++++++++---------------
 1 file changed, 54 insertions(+), 27 deletions(-)
kernel/bpf/verifier.c +2 -1
index 54c6953a8b84..a62dfab9aea6 100644
@@ -198,7 +198,7 @@ struct bpf_verifier_stack_elem {
 
 static int acquire_reference(struct bpf_verifier_env *env, int insn_idx);
 static int release_reference_nomark(struct bpf_verifier_state *state, int ref_obj_id);
-static int release_reference(struct bpf_verifier_env *env, int ref_obj_id);
+static int release_reference(struct bpf_verifier_env *env, int ref_obj_id, bool objects);
 static void invalidate_non_owning_refs(struct bpf_verifier_env *env);
 static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env);
 static int ref_set_non_owning(struct bpf_verifier_env *env,
@@ -299,6 +299,7 @@ struct bpf_kfunc_call_arg_meta {
        const char *func_name;
        /* Out parameters */
        u32 ref_obj_id;
+       u32 ref_obj_cnt;
        u8 release_regno;
        bool r0_rdonly;
        u32 ret_btf_id;

kernel/bpf/verifier.c static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_ · line 763

Quoted 5 lines
kernel/bpf/verifier.c +1 -1
@@ -759,7 +760,7 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_
        mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr,
                               &state->stack[spi - 1].spilled_ptr, type);
 
-       if (dynptr_type_refcounted(type)) {
+       if (dynptr_type_refcounted(type) || clone_ref_obj_id) {

it would be a bug to have both conditions true, right? Should we express this a bit more explicitly?
maybe something like

int id = 0;

if (dynptr_type_refcounted(type)) {
    WARN_ON_ONCE(clone_ref_obj_id);
    id = acquire_reference(env, insn_idx);
}
if (clone_ref_obj_id)
    id = clone_ref_obj_id;
Quoted 6 lines

state->stack[spi].spilled_ptr.ref_obj_id = id; state->stack[spi - 1].spilled_ptr.ref_obj_id = id; state->stack[spi].spilled_ptr.live |= REG_LIVE_WRITTEN; state->stack[spi - 1].spilled_ptr.live |= REG_LIVE_WRITTEN;

?

Yes.

Quoted 22 lines
Quoted 22 lines
kernel/bpf/verifier.c +5 -6
                /* The id is used to track proper releasing */
                int id;
 
@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat
        state->stack[spi - 1].spilled_ptr.live |= REG_LIVE_WRITTEN;
 }
 
-static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
+static int __unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_func_state *state,
+                                      int spi, bool slice)
 {
-       struct bpf_func_state *state = func(env, reg);
-       int spi, ref_obj_id, i;
+       u32 ref_obj_id;
+       int i;
 
-       spi = dynptr_get_spi(env, reg);
-       if (spi < 0)
-               return spi;
+       ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
 
Quoted 4 lines

kernel/bpf/verifier.c static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat · line 830

Quoted 2 lines
kernel/bpf/verifier.c +1 -1
-       if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) {
+       if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {

Eduard already pointed out that !ref_obj_id condition is enough now

Ack, will change.

Quoted 21 lines
Quoted 11 lines
kernel/bpf/verifier.c +0 -2
                invalidate_dynptr(env, state, spi);
                return 0;
        }
 
-       ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
-
        /* If the dynptr has a ref_obj_id, then we need to invalidate
         * two things:
         *

kernel/bpf/verifier.c static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re · line 844

Quoted 6 lines
kernel/bpf/verifier.c +2 -1
@@ -842,7 +840,8 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
         */
 
        /* Invalidate any slices associated with this dynptr */
-       WARN_ON_ONCE(release_reference(env, ref_obj_id));
+       if (slice)
+               WARN_ON_ONCE(release_reference(env, ref_obj_id, false));

tbh, this bool flag to release_reference to prevent cycles feels like a hack and that we are not solving this properly...

Quoted 5 lines

also, changes in this patch still won't properly support the case where we create dynptr -> slice -> dynptr -> slice -> ... chains, right? Any dynptr in the chain will cause "release" of any other dynptr, including parents, right?

If they share the same ref_obj_id, yes.
Also, can we construct dynptr from referenced slices? I don't think anything except map values work right now.
In the chain example, how will we invalidate a dynptr in the later chain? Like by overwriting it? Just thinking what and how to test this.
Regardless, it is a valid point.

I can choose to introduce a ref_level so that we can track hierarchies and only prune things at the same or level below.
Any opinions on how you think this should be done?

Quoted 28 lines
Quoted 3 lines
kernel/bpf/verifier.c +0 -0
        /* Invalidate any dynptr clones */
        for (i = 1; i < state->allocated_stack / BPF_REG_SIZE; i++) {

[...]

kernel/bpf/verifier.c static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob · line 10234

Quoted 6 lines
kernel/bpf/verifier.c +1 -1
@@ -10220,12 +10231,12 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob
  *
  * This is the release function corresponding to acquire_reference(). Idempotent.
  */
-static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)
+static int release_reference(struct bpf_verifier_env *env, int ref_obj_id, bool objects)

hm, "objects" really doesn't describe anything...

Quoted 13 lines
kernel/bpf/verifier.c +1 -1
 {
        struct bpf_verifier_state *vstate = env->cur_state;
        struct bpf_func_state *state;
        struct bpf_reg_state *reg;
-       int err;
+       int err, spi;
 
        err = release_reference_nomark(vstate, ref_obj_id);
        if (err)
@@ -10236,6 +10247,19 @@ static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)
                        mark_reg_invalid(env, reg);
        }));
 
Quoted 5 lines

kernel/bpf/verifier.c static int release_reference(struct bpf_verifier_env *env, int ref_obj_id) · line 10251

Quoted 2 lines
kernel/bpf/verifier.c +2 -0
+       if (!objects)
+               return 0;

instead of a bool flag into release_reference, shouldn't the below logic be just a separate function?

Yes, that can make it clearer. It may be unnecessary if the function knows which level of the ownership hierarchy to prune correctly, rn the flag is partially there because we can call recursively into it from unmark_stack_slots_dynptr.

Quoted 17 lines
Quoted 15 lines
kernel/bpf/verifier.c +11 -0
+
+       bpf_for_each_spilled_reg(spi, state, reg, (1 << STACK_DYNPTR)) {
+               if (!reg)
+                       continue;
+               if (!reg->dynptr.first_slot || reg->ref_obj_id != ref_obj_id)
+                       continue;
+               err = __unmark_stack_slots_dynptr(env, state, spi, false);
+               if (err)
+                       return err;
+       }
+
        return 0;
 }
 

[...]

#8 lore raw

Re: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects

Quoted 55 lines · Kumar Kartikeya Dwivedi
Quoted 53 lines · Andrii Nakryiko
Quoted 50 lines · Kumar Kartikeya Dwivedi

Ensure that for dynptr constructors (MEM_UNINIT arg_type) taking a referenced object (ref_obj_id) as their memory source, we set the ref_obj_id of the dynptr appropriately as well. This allows us to tie the lifetime of the dynptr to its source and properly invalidate it when the source object is released.

For helpers, we don't have such cases yet as bpf_dynptr_from_mem is not permitted for anything other than PTR_TO_MAP_VALUE, but still pass meta->ref_obj_id as clone_ref_obj_id in case this is relaxed in future. Since they are ossified we know dynptr_from_mem is the only relevant helper and takes one memory argument, so we know the meta->ref_obj_id if non-zero will belong to it.

For kfuncs, make sure for constructors, only 1 ref_obj_id argument is seen, as more than one can be ambiguous in terms of ref_obj_id transfer. Since more kfuncs can be added with possibly multiple memory arguments, make sure meta->ref_obj_id reassignment won't cause incorrect lifetime analysis in the future using ref_obj_cnt logic. Set this ref_obj_id as the clone_ref_obj_id, so that it is transferred to the spilled_ptr stack slot register state.

Add support to unmark_stack_slots_dynptr to not descend to its child slices (using bool slice parameter) so that we don't have circular calls when invoking release_reference. With this logic in place, we may have the following object relationships:

                                     +-- slice 1 (ref=1)
 source (ref=1) --> dynptr (ref=1) --|-- slice 2 (ref=1)
                                     +-- slice 3 (ref=1)

Destroying dynptr prunes the dynptr and all its children slices, but does not affect the source. Releasing the source will effectively prune the entire ownership tree. Dynptr clones with same ref=1 will be parallel in the ownership tree.

                  +-- orig  dptr (ref=1) --> slice 1 (ref=1)
 source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1)
                  +-- clone dptr (ref=1) --> slice 3 (ref=1)

In such a case, only child slices of the dynptr clone being destroyed are invalidated. Likewise, if the source object goes away, the whole tree ends up getting pruned.

Cc Amery Hung <[email protected]>
Fixes 81bb1c2c3e8e ("bpf: net_sched: Add basic bpf qdisc kfuncs")
---
 kernel/bpf/verifier.c | 81 ++++++++++++++++++++++++++++---------------
 1 file changed, 54 insertions(+), 27 deletions(-)

[...]

Quoted 27 lines
Quoted 21 lines
Quoted 11 lines
                invalidate_dynptr(env, state, spi);
                return 0;
        }
kernel/bpf/verifier.c +0 -2
-       ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
-
        /* If the dynptr has a ref_obj_id, then we need to invalidate
         * two things:
         *

kernel/bpf/verifier.c static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re · line 844

Quoted 6 lines
kernel/bpf/verifier.c +2 -1
@@ -842,7 +840,8 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
         */
 
        /* Invalidate any slices associated with this dynptr */
-       WARN_ON_ONCE(release_reference(env, ref_obj_id));
+       if (slice)
+               WARN_ON_ONCE(release_reference(env, ref_obj_id, false));

tbh, this bool flag to release_reference to prevent cycles feels like a hack and that we are not solving this properly...

Quoted 5 lines

also, changes in this patch still won't properly support the case where we create dynptr -> slice -> dynptr -> slice -> ... chains, right? Any dynptr in the chain will cause "release" of any other dynptr, including parents, right?

Quoted 3 lines

If they share the same ref_obj_id, yes.
Also, can we construct dynptr from referenced slices? I don't think anything except map values work right now.

Amery wanted to allow creating bpf_dynptr_from_mem() from any PTR_TO_MEM, which would allow creating ringbuf_dynptr -> slice [ -> mem_dynptr -> slice ]* chains.

Quoted 3 lines

In the chain example, how will we invalidate a dynptr in the later chain? Like by overwriting it? Just thinking what and how to test this.

Yes, user can always just write over dynptr slots on the stack, which doesn't even have to be intentional. mem dynptr is never explicitly released, so technically you can just stop using it and use that stack space for some other variables.

Quoted 5 lines

Regardless, it is a valid point.

I can choose to introduce a ref_level so that we can track hierarchies and only prune things at the same or level below.
Any opinions on how you think this should be done?

I was thinking that we can form an (implicit) tree of ref objects through their id and ref_obj_id properties. Each new dynptr gets its own id and sets its ref_obj_id to whatever id of the resource we are creating dynptr from (so if it's a slice, then I guess we'd need to use slices's ref_obj_id, right?). And so you'd have:

ringbuf_dynptr (id=1) -> slice (id=2,ref_obj_id=1) -> mem_dynptr(id=3,ref_obj_id=1) -> slice(id=4,**ref_obj_id=3**) -> mem_dynptr(id=5,ref_obj_id=3)

And so if you overwrite mem_dynptr(id=3), then you invalidate id=3,4,5; if you overwrite just mem_dynptr(id=5), only that one is invalidated (and id=1,2,3,4 stay valid). If you happen to release the original ringbuf_dynptr(id=1), the entire "tree" is invalidated, of course.

Quoted 57 lines
Quoted 29 lines
Quoted 4 lines
kernel/bpf/verifier.c +0 -0
        /* Invalidate any dynptr clones */
        for (i = 1; i < state->allocated_stack / BPF_REG_SIZE; i++) {

[...]

kernel/bpf/verifier.c static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob · line 10234

Quoted 6 lines
kernel/bpf/verifier.c +1 -1
@@ -10220,12 +10231,12 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob
  *
  * This is the release function corresponding to acquire_reference(). Idempotent.
  */
-static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)
+static int release_reference(struct bpf_verifier_env *env, int ref_obj_id, bool objects)

hm, "objects" really doesn't describe anything...

Quoted 13 lines
kernel/bpf/verifier.c +1 -1
 {
        struct bpf_verifier_state *vstate = env->cur_state;
        struct bpf_func_state *state;
        struct bpf_reg_state *reg;
-       int err;
+       int err, spi;
 
        err = release_reference_nomark(vstate, ref_obj_id);
        if (err)
@@ -10236,6 +10247,19 @@ static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)
                        mark_reg_invalid(env, reg);
        }));
 
Quoted 5 lines

kernel/bpf/verifier.c static int release_reference(struct bpf_verifier_env *env, int ref_obj_id) · line 10251

Quoted 2 lines
kernel/bpf/verifier.c +2 -0
+       if (!objects)
+               return 0;

instead of a bool flag into release_reference, shouldn't the below logic be just a separate function?

Yes, that can make it clearer. It may be unnecessary if the function knows which level of the ownership hierarchy to prune correctly, rn the flag is partially there because we can call recursively into it from unmark_stack_slots_dynptr.

Quoted 17 lines
Quoted 15 lines
kernel/bpf/verifier.c +11 -0
+
+       bpf_for_each_spilled_reg(spi, state, reg, (1 << STACK_DYNPTR)) {
+               if (!reg)
+                       continue;
+               if (!reg->dynptr.first_slot || reg->ref_obj_id != ref_obj_id)
+                       continue;
+               err = __unmark_stack_slots_dynptr(env, state, spi, false);
+               if (err)
+                       return err;
+       }
+
        return 0;
 }
 

[...]

author maintainer
#9 lore raw

RFC PATCH 2/13 bpf: Compare dynptr_id in regsafe

In reply to Kumar Kartikeya Dwivedi

Now that PTR_TO_MEM can be invalidated due to skb going away, we must take care to be more careful in regsafe regarding state pruning. While ref_obj_id comparison will ensure that incorrect pruning is prevented, since we attach ref_obj_id of skb to the PTR_TO_MEM emanating from it, it is nonetheless clearer to also compare the dynptr_id as well.

---
 kernel/bpf/verifier.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/bpf/verifier.c +2 -0
index a62dfab9aea6..7e09c4592038 100644
@@ -18426,6 +18426,8 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
 		       range_within(rold, rcur) &&
 		       tnum_in(rold->var_off, rcur->var_off) &&
 		       check_ids(rold->id, rcur->id, idmap) &&
+		       (base_type(rold->type) == PTR_TO_MEM ?
+			check_ids(rold->dynptr_id, rcur->dynptr_id, idmap) : 1) &&
 		       check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap);
 	case PTR_TO_PACKET_META:
 	case PTR_TO_PACKET:
Signature · 1 line
2.47.1
maintainer Acked-by on v1 2/13
#10 lore raw

Re: [RFC PATCH bpf-next/net v1 02/13] bpf: Compare dynptr_id in regsafe

Quoted 8 lines · Kumar Kartikeya Dwivedi

Now that PTR_TO_MEM can be invalidated due to skb going away, we must take care to be more careful in regsafe regarding state pruning. While ref_obj_id comparison will ensure that incorrect pruning is prevented, since we attach ref_obj_id of skb to the PTR_TO_MEM emanating from it, it is nonetheless clearer to also compare the dynptr_id as well.

---

[...]

#11 lore raw

Re: [RFC PATCH bpf-next/net v1 02/13] bpf: Compare dynptr_id in regsafe

In reply to Kumar Kartikeya Dwivedi
Quoted 17 lines · Kumar Kartikeya Dwivedi

Now that PTR_TO_MEM can be invalidated due to skb going away, we must take care to be more careful in regsafe regarding state pruning. While ref_obj_id comparison will ensure that incorrect pruning is prevented, since we attach ref_obj_id of skb to the PTR_TO_MEM emanating from it, it is nonetheless clearer to also compare the dynptr_id as well.

---
 kernel/bpf/verifier.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/bpf/verifier.c +0 -0
index a62dfab9aea6..7e09c4592038 100644

kernel/bpf/verifier.c static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold, · line 18431

Quoted 6 lines
kernel/bpf/verifier.c +2 -0
@@ -18426,6 +18426,8 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
                       range_within(rold, rcur) &&
                       tnum_in(rold->var_off, rcur->var_off) &&
                       check_ids(rold->id, rcur->id, idmap) &&
+                      (base_type(rold->type) == PTR_TO_MEM ?
+                       check_ids(rold->dynptr_id, rcur->dynptr_id, idmap) : 1) &&
                       check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap);

hm... shall we split out PTR_TO_MEM case instead of making this not-so-simple condition even more not-so-simple? or (if people don't like that idea), I'd rather have this special PTR_TO_MEM handling as a separate if with return

Quoted 5 lines
kernel/bpf/verifier.c +0 -1
        case PTR_TO_PACKET_META:
        case PTR_TO_PACKET:
--

2.47.1

author maintainer
#12 lore raw

Re: [RFC PATCH bpf-next/net v1 02/13] bpf: Compare dynptr_id in regsafe

Quoted 15 lines · Andrii Nakryiko
Quoted 12 lines · Kumar Kartikeya Dwivedi

Now that PTR_TO_MEM can be invalidated due to skb going away, we must take care to be more careful in regsafe regarding state pruning. While ref_obj_id comparison will ensure that incorrect pruning is prevented, since we attach ref_obj_id of skb to the PTR_TO_MEM emanating from it, it is nonetheless clearer to also compare the dynptr_id as well.

---
 kernel/bpf/verifier.c | 2 ++
 1 file changed, 2 insertions(+)
Quoted 16 lines

kernel/bpf/verifier.c static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold, · line 18431

Quoted 11 lines
kernel/bpf/verifier.c +2 -0
index a62dfab9aea6..7e09c4592038 100644
@@ -18426,6 +18426,8 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
                       range_within(rold, rcur) &&
                       tnum_in(rold->var_off, rcur->var_off) &&
                       check_ids(rold->id, rcur->id, idmap) &&
+                      (base_type(rold->type) == PTR_TO_MEM ?
+                       check_ids(rold->dynptr_id, rcur->dynptr_id, idmap) : 1) &&
                       check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap);

hm... shall we split out PTR_TO_MEM case instead of making this not-so-simple condition even more not-so-simple? or (if people don't like that idea), I'd rather have this special PTR_TO_MEM handling as a separate if with return

I can split into a separate case.

Quoted 7 lines
Quoted 7 lines
kernel/bpf/verifier.c +0 -1
        case PTR_TO_PACKET_META:
        case PTR_TO_PACKET:
--

2.47.1

author maintainer
#13 lore raw

RFC PATCH 3/13 selftests/bpf: Convert dynptr_fail to use vmlinux.h

In reply to Kumar Kartikeya Dwivedi

Let's make use of vmlinux.h in dynptr_fail.c. No functional change intended, just a drive-by improvement.

---
 tools/testing/selftests/bpf/progs/dynptr_fail.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
tools/testing/selftests/bpf/progs/dynptr_fail.c +6 -8
index bd8f15229f5c..345e704e5346 100644
@@ -1,13 +1,11 @@
 // SPDX-License-Identifier: GPL-2.0
 /* Copyright (c) 2022 Facebook */
-
+#include <vmlinux.h>
 #include <errno.h>
 #include <string.h>
 #include <stdbool.h>
-#include <linux/bpf.h>
 #include <bpf/bpf_helpers.h>
 #include <bpf/bpf_tracing.h>
-#include <linux/if_ether.h>
 #include "bpf_misc.h"
 #include "bpf_kfuncs.h"
 
@@ -46,7 +44,7 @@ struct {
 	__type(value, __u64);
 } array_map4 SEC(".maps");
 
-struct sample {
+struct dynptr_sample {
 	int pid;
 	long value;
 	char comm[16];
@@ -95,7 +93,7 @@ __failure __msg("Unreleased reference id=4")
 int ringbuf_missing_release2(void *ctx)
 {
 	struct bpf_dynptr ptr1, ptr2;
-	struct sample *sample;
+	struct dynptr_sample *sample;
 
 	bpf_ringbuf_reserve_dynptr(&ringbuf, sizeof(*sample), 0, &ptr1);
 	bpf_ringbuf_reserve_dynptr(&ringbuf, sizeof(*sample), 0, &ptr2);
@@ -173,7 +171,7 @@ __failure __msg("type=mem expected=ringbuf_mem")
 int ringbuf_invalid_api(void *ctx)
 {
 	struct bpf_dynptr ptr;
-	struct sample *sample;
+	struct dynptr_sample *sample;
 
 	bpf_ringbuf_reserve_dynptr(&ringbuf, sizeof(*sample), 0, &ptr);
 	sample = bpf_dynptr_data(&ptr, 0, sizeof(*sample));
@@ -295,7 +293,7 @@ __failure __msg("invalid mem access 'scalar'")
 int data_slice_use_after_release1(void *ctx)
 {
 	struct bpf_dynptr ptr;
-	struct sample *sample;
+	struct dynptr_sample *sample;
 
 	bpf_ringbuf_reserve_dynptr(&ringbuf, sizeof(*sample), 0, &ptr);
 	sample = bpf_dynptr_data(&ptr, 0, sizeof(*sample));
@@ -327,7 +325,7 @@ __failure __msg("invalid mem access 'scalar'")
 int data_slice_use_after_release2(void *ctx)
 {
 	struct bpf_dynptr ptr1, ptr2;
-	struct sample *sample;
+	struct dynptr_sample *sample;
 
 	bpf_ringbuf_reserve_dynptr(&ringbuf, 64, 0, &ptr1);
 	bpf_ringbuf_reserve_dynptr(&ringbuf, sizeof(*sample), 0, &ptr2);
Signature · 1 line
2.47.1
author maintainer
#15 lore raw

RFC PATCH 4/13 selftests/bpf: Add tests for dynptr source object interaction

In reply to Kumar Kartikeya Dwivedi

Add a few tests to capture source object relationship with dynptr and their slices and ensure invalidation of everything works correctly.

---
 .../testing/selftests/bpf/prog_tests/dynptr.c |  2 +
 .../testing/selftests/bpf/progs/dynptr_fail.c |  1 +
 .../selftests/bpf/progs/dynptr_fail_qdisc.c   | 38 +++++++++++++++++++
 3 files changed, 41 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/dynptr_fail_qdisc.c
tools/testing/selftests/bpf/prog_tests/dynptr.c +2 -0
index e29cc16124c2..929355de1002 100644
@@ -4,6 +4,7 @@
 #include <test_progs.h>
 #include <network_helpers.h>
 #include "dynptr_fail.skel.h"
+#include "dynptr_fail_qdisc.skel.h"
 #include "dynptr_success.skel.h"
 
 enum test_setup_type {
@@ -161,4 +162,5 @@ void test_dynptr(void)
 	}
 
 	RUN_TESTS(dynptr_fail);
+	RUN_TESTS(dynptr_fail_qdisc);
 }
tools/testing/selftests/bpf/progs/dynptr_fail.c +1 -0
index 345e704e5346..7c67797a5aac 100644
@@ -8,6 +8,7 @@
 #include <bpf/bpf_tracing.h>
 #include "bpf_misc.h"
 #include "bpf_kfuncs.h"
+#include "bpf_experimental.h"
 
 char _license[] SEC("license") = "GPL";
 
tools/testing/selftests/bpf/progs/dynptr_fail_qdisc.c new file +38 -0
new file mode 100644
index 000000000000..fc222213f572
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "bpf_misc.h"
+#include "bpf_experimental.h"
+
+char _license[] SEC("license") = "GPL";
+
+/*
+ * Putting these tests in dynptr_fail.c causes other random tests to fail,
+ * keep them in their isolated CU.
+ */
+
+SEC("?struct_ops")
+__failure __msg("invalid mem access 'scalar'")
+int BPF_PROG(test_dynptr_source_release, struct sk_buff *skb,
+	     struct Qdisc *sch, struct bpf_sk_buff_ptr *to_free)
+{
+	struct bpf_dynptr dptr, dptr2;
+	char buf[8], *data;
+
+	bpf_dynptr_from_skb((struct __sk_buff *)skb, 0, &dptr);
+	bpf_dynptr_read(buf, sizeof(buf), &dptr, 0, 0);
+	bpf_dynptr_clone(&dptr, &dptr2);
+	data = bpf_dynptr_slice(&dptr2, 0, buf, sizeof(buf));
+	bpf_qdisc_skb_drop(skb, to_free);
+	/* These reads/writes now succeed since dynptr is destroyed. */
+	*(char *)&dptr = *(char *)&dptr2;
+	return *data;
+}
+
+SEC("?.struct_ops")
+struct Qdisc_ops test_dynptr_qdisc = {
+	.enqueue   = (void *)test_dynptr_source_release,
+	.id        = "bpf_fq",
+};
Signature · 1 line
2.47.1
maintainer
#16 lore raw

Re: [RFC PATCH bpf-next/net v1 04/13] selftests/bpf: Add tests for dynptr source object interaction

Quoted 5 lines · Kumar Kartikeya Dwivedi

Add a few tests to capture source object relationship with dynptr and their slices and ensure invalidation of everything works correctly.

---

It would be great to have a bit more test cases here, following your nice diagram from patch #1:

		  +-- orig  dptr (ref=1) --> slice 1 (ref=1)
 source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1)
		  +-- clone dptr (ref=1) --> slice 3 (ref=1)

In one test (probably extending the one in this patch):
- check that both orig and clone slices can be read at some point
- destroy orig or clone and check that one of the slices can be read, while another can't.

In another test:
- destroy source and check that both orig and clone slices are no longer accessible.

Also, is it possible to conjure a test case with two hierarchies? E.g. source1(ref=1) and source2(ref=2)?

[...]

author maintainer
#17 lore raw

Re: [RFC PATCH bpf-next/net v1 04/13] selftests/bpf: Add tests for dynptr source object interaction

Quoted 16 lines · Eduard Zingerman
Quoted 5 lines · Kumar Kartikeya Dwivedi

Add a few tests to capture source object relationship with dynptr and their slices and ensure invalidation of everything works correctly.

---

It would be great to have a bit more test cases here, following your nice diagram from patch #1:

                  +-- orig  dptr (ref=1) --> slice 1 (ref=1)
 source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1)
                  +-- clone dptr (ref=1) --> slice 3 (ref=1)
Quoted 8 lines

In one test (probably extending the one in this patch):
- check that both orig and clone slices can be read at some point
- destroy orig or clone and check that one of the slices can be read, while another can't.

In another test:
- destroy source and check that both orig and clone slices are no longer accessible.

Agreed, will try and add more cases explicitly.

I struggled to have more than two callbacks automatically working with RUN_TESTS and struct_ops, so I guess I will do explicit loading and failure checks.

Quoted 4 lines

Also, is it possible to conjure a test case with two hierarchies? E.g. source1(ref=1) and source2(ref=2)?

Should be possible, I'll add it.

Quoted 1 line

[...]

maintainer
#18 lore raw

Re: [RFC PATCH bpf-next/net v1 04/13] selftests/bpf: Add tests for dynptr source object interaction

Kumar Kartikeya Dwivedi <[email protected]> writes:

[...]

Quoted 3 lines

I struggled to have more than two callbacks automatically working with RUN_TESTS and struct_ops, so I guess I will do explicit loading and failure checks.

Could you please clarify the use-case a bit?
In theory two RUN_TESTS should be independent.

[...]

author maintainer
#19 lore raw

Re: [RFC PATCH bpf-next/net v1 04/13] selftests/bpf: Add tests for dynptr source object interaction

Quoted 11 lines · Eduard Zingerman

Kumar Kartikeya Dwivedi <[email protected]> writes:

[...]

Quoted 3 lines

I struggled to have more than two callbacks automatically working with RUN_TESTS and struct_ops, so I guess I will do explicit loading and failure checks.

Could you please clarify the use-case a bit?
In theory two RUN_TESTS should be independent.

I meant I see strange behavior when I have both .enqueue and .dequeue and do RUN_TESTS, for the second callback, I see unexpected load success. When keeping one of the callbacks, everything is ok.
I didn't dig further the other day before sending it out.

Quoted 2 lines

[...]

author maintainer
#20 lore raw

RFC PATCH 5/13 locking/local_lock, mm: Replace localtry_ helpers with local_trylock_t type

In reply to Kumar Kartikeya Dwivedi

From: Alexei Starovoitov <[email protected]>

Partially revert commit 0aaddfb06882 ("locking/local_lock: Introduce localtry_lock_t"). Remove localtry_*() helpers, since localtry_lock() name might
be misinterpreted as "try lock".

Introduce local_trylock[_irqsave]() helpers that only work with newly introduced local_trylock_t type.
Note that attempt to use local_trylock[_irqsave]() with local_lock_t will cause compilation failure.

Usage and behavior in !PREEMPT_RT:

local_lock_t lock;                     // sizeof(lock) == 0
local_lock(&lock);                     // preempt disable
local_lock_irqsave(&lock, ...);        // irq save

if (local_trylock_irqsave(&lock, ...)) // compilation error

local_trylock_t lock;                  // sizeof(lock) == 4
local_lock(&lock);                     // preempt disable, acquired = 1
local_lock_irqsave(&lock, ...);        // irq save, acquired = 1
if (local_trylock(&lock))              // if (!acquired) preempt disable, acquired = 1

if (local_trylock_irqsave(&lock, ...)) // if (!acquired) irq save, acquired = 1

The existing local_lock_*() macros can be used either with local_lock_t or local_trylock_t.
With local_trylock_t they set acquired = 1 while local_unlock_*() clears it.

In !PREEMPT_RT local_lock_irqsave(local_lock_t *) disables interrupts to protect critical section, but it doesn't prevent NMI, so the fully reentrant code cannot use local_lock_irqsave(local_lock_t *) for exclusive access.

The local_lock_irqsave(local_trylock_t *) helper disables interrupts and sets acquired=1, so local_trylock_irqsave(local_trylock_t *) from NMI attempting to acquire the same lock will return false.

In PREEMPT_RT local_lock_irqsave() maps to preemptible spin_lock(). Map local_trylock_irqsave() to preemptible spin_trylock(). When in hard IRQ or NMI return false right away, since spin_trylock() is not safe due to explicit locking in the underneath rt_spin_trylock() implementation. Removing this explicit locking and attempting only "trylock" is undesired due to PI implications.

The local_trylock() without _irqsave can be used to avoid the cost of disabling/enabling interrupts by only disabling preemption, so local_trylock() in an interrupt attempting to acquire the same lock will return false.

Note there is no need to use local_inc for acquired variable, since it's a percpu variable with strict nesting scopes.

Note that guard(local_lock)(&lock) works only for "local_lock_t lock".

The patch also makes sure that local_lock_release(l) is called before WRITE_ONCE(l->acquired, 0). Though IRQs are disabled at this point the local_trylock() from NMI will succeed and local_lock_acquire(l) will warn.

Fixes 0aaddfb06882 ("locking/local_lock: Introduce localtry_lock_t")
---
 include/linux/local_lock.h          |  58 ++------
 include/linux/local_lock_internal.h | 207 ++++++++++++----------------
 mm/memcontrol.c                     |  39 +++---
 3 files changed, 114 insertions(+), 190 deletions(-)
include/linux/local_lock.h +8 -50
index 1a0bc35839e3..16a2ee4f8310 100644
@@ -52,44 +52,23 @@
 	__local_unlock_irqrestore(lock, flags)
 
 /**
- * localtry_lock_init - Runtime initialize a lock instance
- */
-#define localtry_lock_init(lock)		__localtry_lock_init(lock)
-
-/**
- * localtry_lock - Acquire a per CPU local lock
- * @lock:	The lock variable
- */
-#define localtry_lock(lock)		__localtry_lock(lock)
-
-/**
- * localtry_lock_irq - Acquire a per CPU local lock and disable interrupts
- * @lock:	The lock variable
- */
-#define localtry_lock_irq(lock)		__localtry_lock_irq(lock)
-
-/**
- * localtry_lock_irqsave - Acquire a per CPU local lock, save and disable
- *			 interrupts
- * @lock:	The lock variable
- * @flags:	Storage for interrupt flags
+ * local_lock_init - Runtime initialize a lock instance
  */
-#define localtry_lock_irqsave(lock, flags)				\
-	__localtry_lock_irqsave(lock, flags)
+#define local_trylock_init(lock)	__local_trylock_init(lock)
 
 /**
- * localtry_trylock - Try to acquire a per CPU local lock.
+ * local_trylock - Try to acquire a per CPU local lock
  * @lock:	The lock variable
  *
  * The function can be used in any context such as NMI or HARDIRQ. Due to
  * locking constrains it will _always_ fail to acquire the lock in NMI or
  * HARDIRQ context on PREEMPT_RT.
  */
-#define localtry_trylock(lock)		__localtry_trylock(lock)
+#define local_trylock(lock)		__local_trylock(lock)
 
 /**
- * localtry_trylock_irqsave - Try to acquire a per CPU local lock, save and disable
- *			      interrupts if acquired
+ * local_trylock_irqsave - Try to acquire a per CPU local lock, save and disable
+ *			   interrupts if acquired
  * @lock:	The lock variable
  * @flags:	Storage for interrupt flags
  *
@@ -97,29 +76,8 @@
  * locking constrains it will _always_ fail to acquire the lock in NMI or
  * HARDIRQ context on PREEMPT_RT.
  */
-#define localtry_trylock_irqsave(lock, flags)				\
-	__localtry_trylock_irqsave(lock, flags)
-
-/**
- * local_unlock - Release a per CPU local lock
- * @lock:	The lock variable
- */
-#define localtry_unlock(lock)		__localtry_unlock(lock)
-
-/**
- * local_unlock_irq - Release a per CPU local lock and enable interrupts
- * @lock:	The lock variable
- */
-#define localtry_unlock_irq(lock)		__localtry_unlock_irq(lock)
-
-/**
- * localtry_unlock_irqrestore - Release a per CPU local lock and restore
- *			      interrupt flags
- * @lock:	The lock variable
- * @flags:      Interrupt flags to restore
- */
-#define localtry_unlock_irqrestore(lock, flags)			\
-	__localtry_unlock_irqrestore(lock, flags)
+#define local_trylock_irqsave(lock, flags)			\
+	__local_trylock_irqsave(lock, flags)
 
 DEFINE_GUARD(local_lock, local_lock_t __percpu*,
 	     local_lock(_T),
include/linux/local_lock_internal.h +87 -120

315 lines not on this page · raw message

mm/memcontrol.c +19 -20
index 421740f1bcdc..c96c1f2b9cf5 100644
@@ -1759,7 +1759,7 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg)
 }
 
 struct memcg_stock_pcp {
-	localtry_lock_t stock_lock;
+	local_trylock_t stock_lock;
 	struct mem_cgroup *cached; /* this never be root cgroup */
 	unsigned int nr_pages;
 
@@ -1774,7 +1774,7 @@ struct memcg_stock_pcp {
 #define FLUSHING_CACHED_CHARGE	0
 };
 static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock) = {
-	.stock_lock = INIT_LOCALTRY_LOCK(stock_lock),
+	.stock_lock = INIT_LOCAL_TRYLOCK(stock_lock),
 };
 static DEFINE_MUTEX(percpu_charge_mutex);
 
@@ -1805,11 +1805,10 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages,
 	if (nr_pages > MEMCG_CHARGE_BATCH)
 		return ret;
 
-	if (!localtry_trylock_irqsave(&memcg_stock.stock_lock, flags)) {
-		if (!gfpflags_allow_spinning(gfp_mask))
-			return ret;
-		localtry_lock_irqsave(&memcg_stock.stock_lock, flags);
-	}
+	if (gfpflags_allow_spinning(gfp_mask))
+		local_lock_irqsave(&memcg_stock.stock_lock, flags);
+	else if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags))
+		return ret;
 
 	stock = this_cpu_ptr(&memcg_stock);
 	stock_pages = READ_ONCE(stock->nr_pages);
@@ -1818,7 +1817,7 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages,
 		ret = true;
 	}
 
-	localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags);
+	local_unlock_irqrestore(&memcg_stock.stock_lock, flags);
 
 	return ret;
 }
@@ -1857,14 +1856,14 @@ static void drain_local_stock(struct work_struct *dummy)
 	 * drain_stock races is that we always operate on local CPU stock
 	 * here with IRQ disabled
 	 */
-	localtry_lock_irqsave(&memcg_stock.stock_lock, flags);
+	local_lock_irqsave(&memcg_stock.stock_lock, flags);
 
 	stock = this_cpu_ptr(&memcg_stock);
 	old = drain_obj_stock(stock);
 	drain_stock(stock);
 	clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags);
 
-	localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags);
+	local_unlock_irqrestore(&memcg_stock.stock_lock, flags);
 	obj_cgroup_put(old);
 }
 
@@ -1894,7 +1893,7 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
 {
 	unsigned long flags;
 
-	if (!localtry_trylock_irqsave(&memcg_stock.stock_lock, flags)) {
+	if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) {
 		/*
 		 * In case of unlikely failure to lock percpu stock_lock
 		 * uncharge memcg directly.
@@ -1907,7 +1906,7 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
 		return;
 	}
 	__refill_stock(memcg, nr_pages);
-	localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags);
+	local_unlock_irqrestore(&memcg_stock.stock_lock, flags);
 }
 
 /*
@@ -1964,9 +1963,9 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu)
 	stock = &per_cpu(memcg_stock, cpu);
 
 	/* drain_obj_stock requires stock_lock */
-	localtry_lock_irqsave(&memcg_stock.stock_lock, flags);
+	local_lock_irqsave(&memcg_stock.stock_lock, flags);
 	old = drain_obj_stock(stock);
-	localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags);
+	local_unlock_irqrestore(&memcg_stock.stock_lock, flags);
 
 	drain_stock(stock);
 	obj_cgroup_put(old);
@@ -2787,7 +2786,7 @@ static void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat,
 	unsigned long flags;
 	int *bytes;
 
-	localtry_lock_irqsave(&memcg_stock.stock_lock, flags);
+	local_lock_irqsave(&memcg_stock.stock_lock, flags);
 	stock = this_cpu_ptr(&memcg_stock);
 
 	/*
@@ -2836,7 +2835,7 @@ static void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat,
 	if (nr)
 		__mod_objcg_mlstate(objcg, pgdat, idx, nr);
 
-	localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags);
+	local_unlock_irqrestore(&memcg_stock.stock_lock, flags);
 	obj_cgroup_put(old);
 }
 
@@ -2846,7 +2845,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes)
 	unsigned long flags;
 	bool ret = false;
 
-	localtry_lock_irqsave(&memcg_stock.stock_lock, flags);
+	local_lock_irqsave(&memcg_stock.stock_lock, flags);
 
 	stock = this_cpu_ptr(&memcg_stock);
 	if (objcg == READ_ONCE(stock->cached_objcg) && stock->nr_bytes >= nr_bytes) {
@@ -2854,7 +2853,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes)
 		ret = true;
 	}
 
-	localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags);
+	local_unlock_irqrestore(&memcg_stock.stock_lock, flags);
 
 	return ret;
 }
@@ -2946,7 +2945,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes,
 	unsigned long flags;
 	unsigned int nr_pages = 0;
 
-	localtry_lock_irqsave(&memcg_stock.stock_lock, flags);
+	local_lock_irqsave(&memcg_stock.stock_lock, flags);
 
 	stock = this_cpu_ptr(&memcg_stock);
 	if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
@@ -2960,7 +2959,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes,
 		stock->nr_bytes &= (PAGE_SIZE - 1);
 	}
 
-	localtry_unlock_irqrestore(&memcg_stock.stock_lock, flags);
+	local_unlock_irqrestore(&memcg_stock.stock_lock, flags);
 	obj_cgroup_put(old);
 
 	if (nr_pages)
Signature · 1 line
2.47.1
author maintainer
#21 lore raw

RFC PATCH 6/13 bpf: Introduce bpf_dynptr_from_mem_slice

In reply to Kumar Kartikeya Dwivedi

Add a new bpf_dynptr_from_mem_slice kfunc to create a dynptr from a PTR_TO_BTF_ID exposing a variable-length slice of memory, represented by the new bpf_mem_slice type. This slice is read-only, for a read-write slice we can expose a distinct type in the future.

We rely on the previous commits ensuring source objects underpinning dynptr memory are tracked correctly for invalidation to ensure when a PTR_TO_BTF_ID holding a memory slice goes away, it's corresponding dynptrs get invalidated.

---
 include/linux/bpf.h   |  5 +++++
 kernel/bpf/helpers.c  | 32 ++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c |  6 +++++-
 3 files changed, 42 insertions(+), 1 deletion(-)
include/linux/bpf.h +5 -0
index 3f0cc89c0622..9feaa9bbf0a4 100644
@@ -1344,6 +1344,11 @@ enum bpf_dynptr_type {
 	BPF_DYNPTR_TYPE_XDP,
 };
 
+struct bpf_mem_slice {
+	void *ptr;
+	size_t len;
+};
+
 int bpf_dynptr_check_size(u32 size);
 u32 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr);
 const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u32 len);
kernel/bpf/helpers.c +32 -0
index e3a2662f4e33..95e9c9df6062 100644
@@ -2826,6 +2826,37 @@ __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u32 dst_off,
 	return 0;
 }
 
+/**
+ * XXX
+ */
+__bpf_kfunc int bpf_dynptr_from_mem_slice(struct bpf_mem_slice *mem_slice, u64 flags, struct bpf_dynptr *dptr__uninit)
+{
+	struct bpf_dynptr_kern *dptr = (struct bpf_dynptr_kern *)dptr__uninit;
+	int err;
+
+	if (!mem_slice)
+		return -EINVAL;
+
+	err = bpf_dynptr_check_size(mem_slice->len);
+	if (err)
+		goto error;
+
+	/* flags is currently unsupported */
+	if (flags) {
+		err = -EINVAL;
+		goto error;
+	}
+
+	bpf_dynptr_init(dptr, mem_slice->ptr, BPF_DYNPTR_TYPE_LOCAL, 0, mem_slice->len);
+	bpf_dynptr_set_rdonly(dptr);
+
+	return 0;
+
+error:
+	bpf_dynptr_set_null(dptr);
+	return err;
+}
+
 __bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj)
 {
 	return obj;
@@ -3275,6 +3306,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly)
 BTF_ID_FLAGS(func, bpf_dynptr_size)
 BTF_ID_FLAGS(func, bpf_dynptr_clone)
 BTF_ID_FLAGS(func, bpf_dynptr_copy)
+BTF_ID_FLAGS(func, bpf_dynptr_from_mem_slice, KF_TRUSTED_ARGS)
 #ifdef CONFIG_NET
 BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
 #endif
kernel/bpf/verifier.c +5 -1
index 7e09c4592038..26aa70cd5734 100644
@@ -12125,6 +12125,7 @@ enum special_kfunc_type {
 	KF_bpf_res_spin_unlock,
 	KF_bpf_res_spin_lock_irqsave,
 	KF_bpf_res_spin_unlock_irqrestore,
+	KF_bpf_dynptr_from_mem_slice,
 };
 
 BTF_SET_START(special_kfunc_set)
@@ -12218,6 +12219,7 @@ BTF_ID(func, bpf_res_spin_lock)
 BTF_ID(func, bpf_res_spin_unlock)
 BTF_ID(func, bpf_res_spin_lock_irqsave)
 BTF_ID(func, bpf_res_spin_unlock_irqrestore)
+BTF_ID(func, bpf_dynptr_from_mem_slice)
 
 static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta)
 {
@@ -13139,7 +13141,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				}
 			}
 
-			if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) {
+			if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_mem_slice]) {
+				dynptr_arg_type |= DYNPTR_TYPE_LOCAL;
+			} else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) {
 				dynptr_arg_type |= DYNPTR_TYPE_SKB;
 			} else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_xdp]) {
 				dynptr_arg_type |= DYNPTR_TYPE_XDP;
Signature · 1 line
2.47.1
maintainer Acked-by on v1 6/13
#22 lore raw

Re: [RFC PATCH bpf-next/net v1 06/13] bpf: Introduce bpf_dynptr_from_mem_slice

Quoted 11 lines · Kumar Kartikeya Dwivedi

Add a new bpf_dynptr_from_mem_slice kfunc to create a dynptr from a PTR_TO_BTF_ID exposing a variable-length slice of memory, represented by the new bpf_mem_slice type. This slice is read-only, for a read-write slice we can expose a distinct type in the future.

We rely on the previous commits ensuring source objects underpinning dynptr memory are tracked correctly for invalidation to ensure when a PTR_TO_BTF_ID holding a memory slice goes away, it's corresponding dynptrs get invalidated.

If I remove KF_TRUSTED_ARGS flag from the function nothing fails. But that's because with current interface slices will always have a ref_obj_id, right? And pointers with ref_obj_id are always trusted.

[...]

author maintainer
#23 lore raw

Re: [RFC PATCH bpf-next/net v1 06/13] bpf: Introduce bpf_dynptr_from_mem_slice

Quoted 13 lines · Eduard Zingerman
Quoted 10 lines · Kumar Kartikeya Dwivedi

Add a new bpf_dynptr_from_mem_slice kfunc to create a dynptr from a PTR_TO_BTF_ID exposing a variable-length slice of memory, represented by the new bpf_mem_slice type. This slice is read-only, for a read-write slice we can expose a distinct type in the future.

We rely on the previous commits ensuring source objects underpinning dynptr memory are tracked correctly for invalidation to ensure when a PTR_TO_BTF_ID holding a memory slice goes away, it's corresponding dynptrs get invalidated.

Quoted 5 lines
Quoted 1 line

If I remove KF_TRUSTED_ARGS flag from the function nothing fails. But that's because with current interface slices will always have a ref_obj_id, right? And pointers with ref_obj_id are always trusted.

Yes.

Quoted 4 lines

[...]

#24 lore raw

Re: [RFC PATCH bpf-next/net v1 06/13] bpf: Introduce bpf_dynptr_from_mem_slice

In reply to Kumar Kartikeya Dwivedi
Quoted 23 lines · Kumar Kartikeya Dwivedi

Add a new bpf_dynptr_from_mem_slice kfunc to create a dynptr from a PTR_TO_BTF_ID exposing a variable-length slice of memory, represented by the new bpf_mem_slice type. This slice is read-only, for a read-write slice we can expose a distinct type in the future.

We rely on the previous commits ensuring source objects underpinning dynptr memory are tracked correctly for invalidation to ensure when a PTR_TO_BTF_ID holding a memory slice goes away, it's corresponding dynptrs get invalidated.

---
 include/linux/bpf.h   |  5 +++++
 kernel/bpf/helpers.c  | 32 ++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c |  6 +++++-
 3 files changed, 42 insertions(+), 1 deletion(-)
include/linux/bpf.h +0 -0
index 3f0cc89c0622..9feaa9bbf0a4 100644

include/linux/bpf.h enum bpf_dynptr_type { · line 1349

Quoted 6 lines
include/linux/bpf.h +3 -0
@@ -1344,6 +1344,11 @@ enum bpf_dynptr_type {
        BPF_DYNPTR_TYPE_XDP,
 };
 
+struct bpf_mem_slice {
+       void *ptr;
+       size_t len;

for better future extensibility and to avoid big-endian issues, let's have

u32 len;
u32 __reserved; /* or flags */

?

Quoted 13 lines
include/linux/bpf.h +2 -0
+};
+
 int bpf_dynptr_check_size(u32 size);
 u32 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr);
 const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u32 len);
kernel/bpf/helpers.c +0 -0
index e3a2662f4e33..95e9c9df6062 100644
@@ -2826,6 +2826,37 @@ __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u32 dst_off,
        return 0;
 }
 

kernel/bpf/helpers.c __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u32 dst_off, · line 2838

Quoted 10 lines
kernel/bpf/helpers.c +10 -0
+/**
+ * XXX
+ */
+__bpf_kfunc int bpf_dynptr_from_mem_slice(struct bpf_mem_slice *mem_slice, u64 flags, struct bpf_dynptr *dptr__uninit)
+{
+       struct bpf_dynptr_kern *dptr = (struct bpf_dynptr_kern *)dptr__uninit;
+       int err;
+
+       if (!mem_slice)
+               return -EINVAL;

you have to initialize dynptr regardless of errors, just like bpf_dynptr_from_mem() does, so

err = -EINVAL;
goto error;

Quoted 66 lines
kernel/bpf/helpers.c +22 -0
+
+       err = bpf_dynptr_check_size(mem_slice->len);
+       if (err)
+               goto error;
+
+       /* flags is currently unsupported */
+       if (flags) {
+               err = -EINVAL;
+               goto error;
+       }
+
+       bpf_dynptr_init(dptr, mem_slice->ptr, BPF_DYNPTR_TYPE_LOCAL, 0, mem_slice->len);
+       bpf_dynptr_set_rdonly(dptr);
+
+       return 0;
+
+error:
+       bpf_dynptr_set_null(dptr);
+       return err;
+}
+
 __bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj)
 {
        return obj;
@@ -3275,6 +3306,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly)
 BTF_ID_FLAGS(func, bpf_dynptr_size)
 BTF_ID_FLAGS(func, bpf_dynptr_clone)
 BTF_ID_FLAGS(func, bpf_dynptr_copy)
+BTF_ID_FLAGS(func, bpf_dynptr_from_mem_slice, KF_TRUSTED_ARGS)
 #ifdef CONFIG_NET
 BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
 #endif
kernel/bpf/verifier.c +5 -2
index 7e09c4592038..26aa70cd5734 100644
@@ -12125,6 +12125,7 @@ enum special_kfunc_type {
        KF_bpf_res_spin_unlock,
        KF_bpf_res_spin_lock_irqsave,
        KF_bpf_res_spin_unlock_irqrestore,
+       KF_bpf_dynptr_from_mem_slice,
 };
 
 BTF_SET_START(special_kfunc_set)
@@ -12218,6 +12219,7 @@ BTF_ID(func, bpf_res_spin_lock)
 BTF_ID(func, bpf_res_spin_unlock)
 BTF_ID(func, bpf_res_spin_lock_irqsave)
 BTF_ID(func, bpf_res_spin_unlock_irqrestore)
+BTF_ID(func, bpf_dynptr_from_mem_slice)
 
 static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta)
 {
@@ -13139,7 +13141,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
                                }
                        }
 
-                       if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) {
+                       if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_mem_slice]) {
+                               dynptr_arg_type |= DYNPTR_TYPE_LOCAL;
+                       } else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) {
                                dynptr_arg_type |= DYNPTR_TYPE_SKB;
                        } else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_xdp]) {
                                dynptr_arg_type |= DYNPTR_TYPE_XDP;
--

2.47.1

author maintainer
#25 lore raw

Re: [RFC PATCH bpf-next/net v1 06/13] bpf: Introduce bpf_dynptr_from_mem_slice

Quoted 29 lines · Andrii Nakryiko
Quoted 26 lines · Kumar Kartikeya Dwivedi

Add a new bpf_dynptr_from_mem_slice kfunc to create a dynptr from a PTR_TO_BTF_ID exposing a variable-length slice of memory, represented by the new bpf_mem_slice type. This slice is read-only, for a read-write slice we can expose a distinct type in the future.

We rely on the previous commits ensuring source objects underpinning dynptr memory are tracked correctly for invalidation to ensure when a PTR_TO_BTF_ID holding a memory slice goes away, it's corresponding dynptrs get invalidated.

---
 include/linux/bpf.h   |  5 +++++
 kernel/bpf/helpers.c  | 32 ++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c |  6 +++++-
 3 files changed, 42 insertions(+), 1 deletion(-)
include/linux/bpf.h +0 -0
index 3f0cc89c0622..9feaa9bbf0a4 100644
@@ -1344,6 +1344,11 @@ enum bpf_dynptr_type {
        BPF_DYNPTR_TYPE_XDP,
 };
 
Quoted 9 lines

include/linux/bpf.h enum bpf_dynptr_type { · line 1349

Quoted 3 lines
include/linux/bpf.h +3 -0
+struct bpf_mem_slice {
+       void *ptr;
+       size_t len;

for better future extensibility and to avoid big-endian issues, let's have

u32 len;
u32 __reserved; /* or flags */

Ack.

Quoted 29 lines

?

Quoted 13 lines
include/linux/bpf.h +2 -0
+};
+
 int bpf_dynptr_check_size(u32 size);
 u32 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr);
 const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u32 len);
kernel/bpf/helpers.c +0 -0
index e3a2662f4e33..95e9c9df6062 100644
@@ -2826,6 +2826,37 @@ __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u32 dst_off,
        return 0;
 }
 

kernel/bpf/helpers.c __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u32 dst_off, · line 2838

Quoted 10 lines
kernel/bpf/helpers.c +10 -0
+/**
+ * XXX
+ */
+__bpf_kfunc int bpf_dynptr_from_mem_slice(struct bpf_mem_slice *mem_slice, u64 flags, struct bpf_dynptr *dptr__uninit)
+{
+       struct bpf_dynptr_kern *dptr = (struct bpf_dynptr_kern *)dptr__uninit;
+       int err;
+
+       if (!mem_slice)
+               return -EINVAL;

you have to initialize dynptr regardless of errors, just like bpf_dynptr_from_mem() does, so

Quoted 2 lines

err = -EINVAL;
goto error;

Thanks for catching this, will fix.

Quoted 68 lines
Quoted 68 lines
kernel/bpf/helpers.c +22 -0
+
+       err = bpf_dynptr_check_size(mem_slice->len);
+       if (err)
+               goto error;
+
+       /* flags is currently unsupported */
+       if (flags) {
+               err = -EINVAL;
+               goto error;
+       }
+
+       bpf_dynptr_init(dptr, mem_slice->ptr, BPF_DYNPTR_TYPE_LOCAL, 0, mem_slice->len);
+       bpf_dynptr_set_rdonly(dptr);
+
+       return 0;
+
+error:
+       bpf_dynptr_set_null(dptr);
+       return err;
+}
+
 __bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj)
 {
        return obj;
@@ -3275,6 +3306,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly)
 BTF_ID_FLAGS(func, bpf_dynptr_size)
 BTF_ID_FLAGS(func, bpf_dynptr_clone)
 BTF_ID_FLAGS(func, bpf_dynptr_copy)
+BTF_ID_FLAGS(func, bpf_dynptr_from_mem_slice, KF_TRUSTED_ARGS)
 #ifdef CONFIG_NET
 BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
 #endif
kernel/bpf/verifier.c +5 -2
index 7e09c4592038..26aa70cd5734 100644
@@ -12125,6 +12125,7 @@ enum special_kfunc_type {
        KF_bpf_res_spin_unlock,
        KF_bpf_res_spin_lock_irqsave,
        KF_bpf_res_spin_unlock_irqrestore,
+       KF_bpf_dynptr_from_mem_slice,
 };
 
 BTF_SET_START(special_kfunc_set)
@@ -12218,6 +12219,7 @@ BTF_ID(func, bpf_res_spin_lock)
 BTF_ID(func, bpf_res_spin_unlock)
 BTF_ID(func, bpf_res_spin_lock_irqsave)
 BTF_ID(func, bpf_res_spin_unlock_irqrestore)
+BTF_ID(func, bpf_dynptr_from_mem_slice)
 
 static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta)
 {
@@ -13139,7 +13141,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
                                }
                        }
 
-                       if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) {
+                       if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_mem_slice]) {
+                               dynptr_arg_type |= DYNPTR_TYPE_LOCAL;
+                       } else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) {
                                dynptr_arg_type |= DYNPTR_TYPE_SKB;
                        } else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_xdp]) {
                                dynptr_arg_type |= DYNPTR_TYPE_XDP;
--

2.47.1

author maintainer
#26 lore raw

RFC PATCH 7/13 bpf: Introduce per-prog stdout/stderr streams

In reply to Kumar Kartikeya Dwivedi

Introduce a set of kfuncs to implement per BPF program stdout and stderr streams. This is implemented as a linked list of dynamically allocated strings whenever a message is printed. This can be done by the kernel or the program itself using the bpf_prog_stream_vprintk kfunc. In-kernel wrappers are provided over streams to ease the process.

The idea is that everytime messages need to be dumped, the reader would pull out the whole batch of messages from the stream at once, and then pop each string one by one and start printing it out (how exactly is left up to the BPF program reading the log, but usually it will be streaming data back into a ring buffer that is consumed by user space).

The use of a lockless list warrants that we be careful about the ordering of messages. When being added, the order maintained is new to old, therefore after deletion, we must reverse the list before iterating and popping out elements from it.

Overall, this infrastructure provides NMI-safe any context printing built on top of the NMI-safe any context bpf_mem_alloc() interface.

Later patches will add support for printing splats in case of BPF arena page faults, rqspinlock deadlocks, and cond_break timeouts.

---
 include/linux/bpf.h   |  52 ++++-
 kernel/bpf/Makefile   |   2 +-
 kernel/bpf/core.c     |  12 +
 kernel/bpf/helpers.c  |  26 +--
 kernel/bpf/stream.c   | 496 ++++++++++++++++++++++++++++++++++++++++++
 kernel/bpf/syscall.c  |   2 +-
 kernel/bpf/verifier.c |  15 +-
 7 files changed, 582 insertions(+), 23 deletions(-)
 create mode 100644 kernel/bpf/stream.c
include/linux/bpf.h +51 -1
index 9feaa9bbf0a4..ae2ddd3bdea1 100644
@@ -1523,6 +1523,36 @@ struct btf_mod_pair {
 
 struct bpf_kfunc_desc_tab;
 
+enum bpf_stream_id {
+	BPF_STDOUT = 1,
+	BPF_STDERR = 2,
+};
+
+enum {
+	BPF_STREAM_ELEM_F_PAGE	= (1 << 0),
+	BPF_STREAM_ELEM_F_NEXT  = (1 << 1),
+	BPF_STREAM_ELEM_F_MASK	= (BPF_STREAM_ELEM_F_PAGE | BPF_STREAM_ELEM_F_NEXT),
+};
+
+struct bpf_stream_elem {
+	struct llist_node node;
+	struct bpf_mem_slice mem_slice;
+	union {
+		unsigned long flags;
+		struct bpf_stream_elem *next;
+	};
+	char str[];
+};
+
+struct bpf_stream_elem_batch {
+	struct llist_node *node;
+};
+
+struct bpf_stream {
+	enum bpf_stream_id stream_id;
+	struct llist_head log;
+};
+
 struct bpf_prog_aux {
 	atomic64_t refcnt;
 	u32 used_map_cnt;
@@ -1631,6 +1661,7 @@ struct bpf_prog_aux {
 		struct work_struct work;
 		struct rcu_head	rcu;
 	};
+	struct bpf_stream stream[2];
 };
 
 struct bpf_prog {
@@ -2390,6 +2421,8 @@ int  generic_map_delete_batch(struct bpf_map *map,
 struct bpf_map *bpf_map_get_curr_or_next(u32 *id);
 struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id);
 
+
+struct page *__bpf_alloc_page(int nid);
 int bpf_map_alloc_pages(const struct bpf_map *map, int nid,
 			unsigned long nr_pages, struct page **page_array);
 #ifdef CONFIG_MEMCG
@@ -3528,6 +3561,16 @@ bool btf_id_set_contains(const struct btf_id_set *set, u32 id);
 #define MAX_BPRINTF_VARARGS		12
 #define MAX_BPRINTF_BUF			1024
 
+/* Per-cpu temp buffers used by printf-like helpers to store the bprintf binary
+ * arguments representation.
+ */
+#define MAX_BPRINTF_BIN_ARGS	512
+
+struct bpf_bprintf_buffers {
+	char bin_args[MAX_BPRINTF_BIN_ARGS];
+	char buf[MAX_BPRINTF_BUF];
+};
+
 struct bpf_bprintf_data {
 	u32 *bin_args;
 	char *buf;
@@ -3535,9 +3578,16 @@ struct bpf_bprintf_data {
 	bool get_buf;
 };
 
-int bpf_bprintf_prepare(char *fmt, u32 fmt_size, const u64 *raw_args,
+int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
 			u32 num_args, struct bpf_bprintf_data *data);
 void bpf_bprintf_cleanup(struct bpf_bprintf_data *data);
+int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs);
+void bpf_put_buffers(void);
+
+void bpf_prog_stream_init(struct bpf_prog *prog);
+void bpf_prog_stream_free(struct bpf_prog *prog);
+__printf(2, 3)
+int bpf_prog_stderr_printk(struct bpf_prog *prog, const char *fmt, ...);
 
 #ifdef CONFIG_BPF_LSM
 void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype);
kernel/bpf/Makefile +1 -1
index 70502f038b92..a89575822b60 100644
@@ -14,7 +14,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
-obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o rqspinlock.o
+obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o rqspinlock.o stream.o
 ifeq ($(CONFIG_MMU)$(CONFIG_64BIT),yy)
 obj-$(CONFIG_BPF_SYSCALL) += arena.o range_tree.o
 endif
kernel/bpf/core.c +12 -0
index ba6b6118cf50..1bfc6f7ea3da 100644
@@ -134,6 +134,10 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag
 	mutex_init(&fp->aux->ext_mutex);
 	mutex_init(&fp->aux->dst_mutex);
 
+#ifdef CONFIG_BPF_SYSCALL
+	bpf_prog_stream_init(fp);
+#endif
+
 	return fp;
 }
 
@@ -2856,6 +2860,7 @@ static void bpf_prog_free_deferred(struct work_struct *work)
 	aux = container_of(work, struct bpf_prog_aux, work);
 #ifdef CONFIG_BPF_SYSCALL
 	bpf_free_kfunc_btf_tab(aux->kfunc_btf_tab);
+	bpf_prog_stream_free(aux->prog);
 #endif
 #ifdef CONFIG_CGROUP_BPF
 	if (aux->cgroup_atype != CGROUP_BPF_ATTACH_TYPE_INVALID)
@@ -2872,6 +2877,13 @@ static void bpf_prog_free_deferred(struct work_struct *work)
 	if (aux->dst_trampoline)
 		bpf_trampoline_put(aux->dst_trampoline);
 	for (i = 0; i < aux->real_func_cnt; i++) {
+#ifdef CONFIG_BPF_SYSCALL
+		/* Ensure we don't push to subprog lists. */
+		if (bpf_is_subprog(aux->func[i])) {
+			WARN_ON_ONCE(!llist_empty(&aux->func[i]->aux->stream[0].log));
+			WARN_ON_ONCE(!llist_empty(&aux->func[i]->aux->stream[1].log));
+		}
+#endif
 		/* We can just unlink the subprog poke descriptor table as
 		 * it was originally linked to the main program and is also
 		 * released along with it.
kernel/bpf/helpers.c +11 -15
index 95e9c9df6062..7f59367b1101 100644
@@ -761,22 +761,13 @@ static int bpf_trace_copy_string(char *buf, void *unsafe_ptr, char fmt_ptype,
 	return -EINVAL;
 }
 
-/* Per-cpu temp buffers used by printf-like helpers to store the bprintf binary
- * arguments representation.
- */
-#define MAX_BPRINTF_BIN_ARGS	512
-
 /* Support executing three nested bprintf helper calls on a given CPU */
 #define MAX_BPRINTF_NEST_LEVEL	3
-struct bpf_bprintf_buffers {
-	char bin_args[MAX_BPRINTF_BIN_ARGS];
-	char buf[MAX_BPRINTF_BUF];
-};
 
 static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs);
 static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);
 
-static int try_get_buffers(struct bpf_bprintf_buffers **bufs)
+int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
 {
 	int nest_level;
 
@@ -792,16 +783,21 @@ static int try_get_buffers(struct bpf_bprintf_buffers **bufs)
 	return 0;
 }
 
-void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
+void bpf_put_buffers(void)
 {
-	if (!data->bin_args && !data->buf)
-		return;
 	if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
 		return;
 	this_cpu_dec(bpf_bprintf_nest_level);
 	preempt_enable();
 }
 
+void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
+{
+	if (!data->bin_args && !data->buf)
+		return;
+	bpf_put_buffers();
+}
+
 /*
  * bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
  *
@@ -816,7 +812,7 @@ void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
  * In argument preparation mode, if 0 is returned, safe temporary buffers are
  * allocated and bpf_bprintf_cleanup should be called to free them after use.
  */
-int bpf_bprintf_prepare(char *fmt, u32 fmt_size, const u64 *raw_args,
+int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
 			u32 num_args, struct bpf_bprintf_data *data)
 {
 	bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
@@ -832,7 +828,7 @@ int bpf_bprintf_prepare(char *fmt, u32 fmt_size, const u64 *raw_args,
 		return -EINVAL;
 	fmt_size = fmt_end - fmt;
 
-	if (get_buffers && try_get_buffers(&buffers))
+	if (get_buffers && bpf_try_get_buffers(&buffers))
 		return -EBUSY;
 
 	if (data->get_bin_args) {
kernel/bpf/stream.c new file +496 -0

496 lines not on this page · raw message

kernel/bpf/syscall.c +1 -1
index 9794446bc8c6..a8e8286a3d95 100644
@@ -575,7 +575,7 @@ static bool can_alloc_pages(void)
 		!IS_ENABLED(CONFIG_PREEMPT_RT);
 }
 
-static struct page *__bpf_alloc_page(int nid)
+struct page *__bpf_alloc_page(int nid)
 {
 	if (!can_alloc_pages())
 		return try_alloc_pages(nid, 0);
kernel/bpf/verifier.c +10 -5
index 26aa70cd5734..a1ac856ad078 100644
@@ -12126,6 +12126,7 @@ enum special_kfunc_type {
 	KF_bpf_res_spin_lock_irqsave,
 	KF_bpf_res_spin_unlock_irqrestore,
 	KF_bpf_dynptr_from_mem_slice,
+	KF_bpf_stream_get,
 };
 
 BTF_SET_START(special_kfunc_set)
@@ -12220,6 +12221,7 @@ BTF_ID(func, bpf_res_spin_unlock)
 BTF_ID(func, bpf_res_spin_lock_irqsave)
 BTF_ID(func, bpf_res_spin_unlock_irqrestore)
 BTF_ID(func, bpf_dynptr_from_mem_slice)
+BTF_ID(func, bpf_stream_get)
 
 static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta)
 {
@@ -13883,10 +13885,11 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			regs[BPF_REG_0].type = PTR_TO_BTF_ID;
 			regs[BPF_REG_0].btf_id = ptr_type_id;
 
-			if (meta.func_id == special_kfunc_list[KF_bpf_get_kmem_cache])
+			if (meta.func_id == special_kfunc_list[KF_bpf_get_kmem_cache]) {
 				regs[BPF_REG_0].type |= PTR_UNTRUSTED;
-
-			if (is_iter_next_kfunc(&meta)) {
+			} else if (meta.func_id == special_kfunc_list[KF_bpf_stream_get]) {
+				regs[BPF_REG_0].type |= PTR_TRUSTED;
+			} else if (is_iter_next_kfunc(&meta)) {
 				struct bpf_reg_state *cur_iter;
 
 				cur_iter = get_iter_from_state(env->cur_state, &meta);
@@ -21520,8 +21523,10 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		   desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) {
 		insn_buf[0] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1);
 		*cnt = 1;
-	} else if (is_bpf_wq_set_callback_impl_kfunc(desc->func_id)) {
-		struct bpf_insn ld_addrs[2] = { BPF_LD_IMM64(BPF_REG_4, (long)env->prog->aux) };
+	} else if (is_bpf_wq_set_callback_impl_kfunc(desc->func_id) ||
+		   desc->func_id == special_kfunc_list[KF_bpf_stream_get]) {
+		u32 regno = is_bpf_wq_set_callback_impl_kfunc(desc->func_id) ? BPF_REG_4 : BPF_REG_2;
+		struct bpf_insn ld_addrs[2] = { BPF_LD_IMM64(regno, (long)env->prog->aux) };
 
 		insn_buf[0] = ld_addrs[0];
 		insn_buf[1] = ld_addrs[1];
Signature · 1 line
2.47.1
maintainer
#27 lore raw

Re: [RFC PATCH bpf-next/net v1 07/13] bpf: Introduce per-prog stdout/stderr streams

Quoted 25 lines · Kumar Kartikeya Dwivedi

Introduce a set of kfuncs to implement per BPF program stdout and stderr streams. This is implemented as a linked list of dynamically allocated strings whenever a message is printed. This can be done by the kernel or the program itself using the bpf_prog_stream_vprintk kfunc. In-kernel wrappers are provided over streams to ease the process.

The idea is that everytime messages need to be dumped, the reader would pull out the whole batch of messages from the stream at once, and then pop each string one by one and start printing it out (how exactly is left up to the BPF program reading the log, but usually it will be streaming data back into a ring buffer that is consumed by user space).

The use of a lockless list warrants that we be careful about the ordering of messages. When being added, the order maintained is new to old, therefore after deletion, we must reverse the list before iterating and popping out elements from it.

Overall, this infrastructure provides NMI-safe any context printing built on top of the NMI-safe any context bpf_mem_alloc() interface.

Later patches will add support for printing splats in case of BPF arena page faults, rqspinlock deadlocks, and cond_break timeouts.

---

[...]

Quoted 5 lines
kernel/bpf/stream.c new file +0 -0
new file mode 100644
index 000000000000..2019ce134310

[...]

Quoted 1 line

+static int bpf_stream_page_check_room(struct bpf_stream_page *stream_page, int len)

^^^^^^^
len is never used

Quoted 10 lines
kernel/bpf/stream.c new file +10 -0
+{
+	int min = offsetof(struct bpf_stream_elem, str[0]);
+	int consumed = stream_page->consumed;
+	int total = BPF_STREAM_PAGE_SZ;
+	int rem = max(0, total - consumed - min);
+
+	/* Let's give room of at least 8 bytes. */
+	WARN_ON_ONCE(rem % 8 != 0);
+	return rem < 8 ? 0 : rem;
+}

[...]

Quoted 12 lines
kernel/bpf/stream.c new file +12 -0
+static struct bpf_stream_elem *bpf_stream_elem_alloc(int len)
+{
+	const int max_len = ARRAY_SIZE((struct bpf_bprintf_buffers){}.buf);
+	struct bpf_stream_elem *elem;
+
+	/*
+	 * We may overflow, but we should never need more than one page size
+	 * worth of memory. This can be lifted, but we'd need to adjust the
+	 * other code to keep allocating more pages to overflow messages.
+	 */
+	BUILD_BUG_ON(max_len > BPF_STREAM_PAGE_SZ);
+	/*

kernel/bpf/stream.c line 241

Quoted 10 lines
kernel/bpf/stream.c new file +10 -0
+	 * Length denotes the amount of data to be written as part of stream element,
+	 * thus includes '\0' byte. We're capped by how much bpf_bprintf_buffers can
+	 * accomodate, therefore deny allocations that won't fit into them.
+	 */
+	if (len < 0 || len > max_len)
+		return NULL;
+
+	elem = bpf_stream_elem_alloc_from_bpf_ma(len);
+	if (!elem)
+		elem = bpf_stream_elem_alloc_from_stream_page(len);

So, the stream page is a backup mechanism, right?
I'm curious, did you compare how many messages are dropped if there is no such backup? Also, how much memory is wasted if there is no "spillover" mechanism (BPF_STREAM_ELEM_F_NEXT).
Are these complications absolutely necessary?

kernel/bpf/stream.c line 248

Quoted 7 lines
kernel/bpf/stream.c new file +7 -0
+	return elem;
+}
+
+__bpf_kfunc_start_defs();
+
+static int bpf_stream_push_str(struct bpf_stream *stream, const char *str, int len)
+{

This function accumulates elements in &stream->log w/o a cap. Why is this not a problem if e.g. user space never flushes streams for the program?

Quoted 31 lines
kernel/bpf/stream.c new file +31 -0
+	struct bpf_stream_elem *elem, *next = NULL;
+	int room = 0, rem = 0;
+
+	/*
+	 * Allocate a bpf_prog_stream_elem and push it to the bpf_prog_stream
+	 * log, elements will be popped at once and reversed to print the log.
+	 */
+	elem = bpf_stream_elem_alloc(len);
+	if (!elem)
+		return -ENOMEM;
+	room = elem->mem_slice.len;
+	if (elem->flags & BPF_STREAM_ELEM_F_NEXT) {
+		next = (struct bpf_stream_elem *)((unsigned long)elem->next & ~BPF_STREAM_ELEM_F_MASK);
+		rem = next->mem_slice.len;
+	}
+
+	memcpy(elem->str, str, room);
+	if (next)
+		memcpy(next->str, str + room, rem);
+
+	if (next) {
+		elem->node.next = &next->node;
+		next->node.next = NULL;
+
+		llist_add_batch(&elem->node, &next->node, &stream->log);
+	} else {
+		llist_add(&elem->node, &stream->log);
+	}
+
+	return 0;
+}

[...]

kernel/bpf/stream.c line 446

Quoted 8 lines
kernel/bpf/stream.c new file +8 -0
+BTF_KFUNCS_START(stream_consumer_kfunc_set)
+BTF_ID_FLAGS(func, bpf_stream_next_elem_batch, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, bpf_stream_free_elem_batch, KF_RELEASE)
+BTF_ID_FLAGS(func, bpf_stream_next_elem, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, bpf_stream_free_elem, KF_RELEASE)
+BTF_ID_FLAGS(func, bpf_prog_stream_get, KF_ACQUIRE | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_prog_stream_put, KF_RELEASE)
+BTF_KFUNCS_END(stream_consumer_kfunc_set)

This is a complicated API.
If we anticipate that users intend to write this info to ring buffers maybe just provide a function doing that and do not expose complete API?

[...]

I'll continue reading the patch-set tomorrow...

author maintainer
#28 lore raw

Re: [RFC PATCH bpf-next/net v1 07/13] bpf: Introduce per-prog stdout/stderr streams

Quoted 39 lines · Eduard Zingerman
Quoted 25 lines · Kumar Kartikeya Dwivedi

Introduce a set of kfuncs to implement per BPF program stdout and stderr streams. This is implemented as a linked list of dynamically allocated strings whenever a message is printed. This can be done by the kernel or the program itself using the bpf_prog_stream_vprintk kfunc. In-kernel wrappers are provided over streams to ease the process.

The idea is that everytime messages need to be dumped, the reader would pull out the whole batch of messages from the stream at once, and then pop each string one by one and start printing it out (how exactly is left up to the BPF program reading the log, but usually it will be streaming data back into a ring buffer that is consumed by user space).

The use of a lockless list warrants that we be careful about the ordering of messages. When being added, the order maintained is new to old, therefore after deletion, we must reverse the list before iterating and popping out elements from it.

Overall, this infrastructure provides NMI-safe any context printing built on top of the NMI-safe any context bpf_mem_alloc() interface.

Later patches will add support for printing splats in case of BPF arena page faults, rqspinlock deadlocks, and cond_break timeouts.

---

[...]

Quoted 5 lines
kernel/bpf/stream.c new file +0 -0
new file mode 100644
index 000000000000..2019ce134310

[...]

Quoted 3 lines
Quoted 1 line

+static int bpf_stream_page_check_room(struct bpf_stream_page *stream_page, int len)

^^^^^^^
len is never used

Yeah, I will get rid of the parameter.

Quoted 31 lines
Quoted 10 lines
kernel/bpf/stream.c new file +10 -0
+{
+     int min = offsetof(struct bpf_stream_elem, str[0]);
+     int consumed = stream_page->consumed;
+     int total = BPF_STREAM_PAGE_SZ;
+     int rem = max(0, total - consumed - min);
+
+     /* Let's give room of at least 8 bytes. */
+     WARN_ON_ONCE(rem % 8 != 0);
+     return rem < 8 ? 0 : rem;
+}

[...]

Quoted 18 lines
kernel/bpf/stream.c new file +18 -0
+static struct bpf_stream_elem *bpf_stream_elem_alloc(int len)
+{
+     const int max_len = ARRAY_SIZE((struct bpf_bprintf_buffers){}.buf);
+     struct bpf_stream_elem *elem;
+
+     /*
+      * We may overflow, but we should never need more than one page size
+      * worth of memory. This can be lifted, but we'd need to adjust the
+      * other code to keep allocating more pages to overflow messages.
+      */
+     BUILD_BUG_ON(max_len > BPF_STREAM_PAGE_SZ);
+     /*
+      * Length denotes the amount of data to be written as part of stream element,
+      * thus includes '\0' byte. We're capped by how much bpf_bprintf_buffers can
+      * accomodate, therefore deny allocations that won't fit into them.
+      */
+     if (len < 0 || len > max_len)
+             return NULL;
Quoted 8 lines

kernel/bpf/stream.c line 241

Quoted 4 lines
kernel/bpf/stream.c new file +4 -0
+
+     elem = bpf_stream_elem_alloc_from_bpf_ma(len);
+     if (!elem)
+             elem = bpf_stream_elem_alloc_from_stream_page(len);

So, the stream page is a backup mechanism, right?
I'm curious, did you compare how many messages are dropped if there is no such backup?

You can try disabling the backup allocation path. You will notice that the selftest starts failing.
When IRQs are disabled (so arena page fault path or rqspinlock with _irqsave), we won't be
able to refill bpf_mem_alloc() cache, and will end up dropping the message on the floor.

Quoted 3 lines

Also, how much memory is wasted if there is no
"spillover" mechanism (BPF_STREAM_ELEM_F_NEXT).
Are these complications absolutely necessary?

There's no way to know.
It depends on the free space and the message size, right. Say we have 128 bytes left in the page, the message spans 156 bytes, is it better to write 128 bytes and move the rest to the new page?

Say we have 100K messages sitting in the log that followed this pattern, let's take wasted space on average per message as 32 bytes, then that's 3.2 MB of unused memory.

Quoted 12 lines

kernel/bpf/stream.c line 248

Quoted 8 lines
kernel/bpf/stream.c new file +7 -0
+     return elem;
+}
+
+__bpf_kfunc_start_defs();
+
+static int bpf_stream_push_str(struct bpf_stream *stream, const char *str, int len)
+{

This function accumulates elements in &stream->log w/o a cap. Why is this not a problem if e.g. user space never flushes streams for the program?

It is certainly a problem, hence this bit in cover letter:

Quoted 1 line

* Enforcing memory consumption limits on the log usage, so that

in-flight log memory is bounded at runtime.

For now I will do a capacity counter per log, but maybe we want to tie this to memcg? I am not sure we can charge it in any context, but was hoping to get more opinions.

Quoted 35 lines
Quoted 32 lines
kernel/bpf/stream.c new file +31 -0
+     struct bpf_stream_elem *elem, *next = NULL;
+     int room = 0, rem = 0;
+
+     /*
+      * Allocate a bpf_prog_stream_elem and push it to the bpf_prog_stream
+      * log, elements will be popped at once and reversed to print the log.
+      */
+     elem = bpf_stream_elem_alloc(len);
+     if (!elem)
+             return -ENOMEM;
+     room = elem->mem_slice.len;
+     if (elem->flags & BPF_STREAM_ELEM_F_NEXT) {
+             next = (struct bpf_stream_elem *)((unsigned long)elem->next & ~BPF_STREAM_ELEM_F_MASK);
+             rem = next->mem_slice.len;
+     }
+
+     memcpy(elem->str, str, room);
+     if (next)
+             memcpy(next->str, str + room, rem);
+
+     if (next) {
+             elem->node.next = &next->node;
+             next->node.next = NULL;
+
+             llist_add_batch(&elem->node, &next->node, &stream->log);
+     } else {
+             llist_add(&elem->node, &stream->log);
+     }
+
+     return 0;
+}

[...]

Quoted 12 lines

kernel/bpf/stream.c line 446

Quoted 8 lines
kernel/bpf/stream.c new file +8 -0
+BTF_KFUNCS_START(stream_consumer_kfunc_set)
+BTF_ID_FLAGS(func, bpf_stream_next_elem_batch, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, bpf_stream_free_elem_batch, KF_RELEASE)
+BTF_ID_FLAGS(func, bpf_stream_next_elem, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, bpf_stream_free_elem, KF_RELEASE)
+BTF_ID_FLAGS(func, bpf_prog_stream_get, KF_ACQUIRE | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_prog_stream_put, KF_RELEASE)
+BTF_KFUNCS_END(stream_consumer_kfunc_set)

This is a complicated API.
If we anticipate that users intend to write this info to ring buffers maybe just provide a function doing that and do not expose complete API?

I don't think anyone will use these functions directly, though they can if they want to.
It's meant to be hidden behind bpftool, and macros to print stuff like bpf_printk().

We cannot pop one message at a time, since they are not in FIFO order. So we need to splice out the whole batch queued and reverse it, before popping things.
It's a consequence of using lockless lists.

The other option is using a lock to protect the list, but using rqspinlock to then report messages about rqspinlock sounds like a circular dependency.

Quoted 4 lines

[...]

I'll continue reading the patch-set tomorrow...

#29 lore raw

Re: [RFC PATCH bpf-next/net v1 07/13] bpf: Introduce per-prog stdout/stderr streams

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
kernel/bpf/stream.c new file +4 -0
+BTF_KFUNCS_START(stream_consumer_kfunc_set)
+BTF_ID_FLAGS(func, bpf_stream_next_elem_batch, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, bpf_stream_free_elem_batch, KF_RELEASE)
+BTF_ID_FLAGS(func, bpf_stream_next_elem, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)
Quoted 8 lines

kernel/bpf/stream.c line 446

Quoted 4 lines
kernel/bpf/stream.c new file +4 -0
+BTF_ID_FLAGS(func, bpf_stream_free_elem, KF_RELEASE)
+BTF_ID_FLAGS(func, bpf_prog_stream_get, KF_ACQUIRE | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_prog_stream_put, KF_RELEASE)
+BTF_KFUNCS_END(stream_consumer_kfunc_set)

This is a complicated API.
If we anticipate that users intend to write this info to ring buffers maybe just provide a function doing that and do not expose complete API?

Quoted 4 lines

I don't think anyone will use these functions directly, though they can if they want to.
It's meant to be hidden behind bpftool, and macros to print stuff like bpf_printk().

I have to agree with Eduard here.
The api feels overdesigned. I don't see how it can be reused anywhere outside of bpftool.
Instead of introducing mem_slice concept and above kfuncs, let's have one special kfunc that will take prog_id, stream_id and will move things into dynptr returning EAGAIN/ENOENT when necessary. EFAULT shouldn't be possible when the whole
SEC("syscall")
int bpftool_dump_prog_stream(void *ctx) {..}
will be one kfunc.
bpf_stream_dtor_ids won't be needed either.
Hard coding such a big logic into one kfunc is not pretty, and smaller kfuncs/building blocks would be preferred any day, but these stream_consumer_kfunc_set just don't look reusable.

author maintainer
#30 lore raw

Re: [RFC PATCH bpf-next/net v1 07/13] bpf: Introduce per-prog stdout/stderr streams

Quoted 14 lines · Alexei Starovoitov
Quoted 14 lines
Quoted 13 lines

kernel/bpf/stream.c line 446

Quoted 9 lines
kernel/bpf/stream.c new file +8 -0
+BTF_KFUNCS_START(stream_consumer_kfunc_set)
+BTF_ID_FLAGS(func, bpf_stream_next_elem_batch, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, bpf_stream_free_elem_batch, KF_RELEASE)
+BTF_ID_FLAGS(func, bpf_stream_next_elem, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, bpf_stream_free_elem, KF_RELEASE)
+BTF_ID_FLAGS(func, bpf_prog_stream_get, KF_ACQUIRE | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_prog_stream_put, KF_RELEASE)
+BTF_KFUNCS_END(stream_consumer_kfunc_set)

This is a complicated API.
If we anticipate that users intend to write this info to ring buffers maybe just provide a function doing that and do not expose complete API?

Quoted 8 lines
Quoted 4 lines

I don't think anyone will use these functions directly, though they can if they want to.
It's meant to be hidden behind bpftool, and macros to print stuff like bpf_printk().

I have to agree with Eduard here.
The api feels overdesigned. I don't see how it can be reused anywhere outside of bpftool.

Can you explain why?
Apart from syscall prog requirement which is easy to lift (but didn't bother for RFC).

Quoted 11 lines

Instead of introducing mem_slice concept and above kfuncs, let's have one special kfunc that will take prog_id, stream_id and will move things into dynptr returning EAGAIN/ENOENT when necessary. EFAULT shouldn't be possible when the whole
SEC("syscall")
int bpftool_dump_prog_stream(void *ctx) {..}
will be one kfunc.
bpf_stream_dtor_ids won't be needed either.
Hard coding such a big logic into one kfunc is not pretty, and smaller kfuncs/building blocks would be preferred any day, but these stream_consumer_kfunc_set just don't look reusable.

I don't follow. How is "shove all messages into ringbuf" more reusable? I can sympathize with the complexity argument Eduard made, ideally we'd have one pop() operation to pull out things.
We can discuss how to make things simpler.

With some documentation, it is not hard to follow:
Everytime you consume something from a stream, you splice out a batch of N messages, you can then splice out each message individually from a batch.
Both the batch and individual elements need to be freed. You can keep processing the batch until it's finished, but the better idea is to pace oneself and do N at a time.

In user space, you usually fill up your buffer's length (say PAGE_SIZE) when reading something, then delimit and parse and draw message boundaries when parsing.

You can flip the API in this patch around and use it to stream things into the program.
We just need to add one more element in prog->aux->stream[BPF_STDIN = 0]. Even without that, you can make it work on stdout. The writing is bidirectional.

User space side can obtain the stream of a prog and printk to it. Kernel side can pop and parse messages.
Probably not a bright idea to do so when in the fast path of the kernel to read strings, but in other cases, why not?
It can certainly make sense for self contained programs that want to consume options from the cmdline generically.
The generic loader can just forward the string when invoked in user space into the stream, and call some init/main function for the program before attaching it.
Your BPF init/main gets either raw stream, or prepared argv, argc blocks that the loader's infra can set up for you.

At that point all logic in the program can be self-contained, you can probably just write everything in BPF C, including setting config options etc.
There would be no need to put these strings into a ringbuf, it would just be for the BPF side to parse and make sense of.

But anyway, I will not push back too much. If everyone thinks "stream data to ringbuf kfunc" is the way to go, I'll make the change in v2.

#31 lore raw

Re: [RFC PATCH bpf-next/net v1 07/13] bpf: Introduce per-prog stdout/stderr streams

Quoted 22 lines · Kumar Kartikeya Dwivedi
Quoted 19 lines · Alexei Starovoitov
Quoted 18 lines
Quoted 13 lines

kernel/bpf/stream.c line 446

Quoted 9 lines
kernel/bpf/stream.c new file +8 -0
+BTF_KFUNCS_START(stream_consumer_kfunc_set)
+BTF_ID_FLAGS(func, bpf_stream_next_elem_batch, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, bpf_stream_free_elem_batch, KF_RELEASE)
+BTF_ID_FLAGS(func, bpf_stream_next_elem, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, bpf_stream_free_elem, KF_RELEASE)
+BTF_ID_FLAGS(func, bpf_prog_stream_get, KF_ACQUIRE | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_prog_stream_put, KF_RELEASE)
+BTF_KFUNCS_END(stream_consumer_kfunc_set)

This is a complicated API.
If we anticipate that users intend to write this info to ring buffers maybe just provide a function doing that and do not expose complete API?

I don't think anyone will use these functions directly, though they can if they want to.
It's meant to be hidden behind bpftool, and macros to print stuff like bpf_printk().

Quoted 5 lines
Quoted 3 lines

I have to agree with Eduard here.
The api feels overdesigned. I don't see how it can be reused anywhere outside of bpftool.

Can you explain why?

Because I don't see a second user for these bpf_stream_* fetchers. A special prog embedded in bpftool will be the only one afaict.

Quoted 15 lines

Apart from syscall prog requirement which is easy to lift (but didn't bother for RFC).

Quoted 11 lines

Instead of introducing mem_slice concept and above kfuncs, let's have one special kfunc that will take prog_id, stream_id and will move things into dynptr returning EAGAIN/ENOENT when necessary. EFAULT shouldn't be possible when the whole
SEC("syscall")
int bpftool_dump_prog_stream(void *ctx) {..}
will be one kfunc.
bpf_stream_dtor_ids won't be needed either.
Hard coding such a big logic into one kfunc is not pretty, and smaller kfuncs/building blocks would be preferred any day, but these stream_consumer_kfunc_set just don't look reusable.

Quoted 3 lines

I don't follow. How is "shove all messages into ringbuf" more reusable? I can sympathize with the complexity argument Eduard made, ideally we'd have one pop() operation to pull out things.

That's exactly what I'm proposing. One pop-like kfunc with prog_id, stream_id arguments.

Quoted 9 lines

We can discuss how to make things simpler.

With some documentation, it is not hard to follow:
Everytime you consume something from a stream, you splice out a batch of N messages, you can then splice out each message individually from a batch.
Both the batch and individual elements need to be freed. You can keep processing the batch until it's finished, but the better idea is to pace oneself and do N at a time.

Sure. These details can be hidden in pop() kfunc.

Quoted 8 lines

In user space, you usually fill up your buffer's length (say PAGE_SIZE) when reading something, then delimit and parse and draw message boundaries when parsing.

You can flip the API in this patch around and use it to stream things into the program.
We just need to add one more element in prog->aux->stream[BPF_STDIN = 0]. Even without that, you can make it work on stdout. The writing is bidirectional.

Hmm. I don't see the writing part here.
Currently it only consumes llist.
There is bpf_stream_vprintk() to write into it from bpf prog. That part makes sense.
bpf_stream_push_str() can also be a kfunc, but the operation here is _push_.
imo it makes sense to pair it with single _pop_, instead of next_elem, free_elem

Quoted 4 lines

User space side can obtain the stream of a prog and printk to it. Kernel side can pop and parse messages.
Probably not a bright idea to do so when in the fast path of the kernel to read strings, but in other cases, why not?

One bpf prog does bpf_stream_vprintk() while another pop-like. That makes sense.
The manual iterator over messages that requires kptr_xchg to be correct is what I see as overdesign.

Quoted 2 lines

But anyway, I will not push back too much. If everyone thinks "stream data to ringbuf kfunc" is the way to go, I'll make the change in v2.

That's not what I'm suggesting. stream is a concept makes sense. bpf_stream_vprintk() also makes sense, and it will work to push things from "user space bpf prog" into stdin of other bpf prog, and to print into stdout from that other bpf prog.

maintainer
#32 lore raw

Re: [RFC PATCH bpf-next/net v1 07/13] bpf: Introduce per-prog stdout/stderr streams

In reply to Kumar Kartikeya Dwivedi

Kumar Kartikeya Dwivedi <[email protected]> writes:

[...]

Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
kernel/bpf/stream.c new file +4 -0
+BTF_KFUNCS_START(stream_consumer_kfunc_set)
+BTF_ID_FLAGS(func, bpf_stream_next_elem_batch, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, bpf_stream_free_elem_batch, KF_RELEASE)
+BTF_ID_FLAGS(func, bpf_stream_next_elem, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)
Quoted 8 lines

kernel/bpf/stream.c line 446

Quoted 4 lines
kernel/bpf/stream.c new file +4 -0
+BTF_ID_FLAGS(func, bpf_stream_free_elem, KF_RELEASE)
+BTF_ID_FLAGS(func, bpf_prog_stream_get, KF_ACQUIRE | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_prog_stream_put, KF_RELEASE)
+BTF_KFUNCS_END(stream_consumer_kfunc_set)

This is a complicated API.
If we anticipate that users intend to write this info to ring buffers maybe just provide a function doing that and do not expose complete API?

I don't think anyone will use these functions directly, though they can if they want to.
It's meant to be hidden behind bpftool, and macros to print stuff like bpf_printk().

Quoted 8 lines

We cannot pop one message at a time, since they are not in FIFO order. So we need to splice out the whole batch queued and reverse it, before popping things.
It's a consequence of using lockless lists.

The other option is using a lock to protect the list, but using rqspinlock to then report messages about rqspinlock sounds like a circular dependency.

The API exposes 6 kfuncs and 3 data structures.
If things are exposed these things would be used, I wouldn't assume that bpftool would be the only user. I'm sure people would craft their own monitoring solutions.

Imo, exposing 9 entities is a lot. What I don't like about it, is that API is not abstract:
- user cares about getting program log to some buffer, stream or file;
- user does not care how the strings are split internally in order to make logging efficient.

If at some point we decide to change implementation this API would be hard to preserve.

Hence I suggest to instead provide a set of kfuncs that would drain message queue into user provided ringbuf or buffer, w/o exposing details.

A question: why exposing this functionality as kfuncs and not BPF syscall commands?

author maintainer
#33 lore raw

RFC PATCH 8/13 bpf: Add dump_stack() analogue to print to BPF stderr

In reply to Kumar Kartikeya Dwivedi

Introduce a kernel function which is the analogue of dump_stack() printing some useful information and the stack trace. This is not exposed to BPF programs yet, but can be made available in the future.

When we have a PC for a BPF program in the stack trace, also additionally output the filename and line number to make the trace helpful. The rest of the trace can be passed into ./decode_stacktrace.sh to obtain the line numbers for kernel symbols.

---
 include/linux/bpf.h |  2 +
 kernel/bpf/stream.c | 99 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 101 insertions(+)
include/linux/bpf.h +2 -0
index ae2ddd3bdea1..d4687da63645 100644
@@ -3588,6 +3588,8 @@ void bpf_prog_stream_init(struct bpf_prog *prog);
 void bpf_prog_stream_free(struct bpf_prog *prog);
 __printf(2, 3)
 int bpf_prog_stderr_printk(struct bpf_prog *prog, const char *fmt, ...);
+struct bpf_prog *bpf_prog_find_from_stack(void);
+void bpf_prog_stderr_dump_stack(struct bpf_prog *prog);
 
 #ifdef CONFIG_BPF_LSM
 void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype);
kernel/bpf/stream.c +99 -0

108 lines not on this page · raw message

Signature · 1 line
2.47.1
maintainer Reviewed-by on v1 8/13
#34 lore raw

Re: [RFC PATCH bpf-next/net v1 08/13] bpf: Add dump_stack() analogue to print to BPF stderr

Quoted 11 lines · Kumar Kartikeya Dwivedi

Introduce a kernel function which is the analogue of dump_stack() printing some useful information and the stack trace. This is not exposed to BPF programs yet, but can be made available in the future.

When we have a PC for a BPF program in the stack trace, also additionally output the filename and line number to make the trace helpful. The rest of the trace can be passed into ./decode_stacktrace.sh to obtain the line numbers for kernel symbols.

---

Seems to work as advertised.

[...]

Quoted 24 lines
kernel/bpf/stream.c +24 -0
+static int dump_stack_bpf_linfo(u64 ip, const char **filep)
+{
+	struct bpf_prog *prog = bpf_prog_ksym_find(ip);
+	int idx = -1, insn_start, insn_end, len;
+	struct bpf_line_info *linfo;
+	void **jited_linfo;
+	struct btf *btf;
+
+	btf = prog->aux->btf;
+	linfo = prog->aux->linfo;
+	jited_linfo = prog->aux->jited_linfo;
+
+	if (!btf || !linfo || !prog->aux->jited_linfo)
+		return -1;
+	len = prog->aux->func ? prog->aux->func[prog->aux->func_idx]->len : prog->len;
+
+	linfo = &prog->aux->linfo[prog->aux->linfo_idx];
+	jited_linfo = &prog->aux->jited_linfo[prog->aux->linfo_idx];
+
+	insn_start = linfo[0].insn_off;
+	insn_end = insn_start + len;
+
+	for (int i = 0; linfo[i].insn_off >= insn_start && linfo[i].insn_off < insn_end; i++) {
+		if (jited_linfo[i] >= (void *)ip)

kernel/bpf/stream.c int bpf_prog_stderr_printk(struct bpf_prog *prog, const char *fmt, ...) · line 562

Quoted 10 lines
kernel/bpf/stream.c +10 -0
+			break;
+		idx = i;
+	}
+
+	if (idx == -1)
+		return -1;
+
+	*filep = btf_name_by_offset(btf, linfo[idx].file_name_off);
+	*filep = strrchr(*filep, '/') ?: *filep;
+	*filep += 1;

Nit: += 1 not needed if '/' was not found?

Quoted 3 lines
kernel/bpf/stream.c +3 -0
+
+	return BPF_LINE_INFO_LINE_NUM(linfo[idx].line_col);
+}

[...]

author maintainer
#35 lore raw

RFC PATCH 9/13 bpf: Report may_goto timeout to BPF stderr

In reply to Kumar Kartikeya Dwivedi

Begin reporting may_goto timeouts to BPF program's stderr stream. Also, pace the reporting frequency by printing at most 64 instances of any errors, so that we don't end up stressing the allocator and exhausting memory in case of misbehaving programs.

---
 include/linux/bpf.h |  3 +++
 kernel/bpf/core.c   | 17 ++++++++++++++++-
 2 files changed, 19 insertions(+), 1 deletion(-)
include/linux/bpf.h +3 -0
index d4687da63645..cf057327798c 100644
@@ -1662,6 +1662,7 @@ struct bpf_prog_aux {
 		struct rcu_head	rcu;
 	};
 	struct bpf_stream stream[2];
+	atomic_long_t error_count;
 };
 
 struct bpf_prog {
@@ -3584,6 +3585,8 @@ void bpf_bprintf_cleanup(struct bpf_bprintf_data *data);
 int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs);
 void bpf_put_buffers(void);
 
+#define BPF_PROG_ERROR_COUNT_MAX 64
+
 void bpf_prog_stream_init(struct bpf_prog *prog);
 void bpf_prog_stream_free(struct bpf_prog *prog);
 __printf(2, 3)
kernel/bpf/core.c +16 -1
index 1bfc6f7ea3da..cde1be7ffa8b 100644
@@ -3151,6 +3151,19 @@ u64 __weak arch_bpf_timed_may_goto(void)
 	return 0;
 }
 
+static noinline void bpf_prog_report_may_goto_violation(void)
+{
+	struct bpf_prog *prog;
+
+	prog = bpf_prog_find_from_stack();
+	if (!prog)
+		return;
+	if (atomic_long_fetch_add(1, &prog->aux->error_count) >= BPF_PROG_ERROR_COUNT_MAX)
+		return;
+	bpf_prog_stderr_printk(prog, "ERROR: Timeout detected for may_goto instruction\n");
+	bpf_prog_stderr_dump_stack(prog);
+}
+
 u64 bpf_check_timed_may_goto(struct bpf_timed_may_goto *p)
 {
 	u64 time = ktime_get_mono_fast_ns();
@@ -3161,8 +3174,10 @@ u64 bpf_check_timed_may_goto(struct bpf_timed_may_goto *p)
 		return BPF_MAX_TIMED_LOOPS;
 	}
 	/* Check if we've exhausted our time slice, and zero count. */
-	if (time - p->timestamp >= (NSEC_PER_SEC / 4))
+	if (unlikely(time - p->timestamp >= (NSEC_PER_SEC / 4))) {
+		bpf_prog_report_may_goto_violation();
 		return 0;
+	}
 	/* Refresh the count for the stack frame. */
 	return BPF_MAX_TIMED_LOOPS;
 }
Signature · 1 line
2.47.1
author maintainer
#36 lore raw

RFC PATCH 10/13 bpf: Report rqspinlock deadlocks/timeout to BPF stderr

In reply to Kumar Kartikeya Dwivedi

Begin reporting rqspinlock deadlocks and timeout to BPF program's stderr.

---
 kernel/bpf/rqspinlock.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
kernel/bpf/rqspinlock.c +22 -0
index b896c4a75a5c..7cf6bd8e4f78 100644
@@ -666,6 +666,26 @@ EXPORT_SYMBOL_GPL(resilient_queued_spin_lock_slowpath);
 
 __bpf_kfunc_start_defs();
 
+static void bpf_prog_report_rqspinlock_violation(const char *str, void *lock, bool irqsave)
+{
+	struct rqspinlock_held *rqh = this_cpu_ptr(&rqspinlock_held_locks);
+	struct bpf_prog *prog;
+
+	prog = bpf_prog_find_from_stack();
+	if (!prog)
+		return;
+	if (atomic_long_fetch_add(1, &prog->aux->error_count) >= BPF_PROG_ERROR_COUNT_MAX)
+		return;
+	bpf_prog_stderr_printk(prog, "ERROR: %s for bpf_res_spin_lock%s\n", str, irqsave ? "_irqsave" : "");
+	bpf_prog_stderr_printk(prog, "Attempted lock   = 0x%px\n", lock);
+	bpf_prog_stderr_printk(prog, "Total held locks = %d\n", rqh->cnt);
+	for (int i = 0; i < min(RES_NR_HELD, rqh->cnt); i++)
+		bpf_prog_stderr_printk(prog, "Held lock[%2d] = 0x%px\n", i, rqh->locks[i]);
+	bpf_prog_stderr_dump_stack(prog);
+}
+
+#define REPORT_STR(ret) ({ (ret) == -ETIMEDOUT ? "Timeout detected" : "AA or ABBA deadlock detected"; })
+
 __bpf_kfunc int bpf_res_spin_lock(struct bpf_res_spin_lock *lock)
 {
 	int ret;
@@ -676,6 +696,7 @@ __bpf_kfunc int bpf_res_spin_lock(struct bpf_res_spin_lock *lock)
 	preempt_disable();
 	ret = res_spin_lock((rqspinlock_t *)lock);
 	if (unlikely(ret)) {
+		bpf_prog_report_rqspinlock_violation(REPORT_STR(ret), lock, false);
 		preempt_enable();
 		return ret;
 	}
@@ -698,6 +719,7 @@ __bpf_kfunc int bpf_res_spin_lock_irqsave(struct bpf_res_spin_lock *lock, unsign
 	local_irq_save(flags);
 	ret = res_spin_lock((rqspinlock_t *)lock);
 	if (unlikely(ret)) {
+		bpf_prog_report_rqspinlock_violation(REPORT_STR(ret), lock, true);
 		local_irq_restore(flags);
 		preempt_enable();
 		return ret;
Signature · 1 line
2.47.1
author maintainer
#37 lore raw

RFC PATCH 11/13 bpf: Report arena faults to BPF stderr

In reply to Kumar Kartikeya Dwivedi

Begin reporting arena page faults and the faulting address to BPF program's stderr, for now limited to x86, but arm64 support should be easy to add.

---
 arch/x86/net/bpf_jit_comp.c | 21 ++++++++++++++++++---
 include/linux/bpf.h         |  1 +
 kernel/bpf/arena.c          | 14 ++++++++++++++
 3 files changed, 33 insertions(+), 3 deletions(-)
arch/x86/net/bpf_jit_comp.c +18 -3
index 9e5fe2ba858f..434a6eae4621 100644
@@ -1384,15 +1384,27 @@ static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size,
 }
 
 #define DONT_CLEAR 1
+#define ARENA_FAULT (1 << 8)
 
 bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs)
 {
-	u32 reg = x->fixup >> 8;
+	u32 arena_reg = (x->fixup >> 8) & 0xff;
+	bool is_arena = !!arena_reg;
+	u32 reg = x->fixup >> 16;
+	unsigned long addr;
+
+	/* Read here, if src_reg is dst_reg for load, we'll write 0 to it. */
+	if (is_arena)
+		addr = *(unsigned long *)((void *)regs + arena_reg);
 
 	/* jump over faulting load and clear dest register */
 	if (reg != DONT_CLEAR)
 		*(unsigned long *)((void *)regs + reg) = 0;
 	regs->ip += x->fixup & 0xff;
+
+	if (is_arena)
+		bpf_prog_report_arena_violation(reg == DONT_CLEAR, addr);
+
 	return true;
 }
 
@@ -2043,7 +2055,10 @@ st:			if (is_imm8(insn->off))
 				ex->data = EX_TYPE_BPF;
 
 				ex->fixup = (prog - start_of_ldx) |
-					((BPF_CLASS(insn->code) == BPF_LDX ? reg2pt_regs[dst_reg] : DONT_CLEAR) << 8);
+					((BPF_CLASS(insn->code) == BPF_LDX ? reg2pt_regs[dst_reg] : DONT_CLEAR) << 16)
+					| ((BPF_CLASS(insn->code) == BPF_LDX ? reg2pt_regs[src_reg] : reg2pt_regs[dst_reg])<< 8);
+				/* Ensure src_reg offset fits in 1 byte. */
+				BUILD_BUG_ON(sizeof(struct pt_regs) > U8_MAX);
 			}
 			break;
 
@@ -2161,7 +2176,7 @@ st:			if (is_imm8(insn->off))
 				 * End result: x86 insn "mov rbx, qword ptr [rax+0x14]"
 				 * of 4 bytes will be ignored and rbx will be zero inited.
 				 */
-				ex->fixup = (prog - start_of_ldx) | (reg2pt_regs[dst_reg] << 8);
+				ex->fixup = (prog - start_of_ldx) | (reg2pt_regs[dst_reg] << 16);
 			}
 			break;
 
include/linux/bpf.h +1 -0
index cf057327798c..9fae22dde926 100644
@@ -3593,6 +3593,7 @@ __printf(2, 3)
 int bpf_prog_stderr_printk(struct bpf_prog *prog, const char *fmt, ...);
 struct bpf_prog *bpf_prog_find_from_stack(void);
 void bpf_prog_stderr_dump_stack(struct bpf_prog *prog);
+void bpf_prog_report_arena_violation(bool write, unsigned long addr);
 
 #ifdef CONFIG_BPF_LSM
 void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype);
kernel/bpf/arena.c +14 -0
index 0d56cea71602..593a93195cdc 100644
@@ -590,3 +590,17 @@ static int __init kfunc_init(void)
 	return register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &common_kfunc_set);
 }
 late_initcall(kfunc_init);
+
+void bpf_prog_report_arena_violation(bool write, unsigned long addr)
+{
+	struct bpf_prog *prog;
+
+	prog = bpf_prog_find_from_stack();
+	if (!prog)
+		return;
+	if (atomic_long_fetch_add(1, &prog->aux->error_count) >= BPF_PROG_ERROR_COUNT_MAX)
+		return;
+	bpf_prog_stderr_printk(prog, "ERROR: Arena %s access at unmapped address 0x%lx\n",
+			       write ? "WRITE" : "READ", addr);
+	bpf_prog_stderr_dump_stack(prog);
+}
Signature · 1 line
2.47.1
author maintainer
#38 lore raw

RFC PATCH 12/13 bpftool: Add support for dumping streams

In reply to Kumar Kartikeya Dwivedi

Add bpftool support for dumping streams of a given BPF program.
TODO: JSON and filepath support.

---
 tools/bpf/bpftool/Makefile              |  2 +-
 tools/bpf/bpftool/prog.c                | 71 +++++++++++++++++-
 tools/bpf/bpftool/skeleton/stream.bpf.c | 96 +++++++++++++++++++++++++
 3 files changed, 166 insertions(+), 3 deletions(-)
 create mode 100644 tools/bpf/bpftool/skeleton/stream.bpf.c
tools/bpf/bpftool/Makefile +1 -1
index 9e9a5f006cd2..eb908223c3bb 100644
@@ -234,7 +234,7 @@ $(OUTPUT)%.bpf.o: skeleton/%.bpf.c $(OUTPUT)vmlinux.h $(LIBBPF_BOOTSTRAP)
 $(OUTPUT)%.skel.h: $(OUTPUT)%.bpf.o $(BPFTOOL_BOOTSTRAP)
 	$(QUIET_GEN)$(BPFTOOL_BOOTSTRAP) gen skeleton $< > $@
 
-$(OUTPUT)prog.o: $(OUTPUT)profiler.skel.h
+$(OUTPUT)prog.o: $(OUTPUT)profiler.skel.h $(OUTPUT)stream.skel.h
 
 $(OUTPUT)pids.o: $(OUTPUT)pid_iter.skel.h
 
tools/bpf/bpftool/prog.c +69 -2

120 lines not on this page · raw message

tools/bpf/bpftool/skeleton/stream.bpf.c new file +96 -0

96 lines not on this page · raw message

Signature · 1 line
2.47.1
maintainer
#39 lore raw

Re: [RFC PATCH bpf-next/net v1 12/13] bpftool: Add support for dumping streams

When I add a test as at the bottom of the email I get the following output:

    test_stream_cond_break:PASS:load 0 nsec
    test_stream_cond_break:PASS:run_opts 0 nsec
    test_stream_cond_break:PASS:info_by_fd 0 nsec
    ERROR: Timeout detected for may_goto instruction
    CPU: 6 UID: 0 PID: 206 Comm: test_progs
    Call trace:
     bpf_prog_stderr_dump_stack+0xde/0x160
     bpf_check_timed_may_goto+0x5d/0x90
     arch_bpf_timed_may_goto+0x21/0x40
     bpf_prog_34056decf3b2fb2f_long_loop+0x49/0x57: [stream_cond_break.c:8]
     bpf_prog_run_pin_on_cpu+0x5f/0x110
     bpf_prog_test_run_syscall+0x205/0x320
     bpf_prog_test_run+0x234/0x2a0
     __sys_bpf+0x2d7/0x570
     __x64_sys_bpf+0x7c/0x90
     do_syscall_64+0x79/0x120
     entry_SYSCALL_64_after_hwframe+0x76/0x7e
    
    prog_dump_stream: ret==-22
    test_stream_cond_break:FAIL:./tools/sbin/bpftool prog dump stderr id 8
     unexpected error: 59904 (errno 95)

Am I doing something wrong or EINVAL should not be returned from prog_dump_stream in this case?

---

tools/bpf/bpftool/prog.c +1 -0
index d0800fec9c3d..64386737a364 100644
@@ -747,6 +747,7 @@ prog_dump_stream(struct bpf_prog_info *info, enum dump_mode mode, const char *fi
 		ret = -EINVAL;
 end:
 	stream_bpf__destroy(skel);
+	fprintf(stderr, "prog_dump_stream: ret==%d\n", ret);
 	return ret;
 }
 
tools/testing/selftests/bpf/prog_tests/stream_cond_break.c new file +42 -0

42 lines not on this page · raw message

tools/testing/selftests/bpf/progs/stream_cond_break.c new file +12 -0
new file mode 100644
index 000000000000..47c2e5f1b8fd
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+#include "bpf_experimental.h"
+
+SEC("syscall")
+int long_loop(const void *ctx)
+{
+	for (;;)
+		cond_break;
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
maintainer
#40 lore raw

Re: [RFC PATCH bpf-next/net v1 12/13] bpftool: Add support for dumping streams

In reply to Kumar Kartikeya Dwivedi

2025-04-14 09:14 UTC-0700 ~ Kumar Kartikeya Dwivedi <[email protected]>

Quoted 1 line

Add bpftool support for dumping streams of a given BPF program.

Thanks for adding bpftool support!

Quoted 1 line

TODO: JSON and filepath support.

... plus documentation (man page), and bash completion please.

Quoted 144 lines
---
 tools/bpf/bpftool/Makefile              |  2 +-
 tools/bpf/bpftool/prog.c                | 71 +++++++++++++++++-
 tools/bpf/bpftool/skeleton/stream.bpf.c | 96 +++++++++++++++++++++++++
 3 files changed, 166 insertions(+), 3 deletions(-)
 create mode 100644 tools/bpf/bpftool/skeleton/stream.bpf.c
tools/bpf/bpftool/Makefile +1 -1
index 9e9a5f006cd2..eb908223c3bb 100644
@@ -234,7 +234,7 @@ $(OUTPUT)%.bpf.o: skeleton/%.bpf.c $(OUTPUT)vmlinux.h $(LIBBPF_BOOTSTRAP)
 $(OUTPUT)%.skel.h: $(OUTPUT)%.bpf.o $(BPFTOOL_BOOTSTRAP)
 	$(QUIET_GEN)$(BPFTOOL_BOOTSTRAP) gen skeleton $< > $@
 
-$(OUTPUT)prog.o: $(OUTPUT)profiler.skel.h
+$(OUTPUT)prog.o: $(OUTPUT)profiler.skel.h $(OUTPUT)stream.skel.h
 
 $(OUTPUT)pids.o: $(OUTPUT)pid_iter.skel.h
 
tools/bpf/bpftool/prog.c +67 -2
index f010295350be..d0800fec9c3d 100644
@@ -35,12 +35,16 @@
 #include "main.h"
 #include "xlated_dumper.h"
 
+#include "stream.skel.h"
+
 #define BPF_METADATA_PREFIX "bpf_metadata_"
 #define BPF_METADATA_PREFIX_LEN (sizeof(BPF_METADATA_PREFIX) - 1)
 
 enum dump_mode {
 	DUMP_JITED,
 	DUMP_XLATED,
+	DUMP_STDOUT,
+	DUMP_STDERR,
 };
 
 static const bool attach_types[] = {
@@ -697,6 +701,55 @@ static int do_show(int argc, char **argv)
 	return err;
 }
 
+static int process_stream_sample(void *ctx, void *data, size_t len)
+{
+	FILE *file = ctx;
+
+	fprintf(file, "%s", (char *)data);
+	fflush(file);
+	return 0;
+}
+
+static int
+prog_dump_stream(struct bpf_prog_info *info, enum dump_mode mode, const char *filepath)
+{
+	FILE *file = mode == DUMP_STDOUT ? stdout : stderr;
+	LIBBPF_OPTS(bpf_test_run_opts, opts);
+	struct ring_buffer *ringbuf;
+	struct stream_bpf *skel;
+	int map_fd, ret = -1;
+
+	__u32 prog_id = info->id;
+	__u32 stream_id = mode == DUMP_STDOUT ? 1 : 2;
+
+	skel = stream_bpf__open_and_load();
+	if (!skel)
+		return -errno;
+	skel->bss->prog_id = prog_id;
+	skel->bss->stream_id = stream_id;
+
+	//TODO(kkd): Filepath handling
+	map_fd = bpf_map__fd(skel->maps.ringbuf);
+	ringbuf = ring_buffer__new(map_fd, process_stream_sample, file, NULL);
+	if (!ringbuf) {
+		ret = -errno;
+		goto end;
+	}
+	do {
+		skel->bss->written_count = skel->bss->written_size = 0;
+		ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.bpftool_dump_prog_stream), &opts);
+		ret = -EINVAL;
+		if (ring_buffer__consume_n(ringbuf, skel->bss->written_count) != skel->bss->written_count)
+			goto end;
+	} while (!ret && opts.retval == EAGAIN);
+
+	if (opts.retval != 0)
+		ret = -EINVAL;
+end:
+	stream_bpf__destroy(skel);
+	return ret;
+}
+
 static int
 prog_dump(struct bpf_prog_info *info, enum dump_mode mode,
 	  char *filepath, bool opcodes, bool visual, bool linum)
@@ -719,13 +772,15 @@ prog_dump(struct bpf_prog_info *info, enum dump_mode mode,
 		}
 		buf = u64_to_ptr(info->jited_prog_insns);
 		member_len = info->jited_prog_len;
-	} else {	/* DUMP_XLATED */
+	} else if (mode == DUMP_XLATED) {	/* DUMP_XLATED */
 		if (info->xlated_prog_len == 0 || !info->xlated_prog_insns) {
 			p_err("error retrieving insn dump: kernel.kptr_restrict set?");
 			return -1;
 		}
 		buf = u64_to_ptr(info->xlated_prog_insns);
 		member_len = info->xlated_prog_len;
+	} else if (mode == DUMP_STDOUT || mode == DUMP_STDERR) {
+		return prog_dump_stream(info, mode, filepath);
 	}
 
 	if (info->btf_id) {
@@ -898,8 +953,10 @@ static int do_dump(int argc, char **argv)
 		mode = DUMP_JITED;
 	} else if (is_prefix(*argv, "xlated")) {
 		mode = DUMP_XLATED;
+	} else if (is_prefix(*argv, "stdout") || is_prefix(*argv, "stderr")) {
+		mode = is_prefix(*argv, "stdout") ? DUMP_STDOUT : DUMP_STDERR;
 	} else {
-		p_err("expected 'xlated' or 'jited', got: %s", *argv);
+		p_err("expected 'stdout', 'stderr', 'xlated' or 'jited', got: %s", *argv);
 		return -1;
 	}
 	NEXT_ARG();
@@ -950,6 +1007,14 @@ static int do_dump(int argc, char **argv)
 		}
 	}
 
+	if (mode == DUMP_STDOUT || mode == DUMP_STDERR) {
+		if (opcodes || visual || linum) {
+			p_err("'%s' is not compatible with 'opcodes', 'visual', or 'linum'",
+			      mode == DUMP_STDOUT ? "stdout" : "stderr");
+			goto exit_close;
+		}
+	}
+
 	if (filepath && (opcodes || visual || linum)) {
 		p_err("'file' is not compatible with 'opcodes', 'visual', or 'linum'");
 		goto exit_close;

tools/bpf/bpftool/prog.c static int do_help(int argc, char **argv) · line 2537

Quoted 5 lines
tools/bpf/bpftool/prog.c +2 -0
@@ -2468,6 +2533,8 @@ static int do_help(int argc, char **argv)
 		"Usage: %1$s %2$s { show | list } [PROG]\n"
 		"       %1$s %2$s dump xlated PROG [{ file FILE | [opcodes] [linum] [visual] }]\n"
 		"       %1$s %2$s dump jited  PROG [{ file FILE | [opcodes] [linum] }]\n"
+		"	%1$s %2$s dump stdout PROG [{ file FILE }]\n"
+		"	%1$s %2$s dump stderr PROG [{ file FILE }]\n"

I'm not sure "prog dump" is the best subcommand for these features. The "dump" has been associated with printing the program itself, either translated or as JITed instructions. Here we display something else; it's closer to "prog tracelog", that we use to dump the trace pipe. And stdout/stderr are streams anyway, I'm not sure that "dumping" them is the most accurate term.

How about "prog trace (stdout|stderr)"? Bonus: you don't have to handle opcodes/linum/visual, and don't have to add support for "filepath".

tools/bpf/bpftool/skeleton/stream.bpf.c line 1

Quoted 10 lines
tools/bpf/bpftool/prog.c +0 -0
 		"       %1$s %2$s pin   PROG FILE\n"
 		"       %1$s %2$s { load | loadall } OBJ  PATH \\\n"
 		"                         [type TYPE] [{ offload_dev | xdpmeta_dev } NAME] \\\n"
tools/bpf/bpftool/skeleton/stream.bpf.c new file +1 -0
new file mode 100644
index 000000000000..31b5933e0384
@@ -0,0 +1,96 @@
+// SPDX-License-Identifier: GPL-2.0

Bpftool is dual-licensed (GPL-2.0-only OR BSD-2-Clause), please consider using the same here.

Quoted 31 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +31 -0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_helpers.h>
+
+struct {
+	__uint(type, BPF_MAP_TYPE_RINGBUF);
+	__uint(max_entries, 1024 * 1024);
+} ringbuf SEC(".maps");
+
+struct value {
+	struct bpf_stream_elem_batch __kptr *batch;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__type(key, int);
+	__type(value, struct value);
+	__uint(max_entries, 1);
+} array SEC(".maps");
+
+int written_size;
+int written_count;
+int stream_id;
+int prog_id;
+
+#define ENOENT 2
+#define EAGAIN 11
+#define EFAULT 14
+
+SEC("syscall")

tools/bpf/bpftool/skeleton/stream.bpf.c line 42

Quoted 10 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +10 -0
+int bpftool_dump_prog_stream(void *ctx)
+{
+	struct bpf_stream_elem_batch *elem_batch;
+	struct bpf_stream_elem *elem;
+	struct bpf_stream *stream;
+	bool cont = false;
+	struct value *v;
+	bool ret = 0;
+
+	stream = bpf_prog_stream_get(BPF_STDERR, prog_id);

Calls to these new kfuncs will break compilation on older systems that don't support them yet (and don't have the definition in their vmlinux.h). We should provide fallback definitions to make sure that the program compiles.

Quoted 44 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +44 -0
+	if (!stream)
+		return ENOENT;
+
+	v = bpf_map_lookup_elem(&array, &(int){0});
+
+	if (v->batch)
+		elem_batch = bpf_kptr_xchg(&v->batch, NULL);
+	else
+		elem_batch = bpf_stream_next_elem_batch(stream);
+	if (!elem_batch)
+		goto end;
+
+	bpf_repeat(BPF_MAX_LOOPS) {
+		struct bpf_dynptr dst_dptr, src_dptr;
+		int size;
+
+		elem = bpf_stream_next_elem(elem_batch);
+		if (!elem)
+			break;
+		size = elem->mem_slice.len;
+
+		if (bpf_dynptr_from_mem_slice(&elem->mem_slice, 0, &src_dptr))
+			ret = EFAULT;
+		if (bpf_ringbuf_reserve_dynptr(&ringbuf, size, 0, &dst_dptr))
+			ret = EFAULT;
+		if (bpf_dynptr_copy(&dst_dptr, 0, &src_dptr, 0, size))
+			ret = EFAULT;
+		bpf_ringbuf_submit_dynptr(&dst_dptr, 0);
+
+		written_count++;
+		written_size += size;
+
+		bpf_stream_free_elem(elem);
+
+		/* Probe and exit if no more space, probe for twice the typical size.*/
+		if (bpf_ringbuf_reserve_dynptr(&ringbuf, 2048, 0, &dst_dptr))
+			cont = true;
+		bpf_ringbuf_discard_dynptr(&dst_dptr, 0);
+
+		if (ret || cont)
+			break;
+	}
+
+	if (cont)

tools/bpf/bpftool/skeleton/stream.bpf.c line 96

Quoted 10 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +10 -0
+		elem_batch = bpf_kptr_xchg(&v->batch, elem_batch);
+	if (elem_batch)
+		bpf_stream_free_elem_batch(elem_batch);
+end:
+	bpf_prog_stream_put(stream);
+
+	return ret ?: (cont ? EAGAIN : 0);
+}
+
+char _license[] SEC("license") = "GPL";

Same note on the license, other programs used by bpftool have license string "Dual BSD/GPL".

author maintainer
#41 lore raw

Re: [RFC PATCH bpf-next/net v1 12/13] bpftool: Add support for dumping streams

Quoted 5 lines · Quentin Monnet

2025-04-14 09:14 UTC-0700 ~ Kumar Kartikeya Dwivedi <[email protected]>

Quoted 1 line

Add bpftool support for dumping streams of a given BPF program.

Quoted 8 lines

Thanks for adding bpftool support!

Quoted 1 line

TODO: JSON and filepath support.

... plus documentation (man page), and bash completion please.

Done.

Quoted 159 lines
Quoted 145 lines
---
 tools/bpf/bpftool/Makefile              |  2 +-
 tools/bpf/bpftool/prog.c                | 71 +++++++++++++++++-
 tools/bpf/bpftool/skeleton/stream.bpf.c | 96 +++++++++++++++++++++++++
 3 files changed, 166 insertions(+), 3 deletions(-)
 create mode 100644 tools/bpf/bpftool/skeleton/stream.bpf.c
tools/bpf/bpftool/Makefile +1 -1
index 9e9a5f006cd2..eb908223c3bb 100644
@@ -234,7 +234,7 @@ $(OUTPUT)%.bpf.o: skeleton/%.bpf.c $(OUTPUT)vmlinux.h $(LIBBPF_BOOTSTRAP)
 $(OUTPUT)%.skel.h: $(OUTPUT)%.bpf.o $(BPFTOOL_BOOTSTRAP)
      $(QUIET_GEN)$(BPFTOOL_BOOTSTRAP) gen skeleton $< > $@
 
-$(OUTPUT)prog.o: $(OUTPUT)profiler.skel.h
+$(OUTPUT)prog.o: $(OUTPUT)profiler.skel.h $(OUTPUT)stream.skel.h
 
 $(OUTPUT)pids.o: $(OUTPUT)pid_iter.skel.h
 
tools/bpf/bpftool/prog.c +67 -2
index f010295350be..d0800fec9c3d 100644
@@ -35,12 +35,16 @@
 #include "main.h"
 #include "xlated_dumper.h"
 
+#include "stream.skel.h"
+
 #define BPF_METADATA_PREFIX "bpf_metadata_"
 #define BPF_METADATA_PREFIX_LEN (sizeof(BPF_METADATA_PREFIX) - 1)
 
 enum dump_mode {
      DUMP_JITED,
      DUMP_XLATED,
+     DUMP_STDOUT,
+     DUMP_STDERR,
 };
 
 static const bool attach_types[] = {
@@ -697,6 +701,55 @@ static int do_show(int argc, char **argv)
      return err;
 }
 
+static int process_stream_sample(void *ctx, void *data, size_t len)
+{
+     FILE *file = ctx;
+
+     fprintf(file, "%s", (char *)data);
+     fflush(file);
+     return 0;
+}
+
+static int
+prog_dump_stream(struct bpf_prog_info *info, enum dump_mode mode, const char *filepath)
+{
+     FILE *file = mode == DUMP_STDOUT ? stdout : stderr;
+     LIBBPF_OPTS(bpf_test_run_opts, opts);
+     struct ring_buffer *ringbuf;
+     struct stream_bpf *skel;
+     int map_fd, ret = -1;
+
+     __u32 prog_id = info->id;
+     __u32 stream_id = mode == DUMP_STDOUT ? 1 : 2;
+
+     skel = stream_bpf__open_and_load();
+     if (!skel)
+             return -errno;
+     skel->bss->prog_id = prog_id;
+     skel->bss->stream_id = stream_id;
+
+     //TODO(kkd): Filepath handling
+     map_fd = bpf_map__fd(skel->maps.ringbuf);
+     ringbuf = ring_buffer__new(map_fd, process_stream_sample, file, NULL);
+     if (!ringbuf) {
+             ret = -errno;
+             goto end;
+     }
+     do {
+             skel->bss->written_count = skel->bss->written_size = 0;
+             ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.bpftool_dump_prog_stream), &opts);
+             ret = -EINVAL;
+             if (ring_buffer__consume_n(ringbuf, skel->bss->written_count) != skel->bss->written_count)
+                     goto end;
+     } while (!ret && opts.retval == EAGAIN);
+
+     if (opts.retval != 0)
+             ret = -EINVAL;
+end:
+     stream_bpf__destroy(skel);
+     return ret;
+}
+
 static int
 prog_dump(struct bpf_prog_info *info, enum dump_mode mode,
        char *filepath, bool opcodes, bool visual, bool linum)
@@ -719,13 +772,15 @@ prog_dump(struct bpf_prog_info *info, enum dump_mode mode,
              }
              buf = u64_to_ptr(info->jited_prog_insns);
              member_len = info->jited_prog_len;
-     } else {        /* DUMP_XLATED */
+     } else if (mode == DUMP_XLATED) {       /* DUMP_XLATED */
              if (info->xlated_prog_len == 0 || !info->xlated_prog_insns) {
                      p_err("error retrieving insn dump: kernel.kptr_restrict set?");
                      return -1;
              }
              buf = u64_to_ptr(info->xlated_prog_insns);
              member_len = info->xlated_prog_len;
+     } else if (mode == DUMP_STDOUT || mode == DUMP_STDERR) {
+             return prog_dump_stream(info, mode, filepath);
      }
 
      if (info->btf_id) {
@@ -898,8 +953,10 @@ static int do_dump(int argc, char **argv)
              mode = DUMP_JITED;
      } else if (is_prefix(*argv, "xlated")) {
              mode = DUMP_XLATED;
+     } else if (is_prefix(*argv, "stdout") || is_prefix(*argv, "stderr")) {
+             mode = is_prefix(*argv, "stdout") ? DUMP_STDOUT : DUMP_STDERR;
      } else {
-             p_err("expected 'xlated' or 'jited', got: %s", *argv);
+             p_err("expected 'stdout', 'stderr', 'xlated' or 'jited', got: %s", *argv);
              return -1;
      }
      NEXT_ARG();
@@ -950,6 +1007,14 @@ static int do_dump(int argc, char **argv)
              }
      }
 
+     if (mode == DUMP_STDOUT || mode == DUMP_STDERR) {
+             if (opcodes || visual || linum) {
+                     p_err("'%s' is not compatible with 'opcodes', 'visual', or 'linum'",
+                           mode == DUMP_STDOUT ? "stdout" : "stderr");
+                     goto exit_close;
+             }
+     }
+
      if (filepath && (opcodes || visual || linum)) {
              p_err("'file' is not compatible with 'opcodes', 'visual', or 'linum'");
              goto exit_close;

tools/bpf/bpftool/prog.c static int do_help(int argc, char **argv) · line 2537

Quoted 5 lines
tools/bpf/bpftool/prog.c +2 -0
@@ -2468,6 +2533,8 @@ static int do_help(int argc, char **argv)
              "Usage: %1$s %2$s { show | list } [PROG]\n"
              "       %1$s %2$s dump xlated PROG [{ file FILE | [opcodes] [linum] [visual] }]\n"
              "       %1$s %2$s dump jited  PROG [{ file FILE | [opcodes] [linum] }]\n"
+             "       %1$s %2$s dump stdout PROG [{ file FILE }]\n"
+             "       %1$s %2$s dump stderr PROG [{ file FILE }]\n"

I'm not sure "prog dump" is the best subcommand for these features. The "dump" has been associated with printing the program itself, either translated or as JITed instructions. Here we display something else; it's closer to "prog tracelog", that we use to dump the trace pipe. And stdout/stderr are streams anyway, I'm not sure that "dumping" them is the most accurate term.

Quoted 2 lines

How about "prog trace (stdout|stderr)"? Bonus: you don't have to handle opcodes/linum/visual, and don't have to add support for "filepath".

I went with prog tracelog {stdout|stderr}, since trace probably feels a bit odd in this context.
Tracelog alone will print trace pipe, when passed stdout/stderr it gives the stream output.
Let me know what you think.

Quoted 16 lines

tools/bpf/bpftool/skeleton/stream.bpf.c line 1

Quoted 12 lines
tools/bpf/bpftool/prog.c +0 -0
              "       %1$s %2$s pin   PROG FILE\n"
              "       %1$s %2$s { load | loadall } OBJ  PATH \\\n"
              "                         [type TYPE] [{ offload_dev | xdpmeta_dev } NAME] \\\n"
tools/bpf/bpftool/skeleton/stream.bpf.c new file +1 -0
new file mode 100644
index 000000000000..31b5933e0384
@@ -0,0 +1,96 @@
+// SPDX-License-Identifier: GPL-2.0

Bpftool is dual-licensed (GPL-2.0-only OR BSD-2-Clause), please consider using the same here.

Done.

Quoted 41 lines
Quoted 41 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +39 -0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_helpers.h>
+
+struct {
+     __uint(type, BPF_MAP_TYPE_RINGBUF);
+     __uint(max_entries, 1024 * 1024);
+} ringbuf SEC(".maps");
+
+struct value {
+     struct bpf_stream_elem_batch __kptr *batch;
+};
+
+struct {
+     __uint(type, BPF_MAP_TYPE_ARRAY);
+     __type(key, int);
+     __type(value, struct value);
+     __uint(max_entries, 1);
+} array SEC(".maps");
+
+int written_size;
+int written_count;
+int stream_id;
+int prog_id;
+
+#define ENOENT 2
+#define EAGAIN 11
+#define EFAULT 14
+
+SEC("syscall")
+int bpftool_dump_prog_stream(void *ctx)
+{
+     struct bpf_stream_elem_batch *elem_batch;
+     struct bpf_stream_elem *elem;
+     struct bpf_stream *stream;
+     bool cont = false;
+     struct value *v;
+     bool ret = 0;
Quoted 8 lines

tools/bpf/bpftool/skeleton/stream.bpf.c line 42

Quoted 2 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +2 -0
+
+     stream = bpf_prog_stream_get(BPF_STDERR, prog_id);

Calls to these new kfuncs will break compilation on older systems that don't support them yet (and don't have the definition in their vmlinux.h). We should provide fallback definitions to make sure that the program compiles.

This is the only thing I haven't yet addressed in v2, because it seemed a bit ugly.
I tried adding kfunc declarations, but those aren't enough.
We rely on structs introduced and read in this patch.
So I think vmlinux.h needs to be dropped, but it means adding a lot more than just the declarations, all types, plus any types they transitively depend on.
Maybe there is a better way (like detecting compilation failure and skipping?). But if not, I will address like above in v3.

Quoted 52 lines
Quoted 52 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +50 -0
+     if (!stream)
+             return ENOENT;
+
+     v = bpf_map_lookup_elem(&array, &(int){0});
+
+     if (v->batch)
+             elem_batch = bpf_kptr_xchg(&v->batch, NULL);
+     else
+             elem_batch = bpf_stream_next_elem_batch(stream);
+     if (!elem_batch)
+             goto end;
+
+     bpf_repeat(BPF_MAX_LOOPS) {
+             struct bpf_dynptr dst_dptr, src_dptr;
+             int size;
+
+             elem = bpf_stream_next_elem(elem_batch);
+             if (!elem)
+                     break;
+             size = elem->mem_slice.len;
+
+             if (bpf_dynptr_from_mem_slice(&elem->mem_slice, 0, &src_dptr))
+                     ret = EFAULT;
+             if (bpf_ringbuf_reserve_dynptr(&ringbuf, size, 0, &dst_dptr))
+                     ret = EFAULT;
+             if (bpf_dynptr_copy(&dst_dptr, 0, &src_dptr, 0, size))
+                     ret = EFAULT;
+             bpf_ringbuf_submit_dynptr(&dst_dptr, 0);
+
+             written_count++;
+             written_size += size;
+
+             bpf_stream_free_elem(elem);
+
+             /* Probe and exit if no more space, probe for twice the typical size.*/
+             if (bpf_ringbuf_reserve_dynptr(&ringbuf, 2048, 0, &dst_dptr))
+                     cont = true;
+             bpf_ringbuf_discard_dynptr(&dst_dptr, 0);
+
+             if (ret || cont)
+                     break;
+     }
+
+     if (cont)
+             elem_batch = bpf_kptr_xchg(&v->batch, elem_batch);
+     if (elem_batch)
+             bpf_stream_free_elem_batch(elem_batch);
+end:
+     bpf_prog_stream_put(stream);
+
Quoted 8 lines

tools/bpf/bpftool/skeleton/stream.bpf.c line 96

Quoted 4 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +4 -0
+     return ret ?: (cont ? EAGAIN : 0);
+}
+
+char _license[] SEC("license") = "GPL";

Same note on the license, other programs used by bpftool have license string "Dual BSD/GPL".

Done, please take a look at the changes in v2.

author maintainer
#42 lore raw

RFC PATCH 13/13 selftests/bpf: Add tests for prog streams

In reply to Kumar Kartikeya Dwivedi

Add selftests to stress test the various facets of the stream API, memory allocation pattern, and ensuring dumping support is tested and functional.

---
 .../testing/selftests/bpf/prog_tests/stream.c |  57 +++++++
 .../testing/selftests/bpf/progs/dynptr_fail.c |  28 ++++
 tools/testing/selftests/bpf/progs/stream.c    | 150 ++++++++++++++++++
 .../selftests/bpf/progs/stream_bpftool.c      | 142 +++++++++++++++++
 .../testing/selftests/bpf/progs/stream_fail.c |  38 +++++
 5 files changed, 415 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stream.c
 create mode 100644 tools/testing/selftests/bpf/progs/stream.c
 create mode 100644 tools/testing/selftests/bpf/progs/stream_bpftool.c
 create mode 100644 tools/testing/selftests/bpf/progs/stream_fail.c
tools/testing/selftests/bpf/prog_tests/stream.c new file +57 -0
new file mode 100644
index 000000000000..b01cb18d8de5
@@ -0,0 +1,57 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include <sys/mman.h>
+
+#include "stream.skel.h"
+#include "stream_fail.skel.h"
+
+#include "stream_bpftool.skel.h"
+
+void test_stream_failure(void)
+{
+	RUN_TESTS(stream_fail);
+}
+
+void test_stream_success(void)
+{
+	RUN_TESTS(stream);
+	RUN_TESTS(stream_bpftool);
+	return;
+}
+
+static int process_sample(void *ctx, void *data, size_t len)
+{
+	fprintf(stderr, "%s", (char *)data);
+	return 0;
+}
+
+void test_stream_ringbuf_output(void)
+{
+	LIBBPF_OPTS(bpf_test_run_opts, opts);
+	struct ring_buffer *ringbuf;
+	struct stream_bpftool *skel;
+	int fd, ret;
+
+	skel = stream_bpftool__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "stream_bpftool_open_and_load"))
+		return;
+
+	fd = bpf_map__fd(skel->maps.ringbuf);
+
+	ringbuf = ring_buffer__new(fd, process_sample, NULL, NULL);
+	if (!ASSERT_OK_PTR(ringbuf, "ringbuf_new"))
+		goto end;
+
+	do {
+		skel->bss->written_count = skel->bss->written_size = 0;
+		ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.stream_bpftool_dump_prog_stream), &opts);
+		ASSERT_EQ(ring_buffer__consume_n(ringbuf, skel->bss->written_count), skel->bss->written_count, "consume");
+	} while (!ret && opts.retval == EAGAIN);
+
+	ASSERT_OK(ret, "ret");
+	ASSERT_EQ(opts.retval, 0, "retval");
+
+end:
+	stream_bpftool__destroy(skel);
+}
tools/testing/selftests/bpf/progs/dynptr_fail.c +28 -0
index 7c67797a5aac..545831b43fc8 100644
@@ -1734,3 +1734,31 @@ int test_dynptr_reg_type(void *ctx)
 	global_call_bpf_dynptr((const struct bpf_dynptr *)current);
 	return 0;
 }
+
+SEC("?syscall")
+__failure __msg("Expected an initialized dynptr as arg #2")
+int test_dynptr_source_release_btf(void *ctx)
+{
+	struct bpf_stream_elem_batch *elem_batch;
+	struct bpf_stream_elem *elem;
+	struct bpf_stream *stream;
+	struct bpf_dynptr dptr;
+	char buf[8];
+
+	stream = bpf_stream_get(BPF_STDERR, NULL);
+	if (!stream)
+		return 0;
+	elem_batch = bpf_stream_next_elem_batch(stream);
+	if (!elem_batch)
+		return 0;
+	elem = bpf_stream_next_elem(elem_batch);
+	if (!elem) {
+		bpf_stream_free_elem_batch(elem_batch);
+		return 0;
+	}
+	bpf_dynptr_from_mem_slice(&elem->mem_slice, 0, &dptr);
+	bpf_stream_free_elem(elem);
+
+	bpf_dynptr_read(buf, sizeof(buf), &dptr, 0, 0);
+	return 0;
+}
tools/testing/selftests/bpf/progs/stream.c new file +150 -0

150 lines not on this page · raw message

tools/testing/selftests/bpf/progs/stream_bpftool.c new file +142 -0

142 lines not on this page · raw message

tools/testing/selftests/bpf/progs/stream_fail.c new file +38 -0
new file mode 100644
index 000000000000..c44590b3ceda
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+struct map_value {
+	struct bpf_stream_elem_batch __kptr *batch;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, struct map_value);
+} arrmap SEC(".maps");
+
+
+SEC("?tc")
+__failure __msg("untrusted_ptr_bpf_stream_elem_batch()")
+int stream_kptr_ptr_untrusted(struct __sk_buff *ctx)
+{
+	struct bpf_stream_elem_batch *batch;
+	struct map_value *v;
+	int key = 0;
+
+	v = bpf_map_lookup_elem(&arrmap, &key);
+	if (!v)
+		return 0;
+	batch = v->batch;
+	if (!batch)
+		return 0;
+	v->batch = (void *)batch->node->next->next->next;
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
Signature · 1 line
2.47.1
maintainer
#43 lore raw

Re: [RFC PATCH bpf-next/net v1 13/13] selftests/bpf: Add tests for prog streams

Kumar Kartikeya Dwivedi <[email protected]> writes:

[...]

tools/testing/selftests/bpf/progs/stream_bpftool.c

Quoted 5 lines
tools/testing/selftests/bpf/progs/stream_bpftool.c new file +0 -0
new file mode 100644
index 000000000000..438c01a96efc

This file is almost identical to ./tools/bpf/bpftool/skeleton/stream.bpf.c except it also adds a function foo that exhausts may goto budget, as far as I can tell. What is the point of adding it?

In general, I don't think we run bpftool tests on the CI, so it would be good to test stream.bpf.c itself e.g. by creating a symlink to it in selftests and adding a corresponding prog_tests/<smth>.c as you do here.

Also, it would be good if some of the tests checked the content read from the stream. I think existing tests only check the size.

[...]