BPF Standard Streams
Subsystems (10)
Kumar Kartikeya Dwivedi
maintainercover BPF Standard Streams
NOTE: Temporarily based off of bpf-next/net due to Qdisc related fixes.
This set introduces a standard output interface with two streams, namely stdout and stderr, for BPF programs. The idea is that these streams will be written to by BPF programs and the kernel, and serve as standard interfaces for informing user space of any BPF runtime violations. Users can also utilize them for printing normal messages for debugging usage, as is the case with bpf_printk() and trace pipe interface.
BPF programs and the kernel can use these streams to output messages. User space can dump these messages using bpftool.
The stream interface itself is implemented using a lockless list, so that we can queue messages from any context. Every printk statement into the stream leads to memory allocation. Allocation itself relies on the bpf_mem_alloc() API, but in case it fails, we use try_alloc_pages() to construct a bespoke bump allocator to carve out elements. If this fails, we finally give up and drop the message.
Three scenarios (rqspinlock deadlocks, arena page faults, and cond_break timeouts) are covered for reporting errors to user space, the format looks as follows (with file:line info for BPF programs to pin point source of error). The output is as follows:
ERROR: AA or ABBA deadlock detected for bpf_res_spin_lock
Attempted lock = 0xff11000104a875e0
Total held locks = 1
Held lock[ 0] = 0xff11000104a875e0
CPU: 5 UID: 0 PID: 764 Comm: test_progs Call trace: bpf_prog_stderr_dump_stack+0x83/0xc0 bpf_res_spin_lock+0x8c/0xa0 bpf_prog_0f369a1fb6eb23d7_foo+0xe5/0x140: [stream_bpftool.c:62] bpf_prog_986cfd896eb75e8c_stream_bpftool_dump_prog_stream+0x7e/0x2e1: [stream_bpftool.c:93] bpf_prog_test_run_syscall+0x102/0x240 __sys_bpf+0xd68/0x2bf0 __x64_sys_bpf+0x1e/0x30 do_syscall_64+0x68/0x140 entry_SYSCALL_64_after_hwframe+0x76/0x7e
ERROR: Arena READ access at unmapped address 0xdeadbeef
CPU: 5 UID: 0 PID: 764 Comm: test_progs Call trace: bpf_prog_stderr_dump_stack+0x83/0xc0 ex_handler_bpf+0x4a/0xa0 fixup_exception+0xde/0x310 kernelmode_fixup_or_oops.constprop.0+0x2f/0x70 exc_page_fault+0xdd/0x1d0 asm_exc_page_fault+0x26/0x30 bpf_prog_0f369a1fb6eb23d7_foo+0x10c/0x140: [stream_bpftool.c:67] bpf_prog_986cfd896eb75e8c_stream_bpftool_dump_prog_stream+0x7e/0x2e1: [stream_bpftool.c:93] bpf_prog_test_run_syscall+0x102/0x240 __sys_bpf+0xd68/0x2bf0 __x64_sys_bpf+0x1e/0x30 do_syscall_64+0x68/0x140 entry_SYSCALL_64_after_hwframe+0x76/0x7e
ERROR: Arena WRITE access at unmapped address 0xfaceb00c
CPU: 5 UID: 0 PID: 764 Comm: test_progs Call trace: bpf_prog_stderr_dump_stack+0x83/0xc0 ex_handler_bpf+0x4a/0xa0 fixup_exception+0xde/0x310 kernelmode_fixup_or_oops.constprop.0+0x2f/0x70 exc_page_fault+0xdd/0x1d0 asm_exc_page_fault+0x26/0x30 bpf_prog_0f369a1fb6eb23d7_foo+0x111/0x140: [stream_bpftool.c:67] bpf_prog_986cfd896eb75e8c_stream_bpftool_dump_prog_stream+0x7e/0x2e1: [stream_bpftool.c:93] bpf_prog_test_run_syscall+0x102/0x240 __sys_bpf+0xd68/0x2bf0 __x64_sys_bpf+0x1e/0x30 do_syscall_64+0x68/0x140 entry_SYSCALL_64_after_hwframe+0x76/0x7e
ERROR: Timeout detected for may_goto instruction
CPU: 5 UID: 0 PID: 764 Comm: test_progs Call trace: bpf_prog_stderr_dump_stack+0x83/0xc0 bpf_check_timed_may_goto+0x4d/0xa0 arch_bpf_timed_may_goto+0x21/0x40 bpf_prog_0f369a1fb6eb23d7_foo+0x12f/0x140: [stream_bpftool.c:70] bpf_prog_986cfd896eb75e8c_stream_bpftool_dump_prog_stream+0x7e/0x2e1: [stream_bpftool.c:93] bpf_prog_test_run_syscall+0x102/0x240 __sys_bpf+0xd68/0x2bf0 __x64_sys_bpf+0x1e/0x30 do_syscall_64+0x68/0x140 entry_SYSCALL_64_after_hwframe+0x76/0x7e
There is a lot of minor things to address before this is mergeable, but the main logic and approach should be set in stone.
TODO (in order of importance)
-----------------------------
* Add an API for kernel writers to stderr streams to allow them to stage messages to a llist_head and then atomically splice in a bunch of messages into the stream to avoid interleaving from multiple threads. Each "message" consisisting of multiple printk statements then appears as a contiguous sequence to the user space. This will simply be a stage + commit API on top of an object pinning the llist_head that will be del_all'd and then add_batch'd to the stream->log.
* Add bpf_printk() macros for streams.
* Add kdoc comments for kfuncs.
* Add more corner case tests, and negative cases.
* Enforcing memory consumption limits on the log usage, so that in-flight log memory is bounded at runtime.
* Support bpftool filepath printing and JSON output.
Alexei Starovoitov (1):
locking/local_lock, mm: Replace localtry_ helpers with local_trylock_t
type
Kumar Kartikeya Dwivedi (12):
bpf: Tie dynptrs to referenced source objects
bpf: Compare dynptr_id in regsafe
selftests/bpf: Convert dynptr_fail to use vmlinux.h selftests/bpf: Add tests for dynptr source object interaction
bpf: Introduce bpf_dynptr_from_mem_slice
bpf: Introduce per-prog stdout/stderr streams
bpf: Add dump_stack() analogue to print to BPF stderr
bpf: Report may_goto timeout to BPF stderr
bpf: Report rqspinlock deadlocks/timeout to BPF stderr
bpf: Report arena faults to BPF stderr
bpftool: Add support for dumping streams selftests/bpf: Add tests for prog streams
arch/x86/net/bpf_jit_comp.c | 21 +- include/linux/bpf.h | 63 +- include/linux/local_lock.h | 58 +- include/linux/local_lock_internal.h | 207 +++--- kernel/bpf/Makefile | 2 +- kernel/bpf/arena.c | 14 + kernel/bpf/core.c | 29 +- kernel/bpf/helpers.c | 58 +- kernel/bpf/rqspinlock.c | 22 + kernel/bpf/stream.c | 595 ++++++++++++++++++ kernel/bpf/syscall.c | 2 +- kernel/bpf/verifier.c | 104 ++- mm/memcontrol.c | 39 +- tools/bpf/bpftool/Makefile | 2 +- tools/bpf/bpftool/prog.c | 71 ++- tools/bpf/bpftool/skeleton/stream.bpf.c | 96 +++ .../testing/selftests/bpf/prog_tests/dynptr.c | 2 + .../testing/selftests/bpf/prog_tests/stream.c | 57 ++ .../testing/selftests/bpf/progs/dynptr_fail.c | 43 +- .../selftests/bpf/progs/dynptr_fail_qdisc.c | 38 ++ tools/testing/selftests/bpf/progs/stream.c | 150 +++++ .../selftests/bpf/progs/stream_bpftool.c | 142 +++++ .../testing/selftests/bpf/progs/stream_fail.c | 38 ++ 23 files changed, 1597 insertions(+), 256 deletions(-) create mode 100644 kernel/bpf/stream.c create mode 100644 tools/bpf/bpftool/skeleton/stream.bpf.c create mode 100644 tools/testing/selftests/bpf/prog_tests/stream.c create mode 100644 tools/testing/selftests/bpf/progs/dynptr_fail_qdisc.c create mode 100644 tools/testing/selftests/bpf/progs/stream.c create mode 100644 tools/testing/selftests/bpf/progs/stream_bpftool.c create mode 100644 tools/testing/selftests/bpf/progs/stream_fail.c
base-commit: 93cb5375fb16693273144e7627e4fe27b811003a
Signature · 1 line
2.47.1
Kumar Kartikeya Dwivedi
maintainerRFC PATCH 1/13 bpf: Tie dynptrs to referenced source objects
Ensure that for dynptr constructors (MEM_UNINIT arg_type) taking a referenced object (ref_obj_id) as their memory source, we set the ref_obj_id of the dynptr appropriately as well. This allows us to tie the lifetime of the dynptr to its source and properly invalidate it when the source object is released.
For helpers, we don't have such cases yet as bpf_dynptr_from_mem is not permitted for anything other than PTR_TO_MAP_VALUE, but still pass meta->ref_obj_id as clone_ref_obj_id in case this is relaxed in future. Since they are ossified we know dynptr_from_mem is the only relevant helper and takes one memory argument, so we know the meta->ref_obj_id if non-zero will belong to it.
For kfuncs, make sure for constructors, only 1 ref_obj_id argument is seen, as more than one can be ambiguous in terms of ref_obj_id transfer. Since more kfuncs can be added with possibly multiple memory arguments, make sure meta->ref_obj_id reassignment won't cause incorrect lifetime analysis in the future using ref_obj_cnt logic. Set this ref_obj_id as the clone_ref_obj_id, so that it is transferred to the spilled_ptr stack slot register state.
Add support to unmark_stack_slots_dynptr to not descend to its child slices (using bool slice parameter) so that we don't have circular calls when invoking release_reference. With this logic in place, we may have the following object relationships:
+-- slice 1 (ref=1) source (ref=1) --> dynptr (ref=1) --|-- slice 2 (ref=1) +-- slice 3 (ref=1)
Destroying dynptr prunes the dynptr and all its children slices, but does not affect the source. Releasing the source will effectively prune the entire ownership tree. Dynptr clones with same ref=1 will be parallel in the ownership tree.
+-- orig dptr (ref=1) --> slice 1 (ref=1) source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1) +-- clone dptr (ref=1) --> slice 3 (ref=1)
In such a case, only child slices of the dynptr clone being destroyed are invalidated. Likewise, if the source object goes away, the whole tree ends up getting pruned.
--- kernel/bpf/verifier.c | 81 ++++++++++++++++++++++++++++--------------- 1 file changed, 54 insertions(+), 27 deletions(-)
kernel/bpf/verifier.c +54 -27
@@ -198,7 +198,7 @@ struct bpf_verifier_stack_elem {static int acquire_reference (struct bpf_verifier_env *env,int insn_idx);static int release_reference_nomark (struct bpf_verifier_state *state,int ref_obj_id); -static int release_reference (struct bpf_verifier_env *env,int ref_obj_id); +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id,bool objects);static void invalidate_non_owning_refs (struct bpf_verifier_env *env);static bool in_rbtree_lock_required_cb (struct bpf_verifier_env *env); static int ref_set_non_owning(struct bpf_verifier_env *env,
@@ -299,6 +299,7 @@ struct bpf_kfunc_call_arg_meta {const char *func_name;/* Out parameters */ u32 ref_obj_id; +u32 ref_obj_cnt;u8 release_regno;bool r0_rdonly;u32 ret_btf_id;
@@ -759,7 +760,7 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_mark_dynptr_stack_regs (env, &state->stack[spi].spilled_ptr, &state->stack[spi -1 ].spilled_ptr, type); -if (dynptr_type_refcounted (type)) { +if (dynptr_type_refcounted (type) || clone_ref_obj_id) {/* The id is used to track proper releasing */ int id;
@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat state->stack[spi -1 ].spilled_ptr.live |=REG_LIVE_WRITTEN ; } -static int unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +static int __unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_func_state *state, +int spi,bool slice) { -struct bpf_func_state *state =func (env, reg); -int spi, ref_obj_id, i; +u32 ref_obj_id; +int i; - spi =dynptr_get_spi (env, reg); -if (spi <0 ) -return spi; + ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; -if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type)) { +if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {invalidate_dynptr (env, state, spi);return 0 ; } - ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; -/* If the dynptr has a ref_obj_id, then we need to invalidate * two things: *
@@ -842,7 +840,8 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re*/ /* Invalidate any slices associated with this dynptr */ -WARN_ON_ONCE (release_reference (env, ref_obj_id)); +if (slice) +WARN_ON_ONCE (release_reference (env, ref_obj_id, false));/* Invalidate any dynptr clones */ for (i =1 ; i < state->allocated_stack /BPF_REG_SIZE ; i++) {
@@ -864,6 +863,18 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_rereturn 0 ; } +static int unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_reg_state *reg,bool slice) +{ +struct bpf_func_state *state =func (env, reg); +int spi; + + spi =dynptr_get_spi (env, reg); +if (spi <0 ) +return spi; + +return __unmark_stack_slots_dynptr (env, state, spi, slice); +} +static void __mark_reg_unknown (const struct bpf_verifier_env *env,struct bpf_reg_state *reg);
@@ -1075,7 +1086,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,struct bpf_reg_state *st = &slot->spilled_ptr;if (i ==0 ) -WARN_ON_ONCE (release_reference (env, st->ref_obj_id)); +WARN_ON_ONCE (release_reference (env, st->ref_obj_id, false));__mark_reg_not_init (env, st);
@@ -9749,7 +9760,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, true, meta);break ;case ARG_PTR_TO_DYNPTR : - err =process_dynptr_func (env, regno, insn_idx, arg_type,0 ); + err =process_dynptr_func (env, regno, insn_idx, arg_type, meta->ref_obj_id);if (err)return err;break ;
@@ -10220,12 +10231,12 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob* * This is the release function corresponding to acquire_reference(). Idempotent. */ -static int release_reference (struct bpf_verifier_env *env,int ref_obj_id) +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id,bool objects) {struct bpf_verifier_state *vstate = env->cur_state;struct bpf_func_state *state;struct bpf_reg_state *reg; -int err; +int err, spi; err =release_reference_nomark (vstate, ref_obj_id);if (err)
@@ -10236,6 +10247,19 @@ static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)mark_reg_invalid (env, reg); })); +if (!objects) +return 0 ; + +bpf_for_each_spilled_reg (spi, state, reg, (1 <<STACK_DYNPTR )) { +if (!reg) +continue ; +if (!reg->dynptr.first_slot || reg->ref_obj_id != ref_obj_id) +continue ; + err =__unmark_stack_slots_dynptr (env, state, spi, false); +if (err) +return err; + } +return 0 ; }
@@ -11357,7 +11381,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insnverbose (env,"verifier internal error: CONST_PTR_TO_DYNPTR cannot be released\n" );return -EFAULT ; } - err =unmark_stack_slots_dynptr (env, ®s[meta.release_regno]); + err =unmark_stack_slots_dynptr (env, ®s[meta.release_regno], true); }else if (func_id == BPF_FUNC_kptr_xchg && meta.ref_obj_id) {u32 ref_obj_id = meta.ref_obj_id;bool in_rcu =in_rcu_cs (env);
@@ -11379,7 +11403,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn })); } }else if (meta.ref_obj_id) { - err =release_reference (env, meta.ref_obj_id); + err =release_reference (env, meta.ref_obj_id, true); }else if (register_is_null (®s[meta.release_regno])) {/* meta.ref_obj_id can only be 0 if register that is meant to be * released is NULL, which must be > R0.
@@ -12974,6 +12998,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ meta->ref_obj_id = reg->ref_obj_id;if (is_kfunc_release (meta)) meta->release_regno = regno; + meta->ref_obj_cnt++; } ref_t =btf_type_skip_modifiers (btf, t->type, &ref_id);
@@ -13100,13 +13125,19 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_case KF_ARG_PTR_TO_DYNPTR : {enum bpf_arg_type dynptr_arg_type =ARG_PTR_TO_DYNPTR ; -int clone_ref_obj_id =0 ; +int clone_ref_obj_id = meta->ref_obj_id;if (reg->type ==CONST_PTR_TO_DYNPTR ) dynptr_arg_type |=MEM_RDONLY ; -if (is_kfunc_arg_uninit (btf, &args[i])) +if (is_kfunc_arg_uninit (btf, &args[i])) { dynptr_arg_type |=MEM_UNINIT ; +/* It's confusing if dynptr constructor takes multiple referenced arguments. */ +if (meta->ref_obj_cnt >1 ) { +verbose (env,"verifier internal error: multiple referenced arguments\n" ); +return -EFAULT ; + } + }if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) { dynptr_arg_type |=DYNPTR_TYPE_SKB ;
@@ -13582,7 +13613,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,* PTR_TO_BTF_ID in bpf_kfunc_arg_meta, do the release now. */ if (meta.release_regno) { - err =release_reference (env, regs[meta.release_regno].ref_obj_id); + err =release_reference (env, regs[meta.release_regno].ref_obj_id, true);if (err) {verbose (env,"kfunc %s#%d reference has not been acquired before\n" , func_name, meta.func_id);
@@ -13603,7 +13634,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return err; } - err =release_reference (env, release_ref_obj_id); + err =release_reference (env, release_ref_obj_id, true);if (err) {verbose (env,"kfunc %s#%d reference has not been acquired before\n" , func_name, meta.func_id);
@@ -13803,11 +13834,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return -EFAULT ; } regs[BPF_REG_0 ].dynptr_id = meta.initialized_dynptr.id; - -/* we don't need to set BPF_REG_0's ref obj id -* because packet slices are not refcounted (see -* dynptr_type_refcounted) -*/ + regs[BPF_REG_0 ].ref_obj_id = meta.initialized_dynptr.ref_obj_id; }else {verbose (env,"kernel function %s unhandled dynamic return type\n" , meta.func_name);
Signature · 1 line
2.47.1
Re: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects
Quoted 110 lines · Kumar Kartikeya Dwivedi
Ensure that for dynptr constructors (
MEM_UNINITarg_type) taking a referenced object (ref_obj_id) as their memory source, we set theref_obj_idof the dynptr appropriately as well. This allows us to tie the lifetime of the dynptr to its source and properly invalidate it when the source object is released.For helpers, we don't have such cases yet as
bpf_dynptr_from_memis not permitted for anything other thanPTR_TO_MAP_VALUE, but still passmeta->ref_obj_idasclone_ref_obj_idin case this is relaxed in future. Since they are ossified we knowdynptr_from_memis the only relevant helper and takes one memory argument, so we know themeta->ref_obj_idif non-zero will belong to it.For kfuncs, make sure for constructors, only 1
ref_obj_idargument is seen, as more than one can be ambiguous in terms ofref_obj_idtransfer. Since more kfuncs can be added with possibly multiple memory arguments, make suremeta->ref_obj_idreassignment won't cause incorrect lifetime analysis in the future usingref_obj_cntlogic. Set thisref_obj_idas theclone_ref_obj_id, so that it is transferred to thespilled_ptrstack slot register state.Add support to
unmark_stack_slots_dynptrto not descend to its child slices (using bool slice parameter) so that we don't have circular calls when invokingrelease_reference. With this logic in place, we may have the following object relationships:+-- slice 1 (ref=1) source (ref=1) --> dynptr (ref=1) --|-- slice 2 (ref=1) +-- slice 3 (ref=1)Destroying dynptr prunes the dynptr and all its children slices, but does not affect the source. Releasing the source will effectively prune the entire ownership tree. Dynptr clones with same ref=1 will be parallel in the ownership tree.
+-- orig dptr (ref=1) --> slice 1 (ref=1) source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1) +-- clone dptr (ref=1) --> slice 3 (ref=1)In such a case, only child slices of the dynptr clone being destroyed are invalidated. Likewise, if the source object goes away, the whole tree ends up getting pruned.
Cc Amery Hung <[email protected]>Fixes 81bb1c2c3e8e ("bpf: net_sched: Add basic bpf qdisc kfuncs")Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- kernel/bpf/verifier.c | 81 ++++++++++++++++++++++++++++--------------- 1 file changed, 54 insertions(+), 27 deletions(-)kernel/bpf/verifier.c +9
-11@@ -198,7 +198,7 @@ struct bpf_verifier_stack_elem {static int acquire_reference (struct bpf_verifier_env *env,int insn_idx);static int release_reference_nomark (struct bpf_verifier_state *state,int ref_obj_id); -static int release_reference (struct bpf_verifier_env *env,int ref_obj_id); +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id,bool objects);static void invalidate_non_owning_refs (struct bpf_verifier_env *env);static bool in_rbtree_lock_required_cb (struct bpf_verifier_env *env); static int ref_set_non_owning(struct bpf_verifier_env *env,@@ -299,6 +299,7 @@ struct bpf_kfunc_call_arg_meta {const char *func_name;/* Out parameters */ u32 ref_obj_id; +u32 ref_obj_cnt;u8 release_regno;bool r0_rdonly;u32 ret_btf_id;@@ -759,7 +760,7 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_mark_dynptr_stack_regs (env, &state->stack[spi].spilled_ptr, &state->stack[spi -1 ].spilled_ptr, type); -if (dynptr_type_refcounted (type)) { +if (dynptr_type_refcounted (type) || clone_ref_obj_id) {/* The id is used to track proper releasing */ int id;@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat state->stack[spi -1 ].spilled_ptr.live |=REG_LIVE_WRITTEN ; } -static int unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +static int __unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_func_state *state, +int spi,bool slice) { -struct bpf_func_state *state =func (env, reg); -int spi, ref_obj_id, i; +u32 ref_obj_id; +int i; - spi =dynptr_get_spi (env, reg); -if (spi <0 ) -return spi; + ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; -if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type)) { +if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {invalidate_dynptr (env, state, spi);return 0 ; } - ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; -/* If the dynptr has a ref_obj_id, then we need to invalidate * two things: *
kernel/bpf/verifier.c static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re · line 844
Quoted 6 lines
kernel/bpf/verifier.c +2
-1@@ -842,7 +840,8 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re*/ /* Invalidate any slices associated with this dynptr */ -WARN_ON_ONCE (release_reference (env, ref_obj_id)); +if (slice) +WARN_ON_ONCE (release_reference (env, ref_obj_id, false));
I think the slice argument might not be needed if we do the following instead:
if (dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type))
WARN_ON_ONCE(release_reference(env, ref_obj_id, false));When bpf_kfree_skb():
-> release_reference(..., object = true)
-> __unmark_stack_slots_dynptr()
-> not calling into release_reference() sincedynptr_type_refcounted() returns false
When bpf_ringbuf_submit():
-> unmark_stack_slots_dynptr()
-> release_reference(..., object = false)
-> not calling __unmark_stack_slots_dynptr() since object == falseAm I missing anything?
Quoted 94 lines
kernel/bpf/verifier.c +31
-6/* Invalidate any dynptr clones */ for (i =1 ; i < state->allocated_stack /BPF_REG_SIZE ; i++) {@@ -864,6 +863,18 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_rereturn 0 ; } +static int unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_reg_state *reg,bool slice) +{ +struct bpf_func_state *state =func (env, reg); +int spi; + + spi =dynptr_get_spi (env, reg); +if (spi <0 ) +return spi; + +return __unmark_stack_slots_dynptr (env, state, spi, slice); +} +static void __mark_reg_unknown (const struct bpf_verifier_env *env,struct bpf_reg_state *reg);@@ -1075,7 +1086,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,struct bpf_reg_state *st = &slot->spilled_ptr;if (i ==0 ) -WARN_ON_ONCE (release_reference (env, st->ref_obj_id)); +WARN_ON_ONCE (release_reference (env, st->ref_obj_id, false));__mark_reg_not_init (env, st);@@ -9749,7 +9760,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, true, meta);break ;case ARG_PTR_TO_DYNPTR : - err =process_dynptr_func (env, regno, insn_idx, arg_type,0 ); + err =process_dynptr_func (env, regno, insn_idx, arg_type, meta->ref_obj_id);if (err)return err;break ;@@ -10220,12 +10231,12 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob* * This is the release function corresponding to acquire_reference(). Idempotent. */ -static int release_reference (struct bpf_verifier_env *env,int ref_obj_id) +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id,bool objects) {struct bpf_verifier_state *vstate = env->cur_state;struct bpf_func_state *state;struct bpf_reg_state *reg; -int err; +int err, spi; err =release_reference_nomark (vstate, ref_obj_id);if (err)@@ -10236,6 +10247,19 @@ static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)mark_reg_invalid (env, reg); })); +if (!objects) +return 0 ; + +bpf_for_each_spilled_reg (spi, state, reg, (1 <<STACK_DYNPTR )) { +if (!reg) +continue ; +if (!reg->dynptr.first_slot || reg->ref_obj_id != ref_obj_id) +continue ; + err =__unmark_stack_slots_dynptr (env, state, spi, false); +if (err) +return err; + } +return 0 ; }@@ -11357,7 +11381,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insnverbose (env,"verifier internal error: CONST_PTR_TO_DYNPTR cannot be released\n" );return -EFAULT ; } - err =unmark_stack_slots_dynptr (env, ®s[meta.release_regno]); + err =unmark_stack_slots_dynptr (env, ®s[meta.release_regno], true); }else if (func_id == BPF_FUNC_kptr_xchg && meta.ref_obj_id) {u32 ref_obj_id = meta.ref_obj_id;bool in_rcu =in_rcu_cs (env);@@ -11379,7 +11403,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn })); } }else if (meta.ref_obj_id) { - err =release_reference (env, meta.ref_obj_id); + err =release_reference (env, meta.ref_obj_id, true); }else if (register_is_null (®s[meta.release_regno])) {/* meta.ref_obj_id can only be 0 if register that is meant to be * released is NULL, which must be > R0.
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 13001
Quoted 4 lines
kernel/bpf/verifier.c +1
-0@@ -12974,6 +12998,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ meta->ref_obj_id = reg->ref_obj_id;if (is_kfunc_release (meta)) meta->release_regno = regno; + meta->ref_obj_cnt++;
Does it make sense to convert other "more than one arg with ref_obj_id" checks to using ref_obj_cnt to make it more consistent?
Thanks for fixing the bug!
Amery
Quoted 59 lines
kernel/bpf/verifier.c +11
-10} ref_t =btf_type_skip_modifiers (btf, t->type, &ref_id);@@ -13100,13 +13125,19 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_case KF_ARG_PTR_TO_DYNPTR : {enum bpf_arg_type dynptr_arg_type =ARG_PTR_TO_DYNPTR ; -int clone_ref_obj_id =0 ; +int clone_ref_obj_id = meta->ref_obj_id;if (reg->type ==CONST_PTR_TO_DYNPTR ) dynptr_arg_type |=MEM_RDONLY ; -if (is_kfunc_arg_uninit (btf, &args[i])) +if (is_kfunc_arg_uninit (btf, &args[i])) { dynptr_arg_type |=MEM_UNINIT ; +/* It's confusing if dynptr constructor takes multiple referenced arguments. */ +if (meta->ref_obj_cnt >1 ) { +verbose (env,"verifier internal error: multiple referenced arguments\n" ); +return -EFAULT ; + } + }if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) { dynptr_arg_type |=DYNPTR_TYPE_SKB ;@@ -13582,7 +13613,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,* PTR_TO_BTF_ID in bpf_kfunc_arg_meta, do the release now. */ if (meta.release_regno) { - err =release_reference (env, regs[meta.release_regno].ref_obj_id); + err =release_reference (env, regs[meta.release_regno].ref_obj_id, true);if (err) {verbose (env,"kfunc %s#%d reference has not been acquired before\n" , func_name, meta.func_id);@@ -13603,7 +13634,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return err; } - err =release_reference (env, release_ref_obj_id); + err =release_reference (env, release_ref_obj_id, true);if (err) {verbose (env,"kfunc %s#%d reference has not been acquired before\n" , func_name, meta.func_id);@@ -13803,11 +13834,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return -EFAULT ; } regs[BPF_REG_0 ].dynptr_id = meta.initialized_dynptr.id; - -/* we don't need to set BPF_REG_0's ref obj id -* because packet slices are not refcounted (see -* dynptr_type_refcounted) -*/ + regs[BPF_REG_0 ].ref_obj_id = meta.initialized_dynptr.ref_obj_id; }else {verbose (env,"kernel function %s unhandled dynamic return type\n" , meta.func_name); --2.47.1
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects
Quoted 128 lines · Amery Hung
Quoted 110 lines · Kumar Kartikeya Dwivedi
Ensure that for dynptr constructors (
MEM_UNINITarg_type) taking a referenced object (ref_obj_id) as their memory source, we set theref_obj_idof the dynptr appropriately as well. This allows us to tie the lifetime of the dynptr to its source and properly invalidate it when the source object is released.For helpers, we don't have such cases yet as
bpf_dynptr_from_memis not permitted for anything other thanPTR_TO_MAP_VALUE, but still passmeta->ref_obj_idasclone_ref_obj_idin case this is relaxed in future. Since they are ossified we knowdynptr_from_memis the only relevant helper and takes one memory argument, so we know themeta->ref_obj_idif non-zero will belong to it.For kfuncs, make sure for constructors, only 1
ref_obj_idargument is seen, as more than one can be ambiguous in terms ofref_obj_idtransfer. Since more kfuncs can be added with possibly multiple memory arguments, make suremeta->ref_obj_idreassignment won't cause incorrect lifetime analysis in the future usingref_obj_cntlogic. Set thisref_obj_idas theclone_ref_obj_id, so that it is transferred to thespilled_ptrstack slot register state.Add support to
unmark_stack_slots_dynptrto not descend to its child slices (using bool slice parameter) so that we don't have circular calls when invokingrelease_reference. With this logic in place, we may have the following object relationships:+-- slice 1 (ref=1) source (ref=1) --> dynptr (ref=1) --|-- slice 2 (ref=1) +-- slice 3 (ref=1)Destroying dynptr prunes the dynptr and all its children slices, but does not affect the source. Releasing the source will effectively prune the entire ownership tree. Dynptr clones with same ref=1 will be parallel in the ownership tree.
+-- orig dptr (ref=1) --> slice 1 (ref=1) source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1) +-- clone dptr (ref=1) --> slice 3 (ref=1)In such a case, only child slices of the dynptr clone being destroyed are invalidated. Likewise, if the source object goes away, the whole tree ends up getting pruned.
Cc Amery Hung <[email protected]>Fixes 81bb1c2c3e8e ("bpf: net_sched: Add basic bpf qdisc kfuncs")Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- kernel/bpf/verifier.c | 81 ++++++++++++++++++++++++++++--------------- 1 file changed, 54 insertions(+), 27 deletions(-)kernel/bpf/verifier.c +9
-11@@ -198,7 +198,7 @@ struct bpf_verifier_stack_elem {static int acquire_reference (struct bpf_verifier_env *env,int insn_idx);static int release_reference_nomark (struct bpf_verifier_state *state,int ref_obj_id); -static int release_reference (struct bpf_verifier_env *env,int ref_obj_id); +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id,bool objects);static void invalidate_non_owning_refs (struct bpf_verifier_env *env);static bool in_rbtree_lock_required_cb (struct bpf_verifier_env *env); static int ref_set_non_owning(struct bpf_verifier_env *env,@@ -299,6 +299,7 @@ struct bpf_kfunc_call_arg_meta {const char *func_name;/* Out parameters */ u32 ref_obj_id; +u32 ref_obj_cnt;u8 release_regno;bool r0_rdonly;u32 ret_btf_id;@@ -759,7 +760,7 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_mark_dynptr_stack_regs (env, &state->stack[spi].spilled_ptr, &state->stack[spi -1 ].spilled_ptr, type); -if (dynptr_type_refcounted (type)) { +if (dynptr_type_refcounted (type) || clone_ref_obj_id) {/* The id is used to track proper releasing */ int id;@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat state->stack[spi -1 ].spilled_ptr.live |=REG_LIVE_WRITTEN ; } -static int unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +static int __unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_func_state *state, +int spi,bool slice) { -struct bpf_func_state *state =func (env, reg); -int spi, ref_obj_id, i; +u32 ref_obj_id; +int i; - spi =dynptr_get_spi (env, reg); -if (spi <0 ) -return spi; + ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; -if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type)) { +if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {invalidate_dynptr (env, state, spi);return 0 ; } - ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; -/* If the dynptr has a ref_obj_id, then we need to invalidate * two things: * kernel/bpf/verifier.c static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re · line 844
Quoted 6 lines
kernel/bpf/verifier.c +2
-1@@ -842,7 +840,8 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re*/ /* Invalidate any slices associated with this dynptr */ -WARN_ON_ONCE (release_reference (env, ref_obj_id)); +if (slice) +WARN_ON_ONCE (release_reference (env, ref_obj_id, false));I think the slice argument might not be needed if we do the following instead:
if (dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) WARN_ON_ONCE(release_reference(env, ref_obj_id, false));When
bpf_kfree_skb():-> release_reference(..., object = true) -> __unmark_stack_slots_dynptr() -> not calling into release_reference() since
Quoted 8 lines
dynptr_type_refcounted()returns falseWhen
bpf_ringbuf_submit():-> unmark_stack_slots_dynptr() -> release_reference(..., object = false) -> not calling __unmark_stack_slots_dynptr() since object == falseAm I missing anything?
That is true, I will probably try to rework it the way Andrii suggested.
Quoted 94 lines
Quoted 94 lines
kernel/bpf/verifier.c +31
-6/* Invalidate any dynptr clones */ for (i =1 ; i < state->allocated_stack /BPF_REG_SIZE ; i++) {@@ -864,6 +863,18 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_rereturn 0 ; } +static int unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_reg_state *reg,bool slice) +{ +struct bpf_func_state *state =func (env, reg); +int spi; + + spi =dynptr_get_spi (env, reg); +if (spi <0 ) +return spi; + +return __unmark_stack_slots_dynptr (env, state, spi, slice); +} +static void __mark_reg_unknown (const struct bpf_verifier_env *env,struct bpf_reg_state *reg);@@ -1075,7 +1086,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,struct bpf_reg_state *st = &slot->spilled_ptr;if (i ==0 ) -WARN_ON_ONCE (release_reference (env, st->ref_obj_id)); +WARN_ON_ONCE (release_reference (env, st->ref_obj_id, false));__mark_reg_not_init (env, st);@@ -9749,7 +9760,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, true, meta);break ;case ARG_PTR_TO_DYNPTR : - err =process_dynptr_func (env, regno, insn_idx, arg_type,0 ); + err =process_dynptr_func (env, regno, insn_idx, arg_type, meta->ref_obj_id);if (err)return err;break ;@@ -10220,12 +10231,12 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob* * This is the release function corresponding to acquire_reference(). Idempotent. */ -static int release_reference (struct bpf_verifier_env *env,int ref_obj_id) +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id,bool objects) {struct bpf_verifier_state *vstate = env->cur_state;struct bpf_func_state *state;struct bpf_reg_state *reg; -int err; +int err, spi; err =release_reference_nomark (vstate, ref_obj_id);if (err)@@ -10236,6 +10247,19 @@ static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)mark_reg_invalid (env, reg); })); +if (!objects) +return 0 ; + +bpf_for_each_spilled_reg (spi, state, reg, (1 <<STACK_DYNPTR )) { +if (!reg) +continue ; +if (!reg->dynptr.first_slot || reg->ref_obj_id != ref_obj_id) +continue ; + err =__unmark_stack_slots_dynptr (env, state, spi, false); +if (err) +return err; + } +return 0 ; }@@ -11357,7 +11381,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insnverbose (env,"verifier internal error: CONST_PTR_TO_DYNPTR cannot be released\n" );return -EFAULT ; } - err =unmark_stack_slots_dynptr (env, ®s[meta.release_regno]); + err =unmark_stack_slots_dynptr (env, ®s[meta.release_regno], true); }else if (func_id == BPF_FUNC_kptr_xchg && meta.ref_obj_id) {u32 ref_obj_id = meta.ref_obj_id;bool in_rcu =in_rcu_cs (env);@@ -11379,7 +11403,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn })); } }else if (meta.ref_obj_id) { - err =release_reference (env, meta.ref_obj_id); + err =release_reference (env, meta.ref_obj_id, true); }else if (register_is_null (®s[meta.release_regno])) {/* meta.ref_obj_id can only be 0 if register that is meant to be * released is NULL, which must be > R0.
Quoted 8 lines
kernel/bpf/verifier.c static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ · line 13001
Quoted 5 lines
kernel/bpf/verifier.c +1
-0@@ -12974,6 +12998,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ meta->ref_obj_id = reg->ref_obj_id;if (is_kfunc_release (meta)) meta->release_regno = regno; + meta->ref_obj_cnt++;Does it make sense to convert other "more than one arg with
ref_obj_id" checks to usingref_obj_cntto make it more consistent?
Ack, will change.
Quoted 63 lines
Thanks for fixing the bug!
AmeryQuoted 59 lines
kernel/bpf/verifier.c +11
-10} ref_t =btf_type_skip_modifiers (btf, t->type, &ref_id);@@ -13100,13 +13125,19 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_case KF_ARG_PTR_TO_DYNPTR : {enum bpf_arg_type dynptr_arg_type =ARG_PTR_TO_DYNPTR ; -int clone_ref_obj_id =0 ; +int clone_ref_obj_id = meta->ref_obj_id;if (reg->type ==CONST_PTR_TO_DYNPTR ) dynptr_arg_type |=MEM_RDONLY ; -if (is_kfunc_arg_uninit (btf, &args[i])) +if (is_kfunc_arg_uninit (btf, &args[i])) { dynptr_arg_type |=MEM_UNINIT ; +/* It's confusing if dynptr constructor takes multiple referenced arguments. */ +if (meta->ref_obj_cnt >1 ) { +verbose (env,"verifier internal error: multiple referenced arguments\n" ); +return -EFAULT ; + } + }if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) { dynptr_arg_type |=DYNPTR_TYPE_SKB ;@@ -13582,7 +13613,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,* PTR_TO_BTF_ID in bpf_kfunc_arg_meta, do the release now. */ if (meta.release_regno) { - err =release_reference (env, regs[meta.release_regno].ref_obj_id); + err =release_reference (env, regs[meta.release_regno].ref_obj_id, true);if (err) {verbose (env,"kfunc %s#%d reference has not been acquired before\n" , func_name, meta.func_id);@@ -13603,7 +13634,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return err; } - err =release_reference (env, release_ref_obj_id); + err =release_reference (env, release_ref_obj_id, true);if (err) {verbose (env,"kfunc %s#%d reference has not been acquired before\n" , func_name, meta.func_id);@@ -13803,11 +13834,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,return -EFAULT ; } regs[BPF_REG_0 ].dynptr_id = meta.initialized_dynptr.id; - -/* we don't need to set BPF_REG_0's ref obj id -* because packet slices are not refcounted (see -* dynptr_type_refcounted) -*/ + regs[BPF_REG_0 ].ref_obj_id = meta.initialized_dynptr.ref_obj_id; }else {verbose (env,"kernel function %s unhandled dynamic return type\n" , meta.func_name); --2.47.1
Eduard Zingerman
maintainerRe: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects
In reply to Kumar Kartikeya DwivediKumar Kartikeya Dwivedi <[email protected]> writes:
[...]
Quoted 10 lines
kernel/bpf/verifier.c +2
-3@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat state->stack[spi -1 ].spilled_ptr.live |=REG_LIVE_WRITTEN ; } -static int unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +static int __unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_func_state *state, +int spi,bool slice) { -struct bpf_func_state *state =func (env, reg); -int spi, ref_obj_id, i;
kernel/bpf/verifier.c static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat · line 830
Quoted 10 lines
kernel/bpf/verifier.c +4
-4+u32 ref_obj_id; +int i; - spi =dynptr_get_spi (env, reg); -if (spi <0 ) -return spi; + ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; -if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type)) { +if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {
If dynptr_type_refcounted is true, does this mean that ref_obj_id is set? If it does, the check could be simplified to just if (!ref_obj_id).
Quoted 9 lines
kernel/bpf/verifier.c +0
-2invalidate_dynptr (env, state, spi);return 0 ; } - ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; -/* If the dynptr has a ref_obj_id, then we need to invalidate * two things: *
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects
Quoted 23 lines · Eduard Zingerman
Kumar Kartikeya Dwivedi <[email protected]> writes:
[...]
Quoted 18 lines
kernel/bpf/verifier.c +5
-6@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat state->stack[spi -1 ].spilled_ptr.live |=REG_LIVE_WRITTEN ; } -static int unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +static int __unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_func_state *state, +int spi,bool slice) { -struct bpf_func_state *state =func (env, reg); -int spi, ref_obj_id, i; +u32 ref_obj_id; +int i; - spi =dynptr_get_spi (env, reg); -if (spi <0 ) -return spi; + ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
Quoted 5 lines
kernel/bpf/verifier.c static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat · line 830
Quoted 2 lines
kernel/bpf/verifier.c +1
-1-if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type)) { +if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {If
dynptr_type_refcountedis true, does this mean thatref_obj_idis set? If it does, the check could be simplified to justif (!ref_obj_id).
Yes, I will change in non-RFC v1.
Quoted 12 lines
Quoted 10 lines
kernel/bpf/verifier.c +0
-2invalidate_dynptr (env, state, spi);return 0 ; } - ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; -/* If the dynptr has a ref_obj_id, then we need to invalidate * two things: * [...]
Re: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects
In reply to Kumar Kartikeya DwivediQuoted 72 lines · Kumar Kartikeya Dwivedi
Ensure that for dynptr constructors (
MEM_UNINITarg_type) taking a referenced object (ref_obj_id) as their memory source, we set theref_obj_idof the dynptr appropriately as well. This allows us to tie the lifetime of the dynptr to its source and properly invalidate it when the source object is released.For helpers, we don't have such cases yet as
bpf_dynptr_from_memis not permitted for anything other thanPTR_TO_MAP_VALUE, but still passmeta->ref_obj_idasclone_ref_obj_idin case this is relaxed in future. Since they are ossified we knowdynptr_from_memis the only relevant helper and takes one memory argument, so we know themeta->ref_obj_idif non-zero will belong to it.For kfuncs, make sure for constructors, only 1
ref_obj_idargument is seen, as more than one can be ambiguous in terms ofref_obj_idtransfer. Since more kfuncs can be added with possibly multiple memory arguments, make suremeta->ref_obj_idreassignment won't cause incorrect lifetime analysis in the future usingref_obj_cntlogic. Set thisref_obj_idas theclone_ref_obj_id, so that it is transferred to thespilled_ptrstack slot register state.Add support to
unmark_stack_slots_dynptrto not descend to its child slices (using bool slice parameter) so that we don't have circular calls when invokingrelease_reference. With this logic in place, we may have the following object relationships:+-- slice 1 (ref=1) source (ref=1) --> dynptr (ref=1) --|-- slice 2 (ref=1) +-- slice 3 (ref=1)Destroying dynptr prunes the dynptr and all its children slices, but does not affect the source. Releasing the source will effectively prune the entire ownership tree. Dynptr clones with same ref=1 will be parallel in the ownership tree.
+-- orig dptr (ref=1) --> slice 1 (ref=1) source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1) +-- clone dptr (ref=1) --> slice 3 (ref=1)In such a case, only child slices of the dynptr clone being destroyed are invalidated. Likewise, if the source object goes away, the whole tree ends up getting pruned.
Cc Amery Hung <[email protected]>Fixes 81bb1c2c3e8e ("bpf: net_sched: Add basic bpf qdisc kfuncs")Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- kernel/bpf/verifier.c | 81 ++++++++++++++++++++++++++++--------------- 1 file changed, 54 insertions(+), 27 deletions(-)kernel/bpf/verifier.c +2
-1@@ -198,7 +198,7 @@ struct bpf_verifier_stack_elem {static int acquire_reference (struct bpf_verifier_env *env,int insn_idx);static int release_reference_nomark (struct bpf_verifier_state *state,int ref_obj_id); -static int release_reference (struct bpf_verifier_env *env,int ref_obj_id); +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id,bool objects);static void invalidate_non_owning_refs (struct bpf_verifier_env *env);static bool in_rbtree_lock_required_cb (struct bpf_verifier_env *env); static int ref_set_non_owning(struct bpf_verifier_env *env,@@ -299,6 +299,7 @@ struct bpf_kfunc_call_arg_meta {const char *func_name;/* Out parameters */ u32 ref_obj_id; +u32 ref_obj_cnt;u8 release_regno;bool r0_rdonly;u32 ret_btf_id;
kernel/bpf/verifier.c static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_ · line 763
Quoted 5 lines
kernel/bpf/verifier.c +1
-1@@ -759,7 +760,7 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_mark_dynptr_stack_regs (env, &state->stack[spi].spilled_ptr, &state->stack[spi -1 ].spilled_ptr, type); -if (dynptr_type_refcounted (type)) { +if (dynptr_type_refcounted (type) || clone_ref_obj_id) {
it would be a bug to have both conditions true, right? Should we express this a bit more explicitly?
maybe something like
int id = 0;
if (dynptr_type_refcounted(type)) {
WARN_ON_ONCE(clone_ref_obj_id);
id = acquire_reference(env, insn_idx);
}
if (clone_ref_obj_id)
id = clone_ref_obj_id;state->stack[spi].spilled_ptr.ref_obj_id = id; state->stack[spi - 1].spilled_ptr.ref_obj_id = id; state->stack[spi].spilled_ptr.live |= REG_LIVE_WRITTEN; state->stack[spi - 1].spilled_ptr.live |= REG_LIVE_WRITTEN;
?
Quoted 13 lines
kernel/bpf/verifier.c +2
-3/* The id is used to track proper releasing */ int id;@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat state->stack[spi -1 ].spilled_ptr.live |=REG_LIVE_WRITTEN ; } -static int unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +static int __unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_func_state *state, +int spi,bool slice) { -struct bpf_func_state *state =func (env, reg); -int spi, ref_obj_id, i;
kernel/bpf/verifier.c static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat · line 830
Quoted 10 lines
kernel/bpf/verifier.c +4
-4+u32 ref_obj_id; +int i; - spi =dynptr_get_spi (env, reg); -if (spi <0 ) -return spi; + ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; -if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type)) { +if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {
Eduard already pointed out that !ref_obj_id condition is enough now
Quoted 10 lines
kernel/bpf/verifier.c +0
-2invalidate_dynptr (env, state, spi);return 0 ; } - ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; -/* If the dynptr has a ref_obj_id, then we need to invalidate * two things: *
kernel/bpf/verifier.c static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re · line 844
Quoted 6 lines
kernel/bpf/verifier.c +2
-1@@ -842,7 +840,8 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re*/ /* Invalidate any slices associated with this dynptr */ -WARN_ON_ONCE (release_reference (env, ref_obj_id)); +if (slice) +WARN_ON_ONCE (release_reference (env, ref_obj_id, false));
tbh, this bool flag to release_reference to prevent cycles feels like a hack and that we are not solving this properly...
also, changes in this patch still won't properly support the case where we create dynptr -> slice -> dynptr -> slice -> ... chains, right? Any dynptr in the chain will cause "release" of any other dynptr, including parents, right?
Quoted 3 lines
kernel/bpf/verifier.c +0
-0/* Invalidate any dynptr clones */ for (i =1 ; i < state->allocated_stack /BPF_REG_SIZE ; i++) {
[...]
kernel/bpf/verifier.c static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob · line 10234
Quoted 6 lines
kernel/bpf/verifier.c +1
-1@@ -10220,12 +10231,12 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob* * This is the release function corresponding to acquire_reference(). Idempotent. */ -static int release_reference (struct bpf_verifier_env *env,int ref_obj_id) +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id,bool objects)
hm, "objects" really doesn't describe anything...
Quoted 10 lines
kernel/bpf/verifier.c +1
-1{struct bpf_verifier_state *vstate = env->cur_state;struct bpf_func_state *state;struct bpf_reg_state *reg; -int err; +int err, spi; err =release_reference_nomark (vstate, ref_obj_id);if (err)
kernel/bpf/verifier.c static int release_reference(struct bpf_verifier_env *env, int ref_obj_id) · line 10251
Quoted 5 lines
kernel/bpf/verifier.c +2
-0@@ -10236,6 +10247,19 @@ static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)mark_reg_invalid (env, reg); })); +if (!objects) +return 0 ;
instead of a bool flag into release_reference, shouldn't the below logic be just a separate function?
Quoted 14 lines
kernel/bpf/verifier.c +11
-0+ +bpf_for_each_spilled_reg (spi, state, reg, (1 <<STACK_DYNPTR )) { +if (!reg) +continue ; +if (!reg->dynptr.first_slot || reg->ref_obj_id != ref_obj_id) +continue ; + err =__unmark_stack_slots_dynptr (env, state, spi, false); +if (err) +return err; + } +return 0 ; }
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects
Quoted 94 lines · Andrii Nakryiko
Quoted 72 lines · Kumar Kartikeya Dwivedi
Ensure that for dynptr constructors (
MEM_UNINITarg_type) taking a referenced object (ref_obj_id) as their memory source, we set theref_obj_idof the dynptr appropriately as well. This allows us to tie the lifetime of the dynptr to its source and properly invalidate it when the source object is released.For helpers, we don't have such cases yet as
bpf_dynptr_from_memis not permitted for anything other thanPTR_TO_MAP_VALUE, but still passmeta->ref_obj_idasclone_ref_obj_idin case this is relaxed in future. Since they are ossified we knowdynptr_from_memis the only relevant helper and takes one memory argument, so we know themeta->ref_obj_idif non-zero will belong to it.For kfuncs, make sure for constructors, only 1
ref_obj_idargument is seen, as more than one can be ambiguous in terms ofref_obj_idtransfer. Since more kfuncs can be added with possibly multiple memory arguments, make suremeta->ref_obj_idreassignment won't cause incorrect lifetime analysis in the future usingref_obj_cntlogic. Set thisref_obj_idas theclone_ref_obj_id, so that it is transferred to thespilled_ptrstack slot register state.Add support to
unmark_stack_slots_dynptrto not descend to its child slices (using bool slice parameter) so that we don't have circular calls when invokingrelease_reference. With this logic in place, we may have the following object relationships:+-- slice 1 (ref=1) source (ref=1) --> dynptr (ref=1) --|-- slice 2 (ref=1) +-- slice 3 (ref=1)Destroying dynptr prunes the dynptr and all its children slices, but does not affect the source. Releasing the source will effectively prune the entire ownership tree. Dynptr clones with same ref=1 will be parallel in the ownership tree.
+-- orig dptr (ref=1) --> slice 1 (ref=1) source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1) +-- clone dptr (ref=1) --> slice 3 (ref=1)In such a case, only child slices of the dynptr clone being destroyed are invalidated. Likewise, if the source object goes away, the whole tree ends up getting pruned.
Cc Amery Hung <[email protected]>Fixes 81bb1c2c3e8e ("bpf: net_sched: Add basic bpf qdisc kfuncs")Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- kernel/bpf/verifier.c | 81 ++++++++++++++++++++++++++++--------------- 1 file changed, 54 insertions(+), 27 deletions(-)kernel/bpf/verifier.c +2
-1@@ -198,7 +198,7 @@ struct bpf_verifier_stack_elem {static int acquire_reference (struct bpf_verifier_env *env,int insn_idx);static int release_reference_nomark (struct bpf_verifier_state *state,int ref_obj_id); -static int release_reference (struct bpf_verifier_env *env,int ref_obj_id); +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id,bool objects);static void invalidate_non_owning_refs (struct bpf_verifier_env *env);static bool in_rbtree_lock_required_cb (struct bpf_verifier_env *env); static int ref_set_non_owning(struct bpf_verifier_env *env,@@ -299,6 +299,7 @@ struct bpf_kfunc_call_arg_meta {const char *func_name;/* Out parameters */ u32 ref_obj_id; +u32 ref_obj_cnt;u8 release_regno;bool r0_rdonly;u32 ret_btf_id;kernel/bpf/verifier.c static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_ · line 763
Quoted 5 lines
kernel/bpf/verifier.c +1
-1@@ -759,7 +760,7 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_mark_dynptr_stack_regs (env, &state->stack[spi].spilled_ptr, &state->stack[spi -1 ].spilled_ptr, type); -if (dynptr_type_refcounted (type)) { +if (dynptr_type_refcounted (type) || clone_ref_obj_id) {it would be a bug to have both conditions true, right? Should we express this a bit more explicitly?
maybe something likeint id = 0;
if (dynptr_type_refcounted(type)) { WARN_ON_ONCE(clone_ref_obj_id); id = acquire_reference(env, insn_idx); } if (clone_ref_obj_id) id = clone_ref_obj_id;
Quoted 6 lines
state->stack[spi].spilled_ptr.ref_obj_id = id; state->stack[spi - 1].spilled_ptr.ref_obj_id = id; state->stack[spi].spilled_ptr.live |=
REG_LIVE_WRITTEN; state->stack[spi - 1].spilled_ptr.live |=REG_LIVE_WRITTEN;?
Yes.
Quoted 22 lines
Quoted 22 lines
kernel/bpf/verifier.c +5
-6/* The id is used to track proper releasing */ int id;@@ -818,22 +819,19 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat state->stack[spi -1 ].spilled_ptr.live |=REG_LIVE_WRITTEN ; } -static int unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_reg_state *reg) +static int __unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_func_state *state, +int spi,bool slice) { -struct bpf_func_state *state =func (env, reg); -int spi, ref_obj_id, i; +u32 ref_obj_id; +int i; - spi =dynptr_get_spi (env, reg); -if (spi <0 ) -return spi; + ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id;
Quoted 4 lines
kernel/bpf/verifier.c static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat · line 830
Quoted 2 lines
kernel/bpf/verifier.c +1
-1-if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type)) { +if (!dynptr_type_refcounted (state->stack[spi].spilled_ptr.dynptr.type) && !ref_obj_id) {Eduard already pointed out that
!ref_obj_idcondition is enough now
Ack, will change.
Quoted 21 lines
Quoted 11 lines
kernel/bpf/verifier.c +0
-2invalidate_dynptr (env, state, spi);return 0 ; } - ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; -/* If the dynptr has a ref_obj_id, then we need to invalidate * two things: * kernel/bpf/verifier.c static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re · line 844
Quoted 6 lines
kernel/bpf/verifier.c +2
-1@@ -842,7 +840,8 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re*/ /* Invalidate any slices associated with this dynptr */ -WARN_ON_ONCE (release_reference (env, ref_obj_id)); +if (slice) +WARN_ON_ONCE (release_reference (env, ref_obj_id, false));tbh, this bool flag to
release_referenceto prevent cycles feels like a hack and that we are not solving this properly...
Quoted 5 lines
also, changes in this patch still won't properly support the case where we create dynptr -> slice -> dynptr -> slice -> ... chains, right? Any dynptr in the chain will cause "release" of any other dynptr, including parents, right?
If they share the same ref_obj_id, yes.
Also, can we construct dynptr from referenced slices? I don't think anything except map values work right now.
In the chain example, how will we invalidate a dynptr in the later chain? Like by overwriting it? Just thinking what and how to test this.
Regardless, it is a valid point.
I can choose to introduce a ref_level so that we can track hierarchies and only prune things at the same or level below.
Any opinions on how you think this should be done?
Quoted 28 lines
Quoted 3 lines
kernel/bpf/verifier.c +0
-0/* Invalidate any dynptr clones */ for (i =1 ; i < state->allocated_stack /BPF_REG_SIZE ; i++) {[...]
kernel/bpf/verifier.c static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob · line 10234
Quoted 6 lines
kernel/bpf/verifier.c +1
-1@@ -10220,12 +10231,12 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob* * This is the release function corresponding to acquire_reference(). Idempotent. */ -static int release_reference (struct bpf_verifier_env *env,int ref_obj_id) +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id,bool objects)hm, "objects" really doesn't describe anything...
Quoted 13 lines
kernel/bpf/verifier.c +1
-1{struct bpf_verifier_state *vstate = env->cur_state;struct bpf_func_state *state;struct bpf_reg_state *reg; -int err; +int err, spi; err =release_reference_nomark (vstate, ref_obj_id);if (err)@@ -10236,6 +10247,19 @@ static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)mark_reg_invalid (env, reg); }));
Quoted 5 lines
kernel/bpf/verifier.c static int release_reference(struct bpf_verifier_env *env, int ref_obj_id) · line 10251
Quoted 2 lines
kernel/bpf/verifier.c +2
-0+if (!objects) +return 0 ;instead of a bool flag into
release_reference, shouldn't the below logic be just a separate function?
Yes, that can make it clearer. It may be unnecessary if the function knows which level of the ownership hierarchy to prune correctly, rn the flag is partially there because we can call recursively into it from unmark_stack_slots_dynptr.
Quoted 17 lines
Quoted 15 lines
kernel/bpf/verifier.c +11
-0+ +bpf_for_each_spilled_reg (spi, state, reg, (1 <<STACK_DYNPTR )) { +if (!reg) +continue ; +if (!reg->dynptr.first_slot || reg->ref_obj_id != ref_obj_id) +continue ; + err =__unmark_stack_slots_dynptr (env, state, spi, false); +if (err) +return err; + } +return 0 ; }[...]
Re: [RFC PATCH bpf-next/net v1 01/13] bpf: Tie dynptrs to referenced source objects
Quoted 55 lines · Kumar Kartikeya Dwivedi
Quoted 53 lines · Andrii Nakryiko
Quoted 50 lines · Kumar Kartikeya Dwivedi
Ensure that for dynptr constructors (
MEM_UNINITarg_type) taking a referenced object (ref_obj_id) as their memory source, we set theref_obj_idof the dynptr appropriately as well. This allows us to tie the lifetime of the dynptr to its source and properly invalidate it when the source object is released.For helpers, we don't have such cases yet as
bpf_dynptr_from_memis not permitted for anything other thanPTR_TO_MAP_VALUE, but still passmeta->ref_obj_idasclone_ref_obj_idin case this is relaxed in future. Since they are ossified we knowdynptr_from_memis the only relevant helper and takes one memory argument, so we know themeta->ref_obj_idif non-zero will belong to it.For kfuncs, make sure for constructors, only 1
ref_obj_idargument is seen, as more than one can be ambiguous in terms ofref_obj_idtransfer. Since more kfuncs can be added with possibly multiple memory arguments, make suremeta->ref_obj_idreassignment won't cause incorrect lifetime analysis in the future usingref_obj_cntlogic. Set thisref_obj_idas theclone_ref_obj_id, so that it is transferred to thespilled_ptrstack slot register state.Add support to
unmark_stack_slots_dynptrto not descend to its child slices (using bool slice parameter) so that we don't have circular calls when invokingrelease_reference. With this logic in place, we may have the following object relationships:+-- slice 1 (ref=1) source (ref=1) --> dynptr (ref=1) --|-- slice 2 (ref=1) +-- slice 3 (ref=1)Destroying dynptr prunes the dynptr and all its children slices, but does not affect the source. Releasing the source will effectively prune the entire ownership tree. Dynptr clones with same ref=1 will be parallel in the ownership tree.
+-- orig dptr (ref=1) --> slice 1 (ref=1) source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1) +-- clone dptr (ref=1) --> slice 3 (ref=1)In such a case, only child slices of the dynptr clone being destroyed are invalidated. Likewise, if the source object goes away, the whole tree ends up getting pruned.
Cc Amery Hung <[email protected]>Fixes 81bb1c2c3e8e ("bpf: net_sched: Add basic bpf qdisc kfuncs")Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- kernel/bpf/verifier.c | 81 ++++++++++++++++++++++++++++--------------- 1 file changed, 54 insertions(+), 27 deletions(-)
[...]
Quoted 27 lines
Quoted 21 lines
Quoted 11 lines
invalidate_dynptr(env, state, spi); return 0; }kernel/bpf/verifier.c +0
-2- ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; -/* If the dynptr has a ref_obj_id, then we need to invalidate * two things: * kernel/bpf/verifier.c static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re · line 844
Quoted 6 lines
kernel/bpf/verifier.c +2
-1@@ -842,7 +840,8 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re*/ /* Invalidate any slices associated with this dynptr */ -WARN_ON_ONCE (release_reference (env, ref_obj_id)); +if (slice) +WARN_ON_ONCE (release_reference (env, ref_obj_id, false));tbh, this bool flag to
release_referenceto prevent cycles feels like a hack and that we are not solving this properly...Quoted 5 lines
also, changes in this patch still won't properly support the case where we create dynptr -> slice -> dynptr -> slice -> ... chains, right? Any dynptr in the chain will cause "release" of any other dynptr, including parents, right?
Quoted 3 lines
If they share the same
ref_obj_id, yes.
Also, can we construct dynptr from referenced slices? I don't think anything except map values work right now.
Amery wanted to allow creating bpf_dynptr_from_mem() from any PTR_TO_MEM, which would allow creating ringbuf_dynptr -> slice [ -> mem_dynptr -> slice ]* chains.
Quoted 3 lines
In the chain example, how will we invalidate a dynptr in the later chain? Like by overwriting it? Just thinking what and how to test this.
Yes, user can always just write over dynptr slots on the stack, which doesn't even have to be intentional. mem dynptr is never explicitly released, so technically you can just stop using it and use that stack space for some other variables.
Quoted 5 lines
Regardless, it is a valid point.
I can choose to introduce a
ref_levelso that we can track hierarchies and only prune things at the same or level below.
Any opinions on how you think this should be done?
I was thinking that we can form an (implicit) tree of ref objects through their id and ref_obj_id properties. Each new dynptr gets its own id and sets its ref_obj_id to whatever id of the resource we are creating dynptr from (so if it's a slice, then I guess we'd need to use slices's ref_obj_id, right?). And so you'd have:
ringbuf_dynptr (id=1) -> slice (id=2,ref_obj_id=1) -> mem_dynptr(id=3,ref_obj_id=1) -> slice(id=4,**ref_obj_id=3**) -> mem_dynptr(id=5,ref_obj_id=3)
And so if you overwrite mem_dynptr(id=3), then you invalidate id=3,4,5; if you overwrite just mem_dynptr(id=5), only that one is invalidated (and id=1,2,3,4 stay valid). If you happen to release the original ringbuf_dynptr(id=1), the entire "tree" is invalidated, of course.
Quoted 57 lines
Quoted 29 lines
Quoted 4 lines
kernel/bpf/verifier.c +0
-0/* Invalidate any dynptr clones */ for (i =1 ; i < state->allocated_stack /BPF_REG_SIZE ; i++) {[...]
kernel/bpf/verifier.c static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob · line 10234
Quoted 6 lines
kernel/bpf/verifier.c +1
-1@@ -10220,12 +10231,12 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob* * This is the release function corresponding to acquire_reference(). Idempotent. */ -static int release_reference (struct bpf_verifier_env *env,int ref_obj_id) +static int release_reference (struct bpf_verifier_env *env,int ref_obj_id,bool objects)hm, "objects" really doesn't describe anything...
Quoted 13 lines
kernel/bpf/verifier.c +1
-1{struct bpf_verifier_state *vstate = env->cur_state;struct bpf_func_state *state;struct bpf_reg_state *reg; -int err; +int err, spi; err =release_reference_nomark (vstate, ref_obj_id);if (err)@@ -10236,6 +10247,19 @@ static int release_reference(struct bpf_verifier_env *env, int ref_obj_id)mark_reg_invalid (env, reg); }));Quoted 5 lines
kernel/bpf/verifier.c static int release_reference(struct bpf_verifier_env *env, int ref_obj_id) · line 10251
Quoted 2 lines
kernel/bpf/verifier.c +2
-0+if (!objects) +return 0 ;instead of a bool flag into
release_reference, shouldn't the below logic be just a separate function?Yes, that can make it clearer. It may be unnecessary if the function knows which level of the ownership hierarchy to prune correctly, rn the flag is partially there because we can call recursively into it from
unmark_stack_slots_dynptr.Quoted 17 lines
Quoted 15 lines
kernel/bpf/verifier.c +11
-0+ +bpf_for_each_spilled_reg (spi, state, reg, (1 <<STACK_DYNPTR )) { +if (!reg) +continue ; +if (!reg->dynptr.first_slot || reg->ref_obj_id != ref_obj_id) +continue ; + err =__unmark_stack_slots_dynptr (env, state, spi, false); +if (err) +return err; + } +return 0 ; }[...]
Kumar Kartikeya Dwivedi
maintainerRFC PATCH 2/13 bpf: Compare dynptr_id in regsafe
In reply to Kumar Kartikeya DwivediNow that PTR_TO_MEM can be invalidated due to skb going away, we must take care to be more careful in regsafe regarding state pruning. While ref_obj_id comparison will ensure that incorrect pruning is prevented, since we attach ref_obj_id of skb to the PTR_TO_MEM emanating from it, it is nonetheless clearer to also compare the dynptr_id as well.
--- kernel/bpf/verifier.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/bpf/verifier.c +2 -0
@@ -18426,6 +18426,8 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,range_within (rold, rcur) &&tnum_in (rold->var_off, rcur->var_off) &&check_ids (rold->id, rcur->id, idmap) && + (base_type (rold->type) ==PTR_TO_MEM ? +check_ids (rold->dynptr_id, rcur->dynptr_id, idmap) :1 ) &&check_ids (rold->ref_obj_id, rcur->ref_obj_id, idmap);case PTR_TO_PACKET_META :case PTR_TO_PACKET :
Signature · 1 line
2.47.1
Re: [RFC PATCH bpf-next/net v1 02/13] bpf: Compare dynptr_id in regsafe
Quoted 8 lines · Kumar Kartikeya Dwivedi
Now that
PTR_TO_MEMcan be invalidated due to skb going away, we must take care to be more careful in regsafe regarding state pruning. Whileref_obj_idcomparison will ensure that incorrect pruning is prevented, since we attachref_obj_idof skb to thePTR_TO_MEMemanating from it, it is nonetheless clearer to also compare thedynptr_idas well.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
[...]
Re: [RFC PATCH bpf-next/net v1 02/13] bpf: Compare dynptr_id in regsafe
In reply to Kumar Kartikeya DwivediQuoted 17 lines · Kumar Kartikeya Dwivedi
Now that
PTR_TO_MEMcan be invalidated due to skb going away, we must take care to be more careful in regsafe regarding state pruning. Whileref_obj_idcomparison will ensure that incorrect pruning is prevented, since we attachref_obj_idof skb to thePTR_TO_MEMemanating from it, it is nonetheless clearer to also compare thedynptr_idas well.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- kernel/bpf/verifier.c | 2 ++ 1 file changed, 2 insertions(+)kernel/bpf/verifier.c +0
-0
kernel/bpf/verifier.c static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold, · line 18431
Quoted 6 lines
kernel/bpf/verifier.c +2
-0@@ -18426,6 +18426,8 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,range_within (rold, rcur) &&tnum_in (rold->var_off, rcur->var_off) &&check_ids (rold->id, rcur->id, idmap) && + (base_type (rold->type) ==PTR_TO_MEM ? +check_ids (rold->dynptr_id, rcur->dynptr_id, idmap) :1 ) &&check_ids (rold->ref_obj_id, rcur->ref_obj_id, idmap);
hm... shall we split out PTR_TO_MEM case instead of making this not-so-simple condition even more not-so-simple? or (if people don't like that idea), I'd rather have this special PTR_TO_MEM handling as a separate if with return
Quoted 5 lines
kernel/bpf/verifier.c +0
-1case PTR_TO_PACKET_META :case PTR_TO_PACKET : --2.47.1
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH bpf-next/net v1 02/13] bpf: Compare dynptr_id in regsafe
Quoted 15 lines · Andrii Nakryiko
Quoted 12 lines · Kumar Kartikeya Dwivedi
Now that
PTR_TO_MEMcan be invalidated due to skb going away, we must take care to be more careful in regsafe regarding state pruning. Whileref_obj_idcomparison will ensure that incorrect pruning is prevented, since we attachref_obj_idof skb to thePTR_TO_MEMemanating from it, it is nonetheless clearer to also compare thedynptr_idas well.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- kernel/bpf/verifier.c | 2 ++ 1 file changed, 2 insertions(+)
Quoted 16 lines
kernel/bpf/verifier.c static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold, · line 18431
Quoted 11 lines
kernel/bpf/verifier.c +2
-0@@ -18426,6 +18426,8 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,range_within (rold, rcur) &&tnum_in (rold->var_off, rcur->var_off) &&check_ids (rold->id, rcur->id, idmap) && + (base_type (rold->type) ==PTR_TO_MEM ? +check_ids (rold->dynptr_id, rcur->dynptr_id, idmap) :1 ) &&check_ids (rold->ref_obj_id, rcur->ref_obj_id, idmap);hm... shall we split out
PTR_TO_MEMcase instead of making this not-so-simple condition even more not-so-simple? or (if people don't like that idea), I'd rather have this specialPTR_TO_MEMhandling as a separate if with return
I can split into a separate case.
Quoted 7 lines
Quoted 7 lines
kernel/bpf/verifier.c +0
-1case PTR_TO_PACKET_META :case PTR_TO_PACKET : --2.47.1
Kumar Kartikeya Dwivedi
maintainerRFC PATCH 3/13 selftests/bpf: Convert dynptr_fail to use vmlinux.h
In reply to Kumar Kartikeya DwivediLet's make use of vmlinux.h in dynptr_fail.c. No functional change intended, just a drive-by improvement.
--- tools/testing/selftests/bpf/progs/dynptr_fail.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
tools/testing/selftests/bpf/progs/dynptr_fail.c +6 -8
@@ -1,13 +1,11 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2022 Facebook */ - +#include <vmlinux.h> #include <errno.h> #include <string.h> #include <stdbool.h> -#include <linux/bpf.h> #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> -#include <linux/if_ether.h> #include "bpf_misc.h" #include "bpf_kfuncs.h"
@@ -46,7 +44,7 @@ struct {__type (value, __u64); } array_map4 SEC(".maps"); -struct sample { +struct dynptr_sample {int pid;long value;char comm[16 ];
@@ -95,7 +93,7 @@ __failure __msg("Unreleased reference id=4")int ringbuf_missing_release2 (void *ctx) {struct bpf_dynptr ptr1, ptr2; -struct sample *sample; +struct dynptr_sample *sample;bpf_ringbuf_reserve_dynptr (&ringbuf,sizeof (*sample),0 , &ptr1);bpf_ringbuf_reserve_dynptr (&ringbuf,sizeof (*sample),0 , &ptr2);
@@ -173,7 +171,7 @@ __failure __msg("type=mem expected=ringbuf_mem")int ringbuf_invalid_api (void *ctx) {struct bpf_dynptr ptr; -struct sample *sample; +struct dynptr_sample *sample;bpf_ringbuf_reserve_dynptr (&ringbuf,sizeof (*sample),0 , &ptr); sample =bpf_dynptr_data (&ptr,0 ,sizeof (*sample));
@@ -295,7 +293,7 @@ __failure __msg("invalid mem access 'scalar'")int data_slice_use_after_release1 (void *ctx) {struct bpf_dynptr ptr; -struct sample *sample; +struct dynptr_sample *sample;bpf_ringbuf_reserve_dynptr (&ringbuf,sizeof (*sample),0 , &ptr); sample =bpf_dynptr_data (&ptr,0 ,sizeof (*sample));
@@ -327,7 +325,7 @@ __failure __msg("invalid mem access 'scalar'")int data_slice_use_after_release2 (void *ctx) {struct bpf_dynptr ptr1, ptr2; -struct sample *sample; +struct dynptr_sample *sample;bpf_ringbuf_reserve_dynptr (&ringbuf,64 ,0 , &ptr1);bpf_ringbuf_reserve_dynptr (&ringbuf,sizeof (*sample),0 , &ptr2);
Signature · 1 line
2.47.1
Re: [RFC PATCH bpf-next/net v1 03/13] selftests/bpf: Convert dynptr_fail to use vmlinux.h
Quoted 5 lines · Kumar Kartikeya Dwivedi
Let's make use of
vmlinux.hindynptr_fail.c. No functional change intended, just a drive-by improvement.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
[...]
Kumar Kartikeya Dwivedi
maintainerRFC PATCH 4/13 selftests/bpf: Add tests for dynptr source object interaction
In reply to Kumar Kartikeya DwivediAdd a few tests to capture source object relationship with dynptr and their slices and ensure invalidation of everything works correctly.
--- .../testing/selftests/bpf/prog_tests/dynptr.c | 2 + .../testing/selftests/bpf/progs/dynptr_fail.c | 1 + .../selftests/bpf/progs/dynptr_fail_qdisc.c | 38 +++++++++++++++++++ 3 files changed, 41 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/dynptr_fail_qdisc.c
tools/testing/selftests/bpf/prog_tests/dynptr.c +2 -0
@@ -4,6 +4,7 @@#include <test_progs.h> #include <network_helpers.h> #include "dynptr_fail.skel.h" +#include "dynptr_fail_qdisc.skel.h" #include "dynptr_success.skel.h" enum test_setup_type {
@@ -161,4 +162,5 @@ void test_dynptr(void) }RUN_TESTS (dynptr_fail); +RUN_TESTS (dynptr_fail_qdisc); }
tools/testing/selftests/bpf/progs/dynptr_fail.c +1 -0
@@ -8,6 +8,7 @@#include <bpf/bpf_tracing.h> #include "bpf_misc.h" #include "bpf_kfuncs.h" +#include "bpf_experimental.h" char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/dynptr_fail_qdisc.c new file +38 -0
@@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> +#include "bpf_misc.h" +#include "bpf_experimental.h" + +char _license[] SEC("license") = "GPL"; + +/* +* Putting these tests in dynptr_fail.c causes other random tests to fail, +* keep them in their isolated CU. +*/ + +SEC("?struct_ops") +__failure __msg("invalid mem access 'scalar'") +int BPF_PROG(test_dynptr_source_release, struct sk_buff *skb, + struct Qdisc *sch, struct bpf_sk_buff_ptr *to_free) +{ +struct bpf_dynptr dptr, dptr2; +char buf[8 ], *data; + +bpf_dynptr_from_skb ((struct __sk_buff *)skb,0 , &dptr); +bpf_dynptr_read (buf,sizeof (buf), &dptr,0 ,0 ); +bpf_dynptr_clone (&dptr, &dptr2); + data =bpf_dynptr_slice (&dptr2,0 , buf,sizeof (buf)); +bpf_qdisc_skb_drop (skb, to_free); +/* These reads/writes now succeed since dynptr is destroyed. */ + *(char *)&dptr = *(char *)&dptr2; +return *data; +} + +SEC("?.struct_ops") +struct Qdisc_ops test_dynptr_qdisc = { + .enqueue = (void *)test_dynptr_source_release, + .id ="bpf_fq" , +};
Signature · 1 line
2.47.1
Eduard Zingerman
maintainerRe: [RFC PATCH bpf-next/net v1 04/13] selftests/bpf: Add tests for dynptr source object interaction
Quoted 5 lines · Kumar Kartikeya Dwivedi
Add a few tests to capture source object relationship with dynptr and their slices and ensure invalidation of everything works correctly.
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
It would be great to have a bit more test cases here, following your nice diagram from patch #1:
+-- orig dptr (ref=1) --> slice 1 (ref=1) source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1) +-- clone dptr (ref=1) --> slice 3 (ref=1)
In one test (probably extending the one in this patch):
- check that both orig and clone slices can be read at some point
- destroy orig or clone and check that one of the slices can be read, while another can't.
In another test:
- destroy source and check that both orig and clone slices are no longer accessible.
Also, is it possible to conjure a test case with two hierarchies? E.g. source1(ref=1) and source2(ref=2)?
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH bpf-next/net v1 04/13] selftests/bpf: Add tests for dynptr source object interaction
Quoted 16 lines · Eduard Zingerman
Quoted 5 lines · Kumar Kartikeya Dwivedi
Add a few tests to capture source object relationship with dynptr and their slices and ensure invalidation of everything works correctly.
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
It would be great to have a bit more test cases here, following your nice diagram from patch #1:
+-- orig dptr (ref=1) --> slice 1 (ref=1) source (ref=1) --|-- clone dptr (ref=1) --> slice 2 (ref=1) +-- clone dptr (ref=1) --> slice 3 (ref=1)
Quoted 8 lines
In one test (probably extending the one in this patch):
- check that both orig and clone slices can be read at some point
- destroy orig or clone and check that one of the slices can be read, while another can't.In another test:
- destroy source and check that both orig and clone slices are no longer accessible.
Agreed, will try and add more cases explicitly.
I struggled to have more than two callbacks automatically working with RUN_TESTS and struct_ops, so I guess I will do explicit loading and failure checks.
Quoted 4 lines
Also, is it possible to conjure a test case with two hierarchies? E.g. source1(ref=1) and source2(ref=2)?
Should be possible, I'll add it.
Quoted 1 line
[...]
Eduard Zingerman
maintainerRe: [RFC PATCH bpf-next/net v1 04/13] selftests/bpf: Add tests for dynptr source object interaction
Kumar Kartikeya Dwivedi <[email protected]> writes:
[...]
Quoted 3 lines
I struggled to have more than two callbacks automatically working with
RUN_TESTSandstruct_ops, so I guess I will do explicit loading and failure checks.
Could you please clarify the use-case a bit?
In theory two RUN_TESTS should be independent.
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH bpf-next/net v1 04/13] selftests/bpf: Add tests for dynptr source object interaction
Quoted 11 lines · Eduard Zingerman
Kumar Kartikeya Dwivedi <[email protected]> writes:
[...]
Quoted 3 lines
I struggled to have more than two callbacks automatically working with
RUN_TESTSandstruct_ops, so I guess I will do explicit loading and failure checks.Could you please clarify the use-case a bit?
In theory twoRUN_TESTSshould be independent.
I meant I see strange behavior when I have both .enqueue and .dequeue and do RUN_TESTS, for the second callback, I see unexpected load success. When keeping one of the callbacks, everything is ok.
I didn't dig further the other day before sending it out.
Quoted 2 lines
[...]
Kumar Kartikeya Dwivedi
maintainerRFC PATCH 5/13 locking/local_lock, mm: Replace localtry_ helpers with local_trylock_t type
In reply to Kumar Kartikeya DwivediFrom: Alexei Starovoitov <[email protected]>
Partially revert commit 0aaddfb06882 ("locking/local_lock: Introduce localtry_lock_t"). Remove localtry_*() helpers, since localtry_lock() name might
be misinterpreted as "try lock".
Introduce local_trylock[_irqsave]() helpers that only work with newly introduced local_trylock_t type.
Note that attempt to use local_trylock[_irqsave]() with local_lock_t will cause compilation failure.
Usage and behavior in !PREEMPT_RT:
local_lock_t lock; // sizeof(lock) == 0 local_lock(&lock); // preempt disable local_lock_irqsave(&lock, ...); // irq save
if (local_trylock_irqsave(&lock, ...)) // compilation error
local_trylock_t lock; // sizeof(lock) == 4 local_lock(&lock); // preempt disable, acquired = 1 local_lock_irqsave(&lock, ...); // irq save, acquired = 1 if (local_trylock(&lock)) // if (!acquired) preempt disable, acquired = 1
if (local_trylock_irqsave(&lock, ...)) // if (!acquired) irq save, acquired = 1
The existing local_lock_*() macros can be used either with local_lock_t or local_trylock_t.
With local_trylock_t they set acquired = 1 while local_unlock_*() clears it.
In !PREEMPT_RT local_lock_irqsave(local_lock_t *) disables interrupts to protect critical section, but it doesn't prevent NMI, so the fully reentrant code cannot use local_lock_irqsave(local_lock_t *) for exclusive access.
The local_lock_irqsave(local_trylock_t *) helper disables interrupts and sets acquired=1, so local_trylock_irqsave(local_trylock_t *) from NMI attempting to acquire the same lock will return false.
In PREEMPT_RT local_lock_irqsave() maps to preemptible spin_lock(). Map local_trylock_irqsave() to preemptible spin_trylock(). When in hard IRQ or NMI return false right away, since spin_trylock() is not safe due to explicit locking in the underneath rt_spin_trylock() implementation. Removing this explicit locking and attempting only "trylock" is undesired due to PI implications.
The local_trylock() without _irqsave can be used to avoid the cost of disabling/enabling interrupts by only disabling preemption, so local_trylock() in an interrupt attempting to acquire the same lock will return false.
Note there is no need to use local_inc for acquired variable, since it's a percpu variable with strict nesting scopes.
Note that guard(local_lock)(&lock) works only for "local_lock_t lock".
The patch also makes sure that local_lock_release(l) is called before WRITE_ONCE(l->acquired, 0). Though IRQs are disabled at this point the local_trylock() from NMI will succeed and local_lock_acquire(l) will warn.
--- include/linux/local_lock.h | 58 ++------ include/linux/local_lock_internal.h | 207 ++++++++++++---------------- mm/memcontrol.c | 39 +++--- 3 files changed, 114 insertions(+), 190 deletions(-)
include/linux/local_lock.h +8 -50
@@ -52,44 +52,23 @@__local_unlock_irqrestore (lock, flags)/** -* localtry_lock_init - Runtime initialize a lock instance - */ -#define localtry_lock_init (lock) __localtry_lock_init(lock) - -/** -* localtry_lock - Acquire a per CPU local lock -* @lock: The lock variable -*/ -#define localtry_lock (lock) __localtry_lock(lock) - -/** -* localtry_lock_irq - Acquire a per CPU local lock and disable interrupts -* @lock: The lock variable -*/ -#define localtry_lock_irq (lock) __localtry_lock_irq(lock) - -/** -* localtry_lock_irqsave - Acquire a per CPU local lock, save and disable -* interrupts -* @lock: The lock variable -* @flags: Storage for interrupt flags +* local_lock_init - Runtime initialize a lock instance */ -#define localtry_lock_irqsave (lock, flags) \ - __localtry_lock_irqsave(lock, flags) +#define local_trylock_init (lock) __local_trylock_init(lock)/** -* localtry_trylock - Try to acquire a per CPU local lock. +* local_trylock - Try to acquire a per CPU local lock * @lock: The lock variable * * The function can be used in any context such as NMI or HARDIRQ. Due to * locking constrains it will _always_ fail to acquire the lock in NMI or * HARDIRQ context on PREEMPT_RT. */ -#define localtry_trylock (lock) __localtry_trylock(lock) +#define local_trylock (lock) __local_trylock(lock)/** -* localtry_trylock_irqsave - Try to acquire a per CPU local lock, save and disable -* interrupts if acquired +* local_trylock_irqsave - Try to acquire a per CPU local lock, save and disable +* interrupts if acquired * @lock: The lock variable * @flags: Storage for interrupt flags *
@@ -97,29 +76,8 @@* locking constrains it will _always_ fail to acquire the lock in NMI or * HARDIRQ context on PREEMPT_RT. */ -#define localtry_trylock_irqsave (lock, flags) \ - __localtry_trylock_irqsave(lock, flags) - -/** -* local_unlock - Release a per CPU local lock -* @lock: The lock variable -*/ -#define localtry_unlock (lock) __localtry_unlock(lock) - -/** -* local_unlock_irq - Release a per CPU local lock and enable interrupts -* @lock: The lock variable -*/ -#define localtry_unlock_irq (lock) __localtry_unlock_irq(lock) - -/** -* localtry_unlock_irqrestore - Release a per CPU local lock and restore -* interrupt flags -* @lock: The lock variable -* @flags: Interrupt flags to restore -*/ -#define localtry_unlock_irqrestore (lock, flags) \ - __localtry_unlock_irqrestore(lock, flags) +#define local_trylock_irqsave (lock, flags) \ + __local_trylock_irqsave(lock, flags) DEFINE_GUARD(local_lock, local_lock_t __percpu*, local_lock(_T),
include/linux/local_lock_internal.h +87 -120
315 lines not on this page · raw message
mm/memcontrol.c +19 -20
@@ -1759,7 +1759,7 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg) }struct memcg_stock_pcp { -localtry_lock_t stock_lock; +local_trylock_t stock_lock;struct mem_cgroup *cached;/* this never be root cgroup */ unsigned int nr_pages;
@@ -1774,7 +1774,7 @@ struct memcg_stock_pcp {#define FLUSHING_CACHED_CHARGE 0 }; static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock) = { - .stock_lock =INIT_LOCALTRY_LOCK (stock_lock), + .stock_lock =INIT_LOCAL_TRYLOCK (stock_lock), };static DEFINE_MUTEX (percpu_charge_mutex);
@@ -1805,11 +1805,10 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages,if (nr_pages >MEMCG_CHARGE_BATCH )return ret; -if (!localtry_trylock_irqsave (&memcg_stock.stock_lock, flags)) { -if (!gfpflags_allow_spinning (gfp_mask)) -return ret; -localtry_lock_irqsave (&memcg_stock.stock_lock, flags); - } +if (gfpflags_allow_spinning (gfp_mask)) +local_lock_irqsave (&memcg_stock.stock_lock, flags); +else if (!local_trylock_irqsave (&memcg_stock.stock_lock, flags)) +return ret; stock =this_cpu_ptr (&memcg_stock); stock_pages =READ_ONCE (stock->nr_pages);
@@ -1818,7 +1817,7 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, ret = true; } -localtry_unlock_irqrestore (&memcg_stock.stock_lock, flags); +local_unlock_irqrestore (&memcg_stock.stock_lock, flags);return ret; }
@@ -1857,14 +1856,14 @@ static void drain_local_stock(struct work_struct *dummy)* drain_stock races is that we always operate on local CPU stock * here with IRQ disabled */ -localtry_lock_irqsave (&memcg_stock.stock_lock, flags); +local_lock_irqsave (&memcg_stock.stock_lock, flags); stock =this_cpu_ptr (&memcg_stock); old =drain_obj_stock (stock);drain_stock (stock);clear_bit (FLUSHING_CACHED_CHARGE , &stock->flags); -localtry_unlock_irqrestore (&memcg_stock.stock_lock, flags); +local_unlock_irqrestore (&memcg_stock.stock_lock, flags);obj_cgroup_put (old); }
@@ -1894,7 +1893,7 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) {unsigned long flags; -if (!localtry_trylock_irqsave (&memcg_stock.stock_lock, flags)) { +if (!local_trylock_irqsave (&memcg_stock.stock_lock, flags)) {/* * In case of unlikely failure to lock percpu stock_lock * uncharge memcg directly.
@@ -1907,7 +1906,7 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages)return ; }__refill_stock (memcg, nr_pages); -localtry_unlock_irqrestore (&memcg_stock.stock_lock, flags); +local_unlock_irqrestore (&memcg_stock.stock_lock, flags); }/*
@@ -1964,9 +1963,9 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) stock = &per_cpu (memcg_stock, cpu);/* drain_obj_stock requires stock_lock */ -localtry_lock_irqsave (&memcg_stock.stock_lock, flags); +local_lock_irqsave (&memcg_stock.stock_lock, flags); old =drain_obj_stock (stock); -localtry_unlock_irqrestore (&memcg_stock.stock_lock, flags); +local_unlock_irqrestore (&memcg_stock.stock_lock, flags);drain_stock (stock);obj_cgroup_put (old);
@@ -2787,7 +2786,7 @@ static void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat,unsigned long flags;int *bytes; -localtry_lock_irqsave (&memcg_stock.stock_lock, flags); +local_lock_irqsave (&memcg_stock.stock_lock, flags); stock =this_cpu_ptr (&memcg_stock);/*
@@ -2836,7 +2835,7 @@ static void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat,if (nr)__mod_objcg_mlstate (objcg, pgdat, idx, nr); -localtry_unlock_irqrestore (&memcg_stock.stock_lock, flags); +local_unlock_irqrestore (&memcg_stock.stock_lock, flags);obj_cgroup_put (old); }
@@ -2846,7 +2845,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes)unsigned long flags;bool ret = false; -localtry_lock_irqsave (&memcg_stock.stock_lock, flags); +local_lock_irqsave (&memcg_stock.stock_lock, flags); stock =this_cpu_ptr (&memcg_stock);if (objcg ==READ_ONCE (stock->cached_objcg) && stock->nr_bytes >= nr_bytes) {
@@ -2854,7 +2853,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) ret = true; } -localtry_unlock_irqrestore (&memcg_stock.stock_lock, flags); +local_unlock_irqrestore (&memcg_stock.stock_lock, flags);return ret; }
@@ -2946,7 +2945,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes,unsigned long flags;unsigned int nr_pages =0 ; -localtry_lock_irqsave (&memcg_stock.stock_lock, flags); +local_lock_irqsave (&memcg_stock.stock_lock, flags); stock =this_cpu_ptr (&memcg_stock);if (READ_ONCE (stock->cached_objcg) != objcg) {/* reset if necessary */
@@ -2960,7 +2959,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, stock->nr_bytes &= (PAGE_SIZE -1 ); } -localtry_unlock_irqrestore (&memcg_stock.stock_lock, flags); +local_unlock_irqrestore (&memcg_stock.stock_lock, flags);obj_cgroup_put (old);if (nr_pages)
Signature · 1 line
2.47.1
Kumar Kartikeya Dwivedi
maintainerRFC PATCH 6/13 bpf: Introduce bpf_dynptr_from_mem_slice
In reply to Kumar Kartikeya DwivediAdd a new bpf_dynptr_from_mem_slice kfunc to create a dynptr from a PTR_TO_BTF_ID exposing a variable-length slice of memory, represented by the new bpf_mem_slice type. This slice is read-only, for a read-write slice we can expose a distinct type in the future.
We rely on the previous commits ensuring source objects underpinning dynptr memory are tracked correctly for invalidation to ensure when a PTR_TO_BTF_ID holding a memory slice goes away, it's corresponding dynptrs get invalidated.
--- include/linux/bpf.h | 5 +++++ kernel/bpf/helpers.c | 32 ++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 +++++- 3 files changed, 42 insertions(+), 1 deletion(-)
include/linux/bpf.h +5 -0
@@ -1344,6 +1344,11 @@ enum bpf_dynptr_type { BPF_DYNPTR_TYPE_XDP, }; +struct bpf_mem_slice { +void *ptr; +size_t len; +}; +int bpf_dynptr_check_size (u32 size);u32 __bpf_dynptr_size (const struct bpf_dynptr_kern *ptr);const void *__bpf_dynptr_data (const struct bpf_dynptr_kern *ptr,u32 len);
kernel/bpf/helpers.c +32 -0
@@ -2826,6 +2826,37 @@ __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u32 dst_off,return 0 ; } +/** +* XXX +*/ +__bpf_kfunc int bpf_dynptr_from_mem_slice(struct bpf_mem_slice *mem_slice, u64 flags, struct bpf_dynptr *dptr__uninit) +{ +struct bpf_dynptr_kern *dptr = (struct bpf_dynptr_kern *)dptr__uninit; +int err; + +if (!mem_slice) +return -EINVAL ; + + err =bpf_dynptr_check_size (mem_slice->len); +if (err) + goto error; + +/* flags is currently unsupported */ +if (flags) { + err = -EINVAL ; + goto error; + } + +bpf_dynptr_init (dptr, mem_slice->ptr,BPF_DYNPTR_TYPE_LOCAL ,0 , mem_slice->len); +bpf_dynptr_set_rdonly (dptr); + +return 0 ; + +error: +bpf_dynptr_set_null (dptr); +return err; +} + __bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj) {return obj;
@@ -3275,6 +3306,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly) BTF_ID_FLAGS(func, bpf_dynptr_size) BTF_ID_FLAGS(func, bpf_dynptr_clone) BTF_ID_FLAGS(func, bpf_dynptr_copy) +BTF_ID_FLAGS(func, bpf_dynptr_from_mem_slice, KF_TRUSTED_ARGS)#ifdef CONFIG_NET BTF_ID_FLAGS(func, bpf_modify_return_test_tp)#endif
kernel/bpf/verifier.c +5 -1
@@ -12125,6 +12125,7 @@ enum special_kfunc_type { KF_bpf_res_spin_unlock, KF_bpf_res_spin_lock_irqsave, KF_bpf_res_spin_unlock_irqrestore, + KF_bpf_dynptr_from_mem_slice, };BTF_SET_START (special_kfunc_set)
@@ -12218,6 +12219,7 @@ BTF_ID(func, bpf_res_spin_lock) BTF_ID(func, bpf_res_spin_unlock) BTF_ID(func, bpf_res_spin_lock_irqsave) BTF_ID(func, bpf_res_spin_unlock_irqrestore) +BTF_ID(func, bpf_dynptr_from_mem_slice)static bool is_kfunc_ret_null (struct bpf_kfunc_call_arg_meta *meta) {
@@ -13139,7 +13141,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } } -if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) { +if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_mem_slice]) { + dynptr_arg_type |=DYNPTR_TYPE_LOCAL ; + }else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) { dynptr_arg_type |=DYNPTR_TYPE_SKB ; }else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_xdp]) { dynptr_arg_type |=DYNPTR_TYPE_XDP ;
Signature · 1 line
2.47.1
Re: [RFC PATCH bpf-next/net v1 06/13] bpf: Introduce bpf_dynptr_from_mem_slice
Quoted 11 lines · Kumar Kartikeya Dwivedi
Add a new
bpf_dynptr_from_mem_slicekfunc to create a dynptr from aPTR_TO_BTF_IDexposing a variable-length slice of memory, represented by the newbpf_mem_slicetype. This slice is read-only, for a read-write slice we can expose a distinct type in the future.We rely on the previous commits ensuring source objects underpinning dynptr memory are tracked correctly for invalidation to ensure when a
PTR_TO_BTF_IDholding a memory slice goes away, it's corresponding dynptrs get invalidated.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>
If I remove KF_TRUSTED_ARGS flag from the function nothing fails. But that's because with current interface slices will always have a ref_obj_id, right? And pointers with ref_obj_id are always trusted.
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH bpf-next/net v1 06/13] bpf: Introduce bpf_dynptr_from_mem_slice
Quoted 13 lines · Eduard Zingerman
Quoted 10 lines · Kumar Kartikeya Dwivedi
Add a new
bpf_dynptr_from_mem_slicekfunc to create a dynptr from aPTR_TO_BTF_IDexposing a variable-length slice of memory, represented by the newbpf_mem_slicetype. This slice is read-only, for a read-write slice we can expose a distinct type in the future.We rely on the previous commits ensuring source objects underpinning dynptr memory are tracked correctly for invalidation to ensure when a
PTR_TO_BTF_IDholding a memory slice goes away, it's corresponding dynptrs get invalidated.
Quoted 5 lines
Quoted 1 line
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>If I remove
KF_TRUSTED_ARGSflag from the function nothing fails. But that's because with current interface slices will always have aref_obj_id, right? And pointers withref_obj_idare always trusted.
Yes.
Quoted 4 lines
Acked-by Eduard Zingerman <[email protected]>[...]
Re: [RFC PATCH bpf-next/net v1 06/13] bpf: Introduce bpf_dynptr_from_mem_slice
In reply to Kumar Kartikeya DwivediQuoted 23 lines · Kumar Kartikeya Dwivedi
Add a new
bpf_dynptr_from_mem_slicekfunc to create a dynptr from aPTR_TO_BTF_IDexposing a variable-length slice of memory, represented by the newbpf_mem_slicetype. This slice is read-only, for a read-write slice we can expose a distinct type in the future.We rely on the previous commits ensuring source objects underpinning dynptr memory are tracked correctly for invalidation to ensure when a
PTR_TO_BTF_IDholding a memory slice goes away, it's corresponding dynptrs get invalidated.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 5 +++++ kernel/bpf/helpers.c | 32 ++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 +++++- 3 files changed, 42 insertions(+), 1 deletion(-)include/linux/bpf.h +0
-0
include/linux/bpf.h enum bpf_dynptr_type { · line 1349
Quoted 6 lines
include/linux/bpf.h +3
-0@@ -1344,6 +1344,11 @@ enum bpf_dynptr_type { BPF_DYNPTR_TYPE_XDP, }; +struct bpf_mem_slice { +void *ptr; +size_t len;
for better future extensibility and to avoid big-endian issues, let's have
u32 len;
u32 __reserved; /* or flags */
?
Quoted 13 lines
include/linux/bpf.h +2
-0+}; +int bpf_dynptr_check_size (u32 size);u32 __bpf_dynptr_size (const struct bpf_dynptr_kern *ptr);const void *__bpf_dynptr_data (const struct bpf_dynptr_kern *ptr,u32 len);kernel/bpf/helpers.c +0
-0@@ -2826,6 +2826,37 @@ __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u32 dst_off,return 0 ; }
kernel/bpf/helpers.c __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u32 dst_off, · line 2838
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+/** +* XXX +*/ +__bpf_kfunc int bpf_dynptr_from_mem_slice(struct bpf_mem_slice *mem_slice, u64 flags, struct bpf_dynptr *dptr__uninit) +{ +struct bpf_dynptr_kern *dptr = (struct bpf_dynptr_kern *)dptr__uninit; +int err; + +if (!mem_slice) +return -EINVAL ;
you have to initialize dynptr regardless of errors, just like bpf_dynptr_from_mem() does, so
err = -EINVAL;
goto error;
Quoted 66 lines
kernel/bpf/helpers.c +22
-0+ + err =bpf_dynptr_check_size (mem_slice->len); +if (err) + goto error; + +/* flags is currently unsupported */ +if (flags) { + err = -EINVAL ; + goto error; + } + +bpf_dynptr_init (dptr, mem_slice->ptr,BPF_DYNPTR_TYPE_LOCAL ,0 , mem_slice->len); +bpf_dynptr_set_rdonly (dptr); + +return 0 ; + +error: +bpf_dynptr_set_null (dptr); +return err; +} + __bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj) {return obj;@@ -3275,6 +3306,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly) BTF_ID_FLAGS(func, bpf_dynptr_size) BTF_ID_FLAGS(func, bpf_dynptr_clone) BTF_ID_FLAGS(func, bpf_dynptr_copy) +BTF_ID_FLAGS(func, bpf_dynptr_from_mem_slice, KF_TRUSTED_ARGS)#ifdef CONFIG_NET BTF_ID_FLAGS(func, bpf_modify_return_test_tp)#endif kernel/bpf/verifier.c +5
-2@@ -12125,6 +12125,7 @@ enum special_kfunc_type { KF_bpf_res_spin_unlock, KF_bpf_res_spin_lock_irqsave, KF_bpf_res_spin_unlock_irqrestore, + KF_bpf_dynptr_from_mem_slice, };BTF_SET_START (special_kfunc_set)@@ -12218,6 +12219,7 @@ BTF_ID(func, bpf_res_spin_lock) BTF_ID(func, bpf_res_spin_unlock) BTF_ID(func, bpf_res_spin_lock_irqsave) BTF_ID(func, bpf_res_spin_unlock_irqrestore) +BTF_ID(func, bpf_dynptr_from_mem_slice)static bool is_kfunc_ret_null (struct bpf_kfunc_call_arg_meta *meta) {@@ -13139,7 +13141,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } } -if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) { +if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_mem_slice]) { + dynptr_arg_type |=DYNPTR_TYPE_LOCAL ; + }else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) { dynptr_arg_type |=DYNPTR_TYPE_SKB ; }else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_xdp]) { dynptr_arg_type |=DYNPTR_TYPE_XDP ; --2.47.1
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH bpf-next/net v1 06/13] bpf: Introduce bpf_dynptr_from_mem_slice
Quoted 29 lines · Andrii Nakryiko
Quoted 26 lines · Kumar Kartikeya Dwivedi
Add a new
bpf_dynptr_from_mem_slicekfunc to create a dynptr from aPTR_TO_BTF_IDexposing a variable-length slice of memory, represented by the newbpf_mem_slicetype. This slice is read-only, for a read-write slice we can expose a distinct type in the future.We rely on the previous commits ensuring source objects underpinning dynptr memory are tracked correctly for invalidation to ensure when a
PTR_TO_BTF_IDholding a memory slice goes away, it's corresponding dynptrs get invalidated.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 5 +++++ kernel/bpf/helpers.c | 32 ++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 6 +++++- 3 files changed, 42 insertions(+), 1 deletion(-)include/linux/bpf.h +0
-0@@ -1344,6 +1344,11 @@ enum bpf_dynptr_type { BPF_DYNPTR_TYPE_XDP, };
Quoted 9 lines
include/linux/bpf.h enum bpf_dynptr_type { · line 1349
Quoted 3 lines
include/linux/bpf.h +3
-0+struct bpf_mem_slice { +void *ptr; +size_t len;for better future extensibility and to avoid big-endian issues, let's have
u32 len;
u32__reserved; /* or flags */
Ack.
Quoted 29 lines
?
Quoted 13 lines
include/linux/bpf.h +2
-0+}; +int bpf_dynptr_check_size (u32 size);u32 __bpf_dynptr_size (const struct bpf_dynptr_kern *ptr);const void *__bpf_dynptr_data (const struct bpf_dynptr_kern *ptr,u32 len);kernel/bpf/helpers.c +0
-0@@ -2826,6 +2826,37 @@ __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u32 dst_off,return 0 ; }kernel/bpf/helpers.c __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u32 dst_off, · line 2838
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+/** +* XXX +*/ +__bpf_kfunc int bpf_dynptr_from_mem_slice(struct bpf_mem_slice *mem_slice, u64 flags, struct bpf_dynptr *dptr__uninit) +{ +struct bpf_dynptr_kern *dptr = (struct bpf_dynptr_kern *)dptr__uninit; +int err; + +if (!mem_slice) +return -EINVAL ;you have to initialize dynptr regardless of errors, just like
bpf_dynptr_from_mem()does, so
Quoted 2 lines
err =
-EINVAL;
goto error;
Thanks for catching this, will fix.
Quoted 68 lines
Quoted 68 lines
kernel/bpf/helpers.c +22
-0+ + err =bpf_dynptr_check_size (mem_slice->len); +if (err) + goto error; + +/* flags is currently unsupported */ +if (flags) { + err = -EINVAL ; + goto error; + } + +bpf_dynptr_init (dptr, mem_slice->ptr,BPF_DYNPTR_TYPE_LOCAL ,0 , mem_slice->len); +bpf_dynptr_set_rdonly (dptr); + +return 0 ; + +error: +bpf_dynptr_set_null (dptr); +return err; +} + __bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj) {return obj;@@ -3275,6 +3306,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly) BTF_ID_FLAGS(func, bpf_dynptr_size) BTF_ID_FLAGS(func, bpf_dynptr_clone) BTF_ID_FLAGS(func, bpf_dynptr_copy) +BTF_ID_FLAGS(func, bpf_dynptr_from_mem_slice, KF_TRUSTED_ARGS)#ifdef CONFIG_NET BTF_ID_FLAGS(func, bpf_modify_return_test_tp)#endif kernel/bpf/verifier.c +5
-2@@ -12125,6 +12125,7 @@ enum special_kfunc_type { KF_bpf_res_spin_unlock, KF_bpf_res_spin_lock_irqsave, KF_bpf_res_spin_unlock_irqrestore, + KF_bpf_dynptr_from_mem_slice, };BTF_SET_START (special_kfunc_set)@@ -12218,6 +12219,7 @@ BTF_ID(func, bpf_res_spin_lock) BTF_ID(func, bpf_res_spin_unlock) BTF_ID(func, bpf_res_spin_lock_irqsave) BTF_ID(func, bpf_res_spin_unlock_irqrestore) +BTF_ID(func, bpf_dynptr_from_mem_slice)static bool is_kfunc_ret_null (struct bpf_kfunc_call_arg_meta *meta) {@@ -13139,7 +13141,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } } -if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) { +if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_mem_slice]) { + dynptr_arg_type |=DYNPTR_TYPE_LOCAL ; + }else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_skb]) { dynptr_arg_type |=DYNPTR_TYPE_SKB ; }else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_from_xdp]) { dynptr_arg_type |=DYNPTR_TYPE_XDP ; --2.47.1
Kumar Kartikeya Dwivedi
maintainerRFC PATCH 7/13 bpf: Introduce per-prog stdout/stderr streams
In reply to Kumar Kartikeya DwivediIntroduce a set of kfuncs to implement per BPF program stdout and stderr streams. This is implemented as a linked list of dynamically allocated strings whenever a message is printed. This can be done by the kernel or the program itself using the bpf_prog_stream_vprintk kfunc. In-kernel wrappers are provided over streams to ease the process.
The idea is that everytime messages need to be dumped, the reader would pull out the whole batch of messages from the stream at once, and then pop each string one by one and start printing it out (how exactly is left up to the BPF program reading the log, but usually it will be streaming data back into a ring buffer that is consumed by user space).
The use of a lockless list warrants that we be careful about the ordering of messages. When being added, the order maintained is new to old, therefore after deletion, we must reverse the list before iterating and popping out elements from it.
Overall, this infrastructure provides NMI-safe any context printing built on top of the NMI-safe any context bpf_mem_alloc() interface.
Later patches will add support for printing splats in case of BPF arena page faults, rqspinlock deadlocks, and cond_break timeouts.
--- include/linux/bpf.h | 52 ++++- kernel/bpf/Makefile | 2 +- kernel/bpf/core.c | 12 + kernel/bpf/helpers.c | 26 +-- kernel/bpf/stream.c | 496 ++++++++++++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 2 +- kernel/bpf/verifier.c | 15 +- 7 files changed, 582 insertions(+), 23 deletions(-) create mode 100644 kernel/bpf/stream.c
include/linux/bpf.h +51 -1
@@ -1523,6 +1523,36 @@ struct btf_mod_pair {struct bpf_kfunc_desc_tab ; +enum bpf_stream_id { +BPF_STDOUT =1 , +BPF_STDERR =2 , +}; + +enum { +BPF_STREAM_ELEM_F_PAGE = (1 <<0 ), +BPF_STREAM_ELEM_F_NEXT = (1 <<1 ), +BPF_STREAM_ELEM_F_MASK = (BPF_STREAM_ELEM_F_PAGE |BPF_STREAM_ELEM_F_NEXT ), +}; + +struct bpf_stream_elem { +struct llist_node node; +struct bpf_mem_slice mem_slice; +union { +unsigned long flags; +struct bpf_stream_elem *next; + }; +char str[]; +}; + +struct bpf_stream_elem_batch { +struct llist_node *node; +}; + +struct bpf_stream { +enum bpf_stream_id stream_id; +struct llist_head log; +}; +struct bpf_prog_aux {atomic64_t refcnt;u32 used_map_cnt;
@@ -1631,6 +1661,7 @@ struct bpf_prog_aux {struct work_struct work;struct rcu_head rcu; }; +struct bpf_stream stream[2 ]; };struct bpf_prog {
@@ -2390,6 +2421,8 @@ int generic_map_delete_batch(struct bpf_map *map,struct bpf_map *bpf_map_get_curr_or_next (u32 *id);struct bpf_prog *bpf_prog_get_curr_or_next (u32 *id); + +struct page *__bpf_alloc_page (int nid);int bpf_map_alloc_pages (const struct bpf_map *map,int nid,unsigned long nr_pages,struct page **page_array);#ifdef CONFIG_MEMCG
@@ -3528,6 +3561,16 @@ bool btf_id_set_contains(const struct btf_id_set *set, u32 id);#define MAX_BPRINTF_VARARGS 12#define MAX_BPRINTF_BUF 1024 +/* Per-cpu temp buffers used by printf-like helpers to store the bprintf binary +* arguments representation. +*/ +#define MAX_BPRINTF_BIN_ARGS 512 + +struct bpf_bprintf_buffers { +char bin_args[MAX_BPRINTF_BIN_ARGS ]; +char buf[MAX_BPRINTF_BUF ]; +}; +struct bpf_bprintf_data {u32 *bin_args;char *buf;
@@ -3535,9 +3578,16 @@ struct bpf_bprintf_data {bool get_buf; }; -int bpf_bprintf_prepare (char *fmt,u32 fmt_size,const u64 *raw_args, +int bpf_bprintf_prepare (const char *fmt,u32 fmt_size,const u64 *raw_args,u32 num_args,struct bpf_bprintf_data *data);void bpf_bprintf_cleanup (struct bpf_bprintf_data *data); +int bpf_try_get_buffers (struct bpf_bprintf_buffers **bufs); +void bpf_put_buffers (void ); + +void bpf_prog_stream_init (struct bpf_prog *prog); +void bpf_prog_stream_free (struct bpf_prog *prog); +__printf(2, 3) +int bpf_prog_stderr_printk (struct bpf_prog *prog,const char *fmt, ...);#ifdef CONFIG_BPF_LSM void bpf_cgroup_atype_get (u32 attach_btf_id,int cgroup_atype);
kernel/bpf/Makefile +1 -1
@@ -14,7 +14,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o obj-$(CONFIG_BPF_JIT) += trampoline.o -obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o rqspinlock.o +obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o rqspinlock.o stream.o ifeq ($(CONFIG_MMU)$(CONFIG_64BIT),yy) obj-$(CONFIG_BPF_SYSCALL) += arena.o range_tree.o endif
kernel/bpf/core.c +12 -0
@@ -134,6 +134,10 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flagmutex_init (&fp->aux->ext_mutex);mutex_init (&fp->aux->dst_mutex); +#ifdef CONFIG_BPF_SYSCALL +bpf_prog_stream_init (fp); +#endif +return fp; }
@@ -2856,6 +2860,7 @@ static void bpf_prog_free_deferred(struct work_struct *work) aux = container_of(work, struct bpf_prog_aux, work);#ifdef CONFIG_BPF_SYSCALL bpf_free_kfunc_btf_tab (aux->kfunc_btf_tab); +bpf_prog_stream_free (aux->prog);#endif #ifdef CONFIG_CGROUP_BPF if (aux->cgroup_atype !=CGROUP_BPF_ATTACH_TYPE_INVALID )
@@ -2872,6 +2877,13 @@ static void bpf_prog_free_deferred(struct work_struct *work)if (aux->dst_trampoline)bpf_trampoline_put (aux->dst_trampoline);for (i =0 ; i < aux->real_func_cnt; i++) { +#ifdef CONFIG_BPF_SYSCALL +/* Ensure we don't push to subprog lists. */ +if (bpf_is_subprog (aux->func[i])) { +WARN_ON_ONCE (!llist_empty (&aux->func[i]->aux->stream[0 ].log)); +WARN_ON_ONCE (!llist_empty (&aux->func[i]->aux->stream[1 ].log)); + } +#endif /* We can just unlink the subprog poke descriptor table as * it was originally linked to the main program and is also * released along with it.
kernel/bpf/helpers.c +11 -15
@@ -761,22 +761,13 @@ static int bpf_trace_copy_string(char *buf, void *unsafe_ptr, char fmt_ptype,return -EINVAL ; } -/* Per-cpu temp buffers used by printf-like helpers to store the bprintf binary -* arguments representation. -*/ -#define MAX_BPRINTF_BIN_ARGS 512 -/* Support executing three nested bprintf helper calls on a given CPU */ #define MAX_BPRINTF_NEST_LEVEL 3 -struct bpf_bprintf_buffers { -char bin_args[MAX_BPRINTF_BIN_ARGS ]; -char buf[MAX_BPRINTF_BUF ]; -}; static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs); static DEFINE_PER_CPU(int, bpf_bprintf_nest_level); -static int try_get_buffers (struct bpf_bprintf_buffers **bufs) +int bpf_try_get_buffers (struct bpf_bprintf_buffers **bufs) {int nest_level;
@@ -792,16 +783,21 @@ static int try_get_buffers(struct bpf_bprintf_buffers **bufs)return 0 ; } -void bpf_bprintf_cleanup (struct bpf_bprintf_data *data) +void bpf_put_buffers (void ) { -if (!data->bin_args && !data->buf) -return ;if (WARN_ON_ONCE (this_cpu_read (bpf_bprintf_nest_level) ==0 ))return ;this_cpu_dec (bpf_bprintf_nest_level);preempt_enable (); } +void bpf_bprintf_cleanup (struct bpf_bprintf_data *data) +{ +if (!data->bin_args && !data->buf) +return ; +bpf_put_buffers (); +} +/* * bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers *
@@ -816,7 +812,7 @@ void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)* In argument preparation mode, if 0 is returned, safe temporary buffers are * allocated and bpf_bprintf_cleanup should be called to free them after use. */ -int bpf_bprintf_prepare (char *fmt,u32 fmt_size,const u64 *raw_args, +int bpf_bprintf_prepare (const char *fmt,u32 fmt_size,const u64 *raw_args,u32 num_args,struct bpf_bprintf_data *data) {bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
@@ -832,7 +828,7 @@ int bpf_bprintf_prepare(char *fmt, u32 fmt_size, const u64 *raw_args,return -EINVAL ; fmt_size = fmt_end - fmt; -if (get_buffers &&try_get_buffers (&buffers)) +if (get_buffers &&bpf_try_get_buffers (&buffers))return -EBUSY ;if (data->get_bin_args) {
kernel/bpf/stream.c new file +496 -0
496 lines not on this page · raw message
kernel/bpf/syscall.c +1 -1
@@ -575,7 +575,7 @@ static bool can_alloc_pages(void) !IS_ENABLED (CONFIG_PREEMPT_RT ); } -static struct page *__bpf_alloc_page (int nid) +struct page *__bpf_alloc_page (int nid) {if (!can_alloc_pages ())return try_alloc_pages (nid,0 );
kernel/bpf/verifier.c +10 -5
@@ -12126,6 +12126,7 @@ enum special_kfunc_type { KF_bpf_res_spin_lock_irqsave, KF_bpf_res_spin_unlock_irqrestore, KF_bpf_dynptr_from_mem_slice, + KF_bpf_stream_get, };BTF_SET_START (special_kfunc_set)
@@ -12220,6 +12221,7 @@ BTF_ID(func, bpf_res_spin_unlock) BTF_ID(func, bpf_res_spin_lock_irqsave) BTF_ID(func, bpf_res_spin_unlock_irqrestore) BTF_ID(func, bpf_dynptr_from_mem_slice) +BTF_ID(func, bpf_stream_get)static bool is_kfunc_ret_null (struct bpf_kfunc_call_arg_meta *meta) {
@@ -13883,10 +13885,11 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, regs[BPF_REG_0 ].type =PTR_TO_BTF_ID ; regs[BPF_REG_0 ].btf_id = ptr_type_id; -if (meta.func_id == special_kfunc_list[KF_bpf_get_kmem_cache]) +if (meta.func_id == special_kfunc_list[KF_bpf_get_kmem_cache]) { regs[BPF_REG_0 ].type |=PTR_UNTRUSTED ; - -if (is_iter_next_kfunc (&meta)) { + }else if (meta.func_id == special_kfunc_list[KF_bpf_stream_get]) { + regs[BPF_REG_0 ].type |=PTR_TRUSTED ; + }else if (is_iter_next_kfunc (&meta)) {struct bpf_reg_state *cur_iter; cur_iter =get_iter_from_state (env->cur_state, &meta);
@@ -21520,8 +21523,10 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) { insn_buf[0 ] =BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_1 ); *cnt =1 ; - } else if (is_bpf_wq_set_callback_impl_kfunc(desc->func_id)) { -struct bpf_insn ld_addrs[2 ] = {BPF_LD_IMM64 (BPF_REG_4 , (long )env->prog->aux) }; + } else if (is_bpf_wq_set_callback_impl_kfunc(desc->func_id) || + desc->func_id == special_kfunc_list[KF_bpf_stream_get]) { +u32 regno =is_bpf_wq_set_callback_impl_kfunc (desc->func_id) ?BPF_REG_4 :BPF_REG_2 ; +struct bpf_insn ld_addrs[2 ] = {BPF_LD_IMM64 (regno, (long )env->prog->aux) }; insn_buf[0 ] = ld_addrs[0 ]; insn_buf[1 ] = ld_addrs[1 ];
Signature · 1 line
2.47.1
Eduard Zingerman
maintainerRe: [RFC PATCH bpf-next/net v1 07/13] bpf: Introduce per-prog stdout/stderr streams
Quoted 25 lines · Kumar Kartikeya Dwivedi
Introduce a set of kfuncs to implement per BPF program stdout and stderr streams. This is implemented as a linked list of dynamically allocated strings whenever a message is printed. This can be done by the kernel or the program itself using the
bpf_prog_stream_vprintkkfunc. In-kernel wrappers are provided over streams to ease the process.The idea is that everytime messages need to be dumped, the reader would pull out the whole batch of messages from the stream at once, and then pop each string one by one and start printing it out (how exactly is left up to the BPF program reading the log, but usually it will be streaming data back into a ring buffer that is consumed by user space).
The use of a lockless list warrants that we be careful about the ordering of messages. When being added, the order maintained is new to old, therefore after deletion, we must reverse the list before iterating and popping out elements from it.
Overall, this infrastructure provides NMI-safe any context printing built on top of the NMI-safe any context
bpf_mem_alloc()interface.Later patches will add support for printing splats in case of BPF arena page faults, rqspinlock deadlocks, and
cond_breaktimeouts.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
[...]
Quoted 5 lines
kernel/bpf/stream.c new file +0
-0
[...]
Quoted 1 line
+static int
bpf_stream_page_check_room(struct bpf_stream_page *stream_page, int len)
^^^^^^^
len is never used
Quoted 10 lines
kernel/bpf/stream.c new file +10
-0+{ + int min = offsetof(struct bpf_stream_elem, str[0]); +int consumed = stream_page->consumed; +int total =BPF_STREAM_PAGE_SZ ; +int rem =max (0 , total - consumed - min); + +/* Let's give room of at least 8 bytes. */ +WARN_ON_ONCE (rem %8 !=0 ); +return rem <8 ?0 : rem; +}
[...]
Quoted 12 lines
kernel/bpf/stream.c new file +12
-0+static struct bpf_stream_elem *bpf_stream_elem_alloc (int len) +{ +const int max_len =ARRAY_SIZE ((struct bpf_bprintf_buffers ){}.buf); +struct bpf_stream_elem *elem; + +/* +* We may overflow, but we should never need more than one page size +* worth of memory. This can be lifted, but we'd need to adjust the +* other code to keep allocating more pages to overflow messages. +*/ +BUILD_BUG_ON (max_len >BPF_STREAM_PAGE_SZ ); +/*
kernel/bpf/stream.c line 241
Quoted 10 lines
kernel/bpf/stream.c new file +10
-0+* Length denotes the amount of data to be written as part of stream element, +* thus includes '\0' byte. We're capped by how much bpf_bprintf_buffers can +* accomodate, therefore deny allocations that won't fit into them. +*/ +if (len <0 || len > max_len) +return NULL ; + + elem =bpf_stream_elem_alloc_from_bpf_ma (len); +if (!elem) + elem =bpf_stream_elem_alloc_from_stream_page (len);
So, the stream page is a backup mechanism, right?
I'm curious, did you compare how many messages are dropped if there is no such backup? Also, how much memory is wasted if there is no "spillover" mechanism (BPF_STREAM_ELEM_F_NEXT).
Are these complications absolutely necessary?
kernel/bpf/stream.c line 248
Quoted 7 lines
kernel/bpf/stream.c new file +7
-0+return elem; +} + +__bpf_kfunc_start_defs (); + +static int bpf_stream_push_str (struct bpf_stream *stream,const char *str,int len) +{
This function accumulates elements in &stream->log w/o a cap. Why is this not a problem if e.g. user space never flushes streams for the program?
Quoted 31 lines
kernel/bpf/stream.c new file +31
-0+struct bpf_stream_elem *elem, *next =NULL ; +int room =0 , rem =0 ; + +/* +* Allocate a bpf_prog_stream_elem and push it to the bpf_prog_stream +* log, elements will be popped at once and reversed to print the log. +*/ + elem =bpf_stream_elem_alloc (len); +if (!elem) +return -ENOMEM ; + room = elem->mem_slice.len; +if (elem->flags &BPF_STREAM_ELEM_F_NEXT ) { + next = (struct bpf_stream_elem *)((unsigned long )elem->next & ~BPF_STREAM_ELEM_F_MASK ); + rem = next->mem_slice.len; + } + +memcpy (elem->str, str, room); +if (next) +memcpy (next->str, str + room, rem); + +if (next) { + elem->node.next = &next->node; + next->node.next =NULL ; + +llist_add_batch (&elem->node, &next->node, &stream->log); + }else { +llist_add (&elem->node, &stream->log); + } + +return 0 ; +}
[...]
kernel/bpf/stream.c line 446
Quoted 8 lines
kernel/bpf/stream.c new file +8
-0+BTF_KFUNCS_START(stream_consumer_kfunc_set) +BTF_ID_FLAGS(func, bpf_stream_next_elem_batch, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_stream_free_elem_batch, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_stream_next_elem, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_stream_free_elem, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_prog_stream_get, KF_ACQUIRE | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_prog_stream_put, KF_RELEASE) +BTF_KFUNCS_END(stream_consumer_kfunc_set)
This is a complicated API.
If we anticipate that users intend to write this info to ring buffers maybe just provide a function doing that and do not expose complete API?
[...]
I'll continue reading the patch-set tomorrow...
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH bpf-next/net v1 07/13] bpf: Introduce per-prog stdout/stderr streams
Quoted 39 lines · Eduard Zingerman
Quoted 25 lines · Kumar Kartikeya Dwivedi
Introduce a set of kfuncs to implement per BPF program stdout and stderr streams. This is implemented as a linked list of dynamically allocated strings whenever a message is printed. This can be done by the kernel or the program itself using the
bpf_prog_stream_vprintkkfunc. In-kernel wrappers are provided over streams to ease the process.The idea is that everytime messages need to be dumped, the reader would pull out the whole batch of messages from the stream at once, and then pop each string one by one and start printing it out (how exactly is left up to the BPF program reading the log, but usually it will be streaming data back into a ring buffer that is consumed by user space).
The use of a lockless list warrants that we be careful about the ordering of messages. When being added, the order maintained is new to old, therefore after deletion, we must reverse the list before iterating and popping out elements from it.
Overall, this infrastructure provides NMI-safe any context printing built on top of the NMI-safe any context
bpf_mem_alloc()interface.Later patches will add support for printing splats in case of BPF arena page faults, rqspinlock deadlocks, and
cond_breaktimeouts.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
[...]
Quoted 5 lines
kernel/bpf/stream.c new file +0
-0[...]
Quoted 3 lines
Quoted 1 line
+static int
bpf_stream_page_check_room(struct bpf_stream_page *stream_page, int len)^^^^^^^
len is never used
Yeah, I will get rid of the parameter.
Quoted 31 lines
Quoted 10 lines
kernel/bpf/stream.c new file +10
-0+{ + int min = offsetof(struct bpf_stream_elem, str[0]); +int consumed = stream_page->consumed; +int total =BPF_STREAM_PAGE_SZ ; +int rem =max (0 , total - consumed - min); + +/* Let's give room of at least 8 bytes. */ +WARN_ON_ONCE (rem %8 !=0 ); +return rem <8 ?0 : rem; +}[...]
Quoted 18 lines
kernel/bpf/stream.c new file +18
-0+static struct bpf_stream_elem *bpf_stream_elem_alloc (int len) +{ +const int max_len =ARRAY_SIZE ((struct bpf_bprintf_buffers ){}.buf); +struct bpf_stream_elem *elem; + +/* +* We may overflow, but we should never need more than one page size +* worth of memory. This can be lifted, but we'd need to adjust the +* other code to keep allocating more pages to overflow messages. +*/ +BUILD_BUG_ON (max_len >BPF_STREAM_PAGE_SZ ); +/* +* Length denotes the amount of data to be written as part of stream element, +* thus includes '\0' byte. We're capped by how much bpf_bprintf_buffers can +* accomodate, therefore deny allocations that won't fit into them. +*/ +if (len <0 || len > max_len) +return NULL ;
Quoted 8 lines
kernel/bpf/stream.c line 241
Quoted 4 lines
kernel/bpf/stream.c new file +4
-0+ + elem =bpf_stream_elem_alloc_from_bpf_ma (len); +if (!elem) + elem =bpf_stream_elem_alloc_from_stream_page (len);So, the stream page is a backup mechanism, right?
I'm curious, did you compare how many messages are dropped if there is no such backup?
You can try disabling the backup allocation path. You will notice that the selftest starts failing.
When IRQs are disabled (so arena page fault path or rqspinlock with _irqsave), we won't be
able to refill bpf_mem_alloc() cache, and will end up dropping the message on the floor.
Quoted 3 lines
Also, how much memory is wasted if there is no
"spillover" mechanism (BPF_STREAM_ELEM_F_NEXT).
Are these complications absolutely necessary?
There's no way to know.
It depends on the free space and the message size, right. Say we have 128 bytes left in the page, the message spans 156 bytes, is it better to write 128 bytes and move the rest to the new page?
Say we have 100K messages sitting in the log that followed this pattern, let's take wasted space on average per message as 32 bytes, then that's 3.2 MB of unused memory.
Quoted 12 lines
kernel/bpf/stream.c line 248
Quoted 8 lines
kernel/bpf/stream.c new file +7
-0+return elem; +} + +__bpf_kfunc_start_defs (); + +static int bpf_stream_push_str (struct bpf_stream *stream,const char *str,int len) +{This function accumulates elements in
&stream->logw/o a cap. Why is this not a problem if e.g. user space never flushes streams for the program?
It is certainly a problem, hence this bit in cover letter:
Quoted 1 line
* Enforcing memory consumption limits on the log usage, so that
in-flight log memory is bounded at runtime.
For now I will do a capacity counter per log, but maybe we want to tie this to memcg? I am not sure we can charge it in any context, but was hoping to get more opinions.
Quoted 35 lines
Quoted 32 lines
kernel/bpf/stream.c new file +31
-0+struct bpf_stream_elem *elem, *next =NULL ; +int room =0 , rem =0 ; + +/* +* Allocate a bpf_prog_stream_elem and push it to the bpf_prog_stream +* log, elements will be popped at once and reversed to print the log. +*/ + elem =bpf_stream_elem_alloc (len); +if (!elem) +return -ENOMEM ; + room = elem->mem_slice.len; +if (elem->flags &BPF_STREAM_ELEM_F_NEXT ) { + next = (struct bpf_stream_elem *)((unsigned long )elem->next & ~BPF_STREAM_ELEM_F_MASK ); + rem = next->mem_slice.len; + } + +memcpy (elem->str, str, room); +if (next) +memcpy (next->str, str + room, rem); + +if (next) { + elem->node.next = &next->node; + next->node.next =NULL ; + +llist_add_batch (&elem->node, &next->node, &stream->log); + }else { +llist_add (&elem->node, &stream->log); + } + +return 0 ; +}[...]
Quoted 12 lines
kernel/bpf/stream.c line 446
Quoted 8 lines
kernel/bpf/stream.c new file +8
-0+BTF_KFUNCS_START(stream_consumer_kfunc_set) +BTF_ID_FLAGS(func, bpf_stream_next_elem_batch, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_stream_free_elem_batch, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_stream_next_elem, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_stream_free_elem, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_prog_stream_get, KF_ACQUIRE | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_prog_stream_put, KF_RELEASE) +BTF_KFUNCS_END(stream_consumer_kfunc_set)This is a complicated API.
If we anticipate that users intend to write this info to ring buffers maybe just provide a function doing that and do not expose complete API?
I don't think anyone will use these functions directly, though they can if they want to.
It's meant to be hidden behind bpftool, and macros to print stuff like bpf_printk().
We cannot pop one message at a time, since they are not in FIFO order. So we need to splice out the whole batch queued and reverse it, before popping things.
It's a consequence of using lockless lists.
The other option is using a lock to protect the list, but using rqspinlock to then report messages about rqspinlock sounds like a circular dependency.
Quoted 4 lines
[...]
I'll continue reading the patch-set tomorrow...
Re: [RFC PATCH bpf-next/net v1 07/13] bpf: Introduce per-prog stdout/stderr streams
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
kernel/bpf/stream.c new file +4
-0+BTF_KFUNCS_START(stream_consumer_kfunc_set) +BTF_ID_FLAGS(func, bpf_stream_next_elem_batch, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_stream_free_elem_batch, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_stream_next_elem, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)Quoted 8 lines
kernel/bpf/stream.c line 446
Quoted 4 lines
kernel/bpf/stream.c new file +4
-0+BTF_ID_FLAGS(func, bpf_stream_free_elem, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_prog_stream_get, KF_ACQUIRE | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_prog_stream_put, KF_RELEASE) +BTF_KFUNCS_END (stream_consumer_kfunc_set)This is a complicated API.
If we anticipate that users intend to write this info to ring buffers maybe just provide a function doing that and do not expose complete API?
Quoted 4 lines
I don't think anyone will use these functions directly, though they can if they want to.
It's meant to be hidden behind bpftool, and macros to print stuff likebpf_printk().
I have to agree with Eduard here.
The api feels overdesigned. I don't see how it can be reused anywhere outside of bpftool.
Instead of introducing mem_slice concept and above kfuncs, let's have one special kfunc that will take prog_id, stream_id and will move things into dynptr returning EAGAIN/ENOENT when necessary. EFAULT shouldn't be possible when the whole
SEC("syscall")
int bpftool_dump_prog_stream(void *ctx) {..}
will be one kfunc.bpf_stream_dtor_ids won't be needed either.
Hard coding such a big logic into one kfunc is not pretty, and smaller kfuncs/building blocks would be preferred any day, but these stream_consumer_kfunc_set just don't look reusable.
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH bpf-next/net v1 07/13] bpf: Introduce per-prog stdout/stderr streams
Quoted 14 lines · Alexei Starovoitov
Quoted 14 lines
Quoted 13 lines
kernel/bpf/stream.c line 446
Quoted 9 lines
kernel/bpf/stream.c new file +8
-0+BTF_KFUNCS_START(stream_consumer_kfunc_set) +BTF_ID_FLAGS(func, bpf_stream_next_elem_batch, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_stream_free_elem_batch, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_stream_next_elem, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_stream_free_elem, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_prog_stream_get, KF_ACQUIRE | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_prog_stream_put, KF_RELEASE) +BTF_KFUNCS_END(stream_consumer_kfunc_set)This is a complicated API.
If we anticipate that users intend to write this info to ring buffers maybe just provide a function doing that and do not expose complete API?
Quoted 8 lines
Quoted 4 lines
I don't think anyone will use these functions directly, though they can if they want to.
It's meant to be hidden behind bpftool, and macros to print stuff likebpf_printk().I have to agree with Eduard here.
The api feels overdesigned. I don't see how it can be reused anywhere outside of bpftool.
Can you explain why?
Apart from syscall prog requirement which is easy to lift (but didn't bother for RFC).
Quoted 11 lines
Instead of introducing
mem_sliceconcept and above kfuncs, let's have one special kfunc that will takeprog_id,stream_idand will move things into dynptr returning EAGAIN/ENOENT when necessary. EFAULT shouldn't be possible when the whole
SEC("syscall")
intbpftool_dump_prog_stream(void *ctx){..}
will be one kfunc.bpf_stream_dtor_idswon't be needed either.
Hard coding such a big logic into one kfunc is not pretty, and smaller kfuncs/building blocks would be preferred any day, but thesestream_consumer_kfunc_setjust don't look reusable.
I don't follow. How is "shove all messages into ringbuf" more reusable? I can sympathize with the complexity argument Eduard made, ideally we'd have one pop() operation to pull out things.
We can discuss how to make things simpler.
With some documentation, it is not hard to follow:
Everytime you consume something from a stream, you splice out a batch of N messages, you can then splice out each message individually from a batch.
Both the batch and individual elements need to be freed. You can keep processing the batch until it's finished, but the better idea is to pace oneself and do N at a time.
In user space, you usually fill up your buffer's length (say PAGE_SIZE) when reading something, then delimit and parse and draw message boundaries when parsing.
You can flip the API in this patch around and use it to stream things into the program.
We just need to add one more element in prog->aux->stream[BPF_STDIN = 0]. Even without that, you can make it work on stdout. The writing is bidirectional.
User space side can obtain the stream of a prog and printk to it. Kernel side can pop and parse messages.
Probably not a bright idea to do so when in the fast path of the kernel to read strings, but in other cases, why not?
It can certainly make sense for self contained programs that want to consume options from the cmdline generically.
The generic loader can just forward the string when invoked in user space into the stream, and call some init/main function for the program before attaching it.
Your BPF init/main gets either raw stream, or prepared argv, argc blocks that the loader's infra can set up for you.
At that point all logic in the program can be self-contained, you can probably just write everything in BPF C, including setting config options etc.
There would be no need to put these strings into a ringbuf, it would just be for the BPF side to parse and make sense of.
But anyway, I will not push back too much. If everyone thinks "stream data to ringbuf kfunc" is the way to go, I'll make the change in v2.
Re: [RFC PATCH bpf-next/net v1 07/13] bpf: Introduce per-prog stdout/stderr streams
Quoted 22 lines · Kumar Kartikeya Dwivedi
Quoted 19 lines · Alexei Starovoitov
Quoted 18 lines
Quoted 13 lines
kernel/bpf/stream.c line 446
Quoted 9 lines
kernel/bpf/stream.c new file +8
-0+BTF_KFUNCS_START(stream_consumer_kfunc_set) +BTF_ID_FLAGS(func, bpf_stream_next_elem_batch, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_stream_free_elem_batch, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_stream_next_elem, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_stream_free_elem, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_prog_stream_get, KF_ACQUIRE | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_prog_stream_put, KF_RELEASE) +BTF_KFUNCS_END(stream_consumer_kfunc_set)This is a complicated API.
If we anticipate that users intend to write this info to ring buffers maybe just provide a function doing that and do not expose complete API?I don't think anyone will use these functions directly, though they can if they want to.
It's meant to be hidden behind bpftool, and macros to print stuff likebpf_printk().
Quoted 5 lines
Quoted 3 lines
I have to agree with Eduard here.
The api feels overdesigned. I don't see how it can be reused anywhere outside of bpftool.Can you explain why?
Because I don't see a second user for these bpf_stream_* fetchers. A special prog embedded in bpftool will be the only one afaict.
Quoted 15 lines
Apart from syscall prog requirement which is easy to lift (but didn't bother for RFC).
Quoted 11 lines
Instead of introducing
mem_sliceconcept and above kfuncs, let's have one special kfunc that will takeprog_id,stream_idand will move things into dynptr returning EAGAIN/ENOENT when necessary. EFAULT shouldn't be possible when the whole
SEC("syscall")
intbpftool_dump_prog_stream(void *ctx){..}
will be one kfunc.bpf_stream_dtor_idswon't be needed either.
Hard coding such a big logic into one kfunc is not pretty, and smaller kfuncs/building blocks would be preferred any day, but thesestream_consumer_kfunc_setjust don't look reusable.
Quoted 3 lines
I don't follow. How is "shove all messages into ringbuf" more reusable? I can sympathize with the complexity argument Eduard made, ideally we'd have one
pop()operation to pull out things.
That's exactly what I'm proposing. One pop-like kfunc with prog_id, stream_id arguments.
Quoted 9 lines
We can discuss how to make things simpler.
With some documentation, it is not hard to follow:
Everytime you consume something from a stream, you splice out a batch of N messages, you can then splice out each message individually from a batch.
Both the batch and individual elements need to be freed. You can keep processing the batch until it's finished, but the better idea is to pace oneself and do N at a time.
Sure. These details can be hidden in pop() kfunc.
Quoted 8 lines
In user space, you usually fill up your buffer's length (say
PAGE_SIZE) when reading something, then delimit and parse and draw message boundaries when parsing.You can flip the API in this patch around and use it to stream things into the program.
We just need to add one more element in prog->aux->stream[BPF_STDIN = 0]. Even without that, you can make it work on stdout. The writing is bidirectional.
Hmm. I don't see the writing part here.
Currently it only consumes llist.
There is bpf_stream_vprintk() to write into it from bpf prog. That part makes sense.bpf_stream_push_str() can also be a kfunc, but the operation here is _push_.
imo it makes sense to pair it with single _pop_, instead of next_elem, free_elem
Quoted 4 lines
User space side can obtain the stream of a prog and printk to it. Kernel side can pop and parse messages.
Probably not a bright idea to do so when in the fast path of the kernel to read strings, but in other cases, why not?
One bpf prog does bpf_stream_vprintk() while another pop-like. That makes sense.
The manual iterator over messages that requires kptr_xchg to be correct is what I see as overdesign.
Quoted 2 lines
But anyway, I will not push back too much. If everyone thinks "stream data to ringbuf kfunc" is the way to go, I'll make the change in v2.
That's not what I'm suggesting. stream is a concept makes sense. bpf_stream_vprintk() also makes sense, and it will work to push things from "user space bpf prog" into stdin of other bpf prog, and to print into stdout from that other bpf prog.
Eduard Zingerman
maintainerRe: [RFC PATCH bpf-next/net v1 07/13] bpf: Introduce per-prog stdout/stderr streams
In reply to Kumar Kartikeya DwivediKumar Kartikeya Dwivedi <[email protected]> writes:
[...]
Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
kernel/bpf/stream.c new file +4
-0+BTF_KFUNCS_START(stream_consumer_kfunc_set) +BTF_ID_FLAGS(func, bpf_stream_next_elem_batch, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_stream_free_elem_batch, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_stream_next_elem, KF_ACQUIRE | KF_RET_NULL | KF_TRUSTED_ARGS)Quoted 8 lines
kernel/bpf/stream.c line 446
Quoted 4 lines
kernel/bpf/stream.c new file +4
-0+BTF_ID_FLAGS(func, bpf_stream_free_elem, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_prog_stream_get, KF_ACQUIRE | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_prog_stream_put, KF_RELEASE) +BTF_KFUNCS_END (stream_consumer_kfunc_set)This is a complicated API.
If we anticipate that users intend to write this info to ring buffers maybe just provide a function doing that and do not expose complete API?I don't think anyone will use these functions directly, though they can if they want to.
It's meant to be hidden behind bpftool, and macros to print stuff likebpf_printk().
Quoted 8 lines
We cannot pop one message at a time, since they are not in FIFO order. So we need to splice out the whole batch queued and reverse it, before popping things.
It's a consequence of using lockless lists.The other option is using a lock to protect the list, but using rqspinlock to then report messages about rqspinlock sounds like a circular dependency.
The API exposes 6 kfuncs and 3 data structures.
If things are exposed these things would be used, I wouldn't assume that bpftool would be the only user. I'm sure people would craft their own monitoring solutions.
Imo, exposing 9 entities is a lot. What I don't like about it, is that API is not abstract:
- user cares about getting program log to some buffer, stream or file;
- user does not care how the strings are split internally in order to make logging efficient.
If at some point we decide to change implementation this API would be hard to preserve.
Hence I suggest to instead provide a set of kfuncs that would drain message queue into user provided ringbuf or buffer, w/o exposing details.
A question: why exposing this functionality as kfuncs and not BPF syscall commands?
Kumar Kartikeya Dwivedi
maintainerRFC PATCH 8/13 bpf: Add dump_stack() analogue to print to BPF stderr
In reply to Kumar Kartikeya DwivediIntroduce a kernel function which is the analogue of dump_stack() printing some useful information and the stack trace. This is not exposed to BPF programs yet, but can be made available in the future.
When we have a PC for a BPF program in the stack trace, also additionally output the filename and line number to make the trace helpful. The rest of the trace can be passed into ./decode_stacktrace.sh to obtain the line numbers for kernel symbols.
--- include/linux/bpf.h | 2 + kernel/bpf/stream.c | 99 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+)
include/linux/bpf.h +2 -0
@@ -3588,6 +3588,8 @@ void bpf_prog_stream_init(struct bpf_prog *prog);void bpf_prog_stream_free (struct bpf_prog *prog); __printf(2, 3)int bpf_prog_stderr_printk (struct bpf_prog *prog,const char *fmt, ...); +struct bpf_prog *bpf_prog_find_from_stack (void ); +void bpf_prog_stderr_dump_stack (struct bpf_prog *prog);#ifdef CONFIG_BPF_LSM void bpf_cgroup_atype_get (u32 attach_btf_id,int cgroup_atype);
kernel/bpf/stream.c +99 -0
108 lines not on this page · raw message
Signature · 1 line
2.47.1
Re: [RFC PATCH bpf-next/net v1 08/13] bpf: Add dump_stack() analogue to print to BPF stderr
Quoted 11 lines · Kumar Kartikeya Dwivedi
Introduce a kernel function which is the analogue of
dump_stack()printing some useful information and the stack trace. This is not exposed to BPF programs yet, but can be made available in the future.When we have a PC for a BPF program in the stack trace, also additionally output the filename and line number to make the trace helpful. The rest of the trace can be passed into ./decode_stacktrace.sh to obtain the line numbers for kernel symbols.
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
Seems to work as advertised.
[...]
Quoted 24 lines
kernel/bpf/stream.c +24
-0+static int dump_stack_bpf_linfo (u64 ip,const char **filep) +{ +struct bpf_prog *prog =bpf_prog_ksym_find (ip); +int idx =-1 , insn_start, insn_end, len; +struct bpf_line_info *linfo; +void **jited_linfo; +struct btf *btf; + + btf = prog->aux->btf; + linfo = prog->aux->linfo; + jited_linfo = prog->aux->jited_linfo; + +if (!btf || !linfo || !prog->aux->jited_linfo) +return -1 ; + len = prog->aux->func ? prog->aux->func[prog->aux->func_idx]->len : prog->len; + + linfo = &prog->aux->linfo[prog->aux->linfo_idx]; + jited_linfo = &prog->aux->jited_linfo[prog->aux->linfo_idx]; + + insn_start = linfo[0 ].insn_off; + insn_end = insn_start + len; + +for (int i =0 ; linfo[i].insn_off >= insn_start && linfo[i].insn_off < insn_end; i++) { +if (jited_linfo[i] >= (void *)ip)
kernel/bpf/stream.c int bpf_prog_stderr_printk(struct bpf_prog *prog, const char *fmt, ...) · line 562
Quoted 10 lines
kernel/bpf/stream.c +10
-0+break ; + idx = i; + } + +if (idx ==-1 ) +return -1 ; + + *filep =btf_name_by_offset (btf, linfo[idx].file_name_off); + *filep =strrchr (*filep,'/' ) ?: *filep; + *filep +=1 ;
Nit: += 1 not needed if '/' was not found?
Quoted 3 lines
kernel/bpf/stream.c +3
-0+ +return BPF_LINE_INFO_LINE_NUM (linfo[idx].line_col); +}
[...]
Kumar Kartikeya Dwivedi
maintainerRFC PATCH 9/13 bpf: Report may_goto timeout to BPF stderr
In reply to Kumar Kartikeya DwivediBegin reporting may_goto timeouts to BPF program's stderr stream. Also, pace the reporting frequency by printing at most 64 instances of any errors, so that we don't end up stressing the allocator and exhausting memory in case of misbehaving programs.
--- include/linux/bpf.h | 3 +++ kernel/bpf/core.c | 17 ++++++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-)
include/linux/bpf.h +3 -0
@@ -1662,6 +1662,7 @@ struct bpf_prog_aux {struct rcu_head rcu; };struct bpf_stream stream[2 ]; +atomic_long_t error_count; };struct bpf_prog {
@@ -3584,6 +3585,8 @@ void bpf_bprintf_cleanup(struct bpf_bprintf_data *data);int bpf_try_get_buffers (struct bpf_bprintf_buffers **bufs);void bpf_put_buffers (void ); +#define BPF_PROG_ERROR_COUNT_MAX 64 +void bpf_prog_stream_init (struct bpf_prog *prog);void bpf_prog_stream_free (struct bpf_prog *prog);__printf (2 ,3 )
kernel/bpf/core.c +16 -1
@@ -3151,6 +3151,19 @@ u64 __weak arch_bpf_timed_may_goto(void)return 0 ; } +static noinline void bpf_prog_report_may_goto_violation(void) +{ +struct bpf_prog *prog; + + prog =bpf_prog_find_from_stack (); +if (!prog) +return ; +if (atomic_long_fetch_add (1 , &prog->aux->error_count) >=BPF_PROG_ERROR_COUNT_MAX ) +return ; +bpf_prog_stderr_printk (prog,"ERROR: Timeout detected for may_goto instruction\n" ); +bpf_prog_stderr_dump_stack (prog); +} +u64 bpf_check_timed_may_goto (struct bpf_timed_may_goto *p) {u64 time =ktime_get_mono_fast_ns ();
@@ -3161,8 +3174,10 @@ u64 bpf_check_timed_may_goto(struct bpf_timed_may_goto *p)return BPF_MAX_TIMED_LOOPS ; }/* Check if we've exhausted our time slice, and zero count. */ -if (time - p->timestamp >= (NSEC_PER_SEC /4 )) +if (unlikely (time - p->timestamp >= (NSEC_PER_SEC /4 ))) { +bpf_prog_report_may_goto_violation ();return 0 ; + }/* Refresh the count for the stack frame. */ return BPF_MAX_TIMED_LOOPS ; }
Signature · 1 line
2.47.1
Kumar Kartikeya Dwivedi
maintainerRFC PATCH 10/13 bpf: Report rqspinlock deadlocks/timeout to BPF stderr
In reply to Kumar Kartikeya DwivediBegin reporting rqspinlock deadlocks and timeout to BPF program's stderr.
--- kernel/bpf/rqspinlock.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
kernel/bpf/rqspinlock.c +22 -0
@@ -666,6 +666,26 @@ EXPORT_SYMBOL_GPL(resilient_queued_spin_lock_slowpath);__bpf_kfunc_start_defs (); +static void bpf_prog_report_rqspinlock_violation (const char *str,void *lock,bool irqsave) +{ +struct rqspinlock_held *rqh =this_cpu_ptr (&rqspinlock_held_locks); +struct bpf_prog *prog; + + prog =bpf_prog_find_from_stack (); +if (!prog) +return ; +if (atomic_long_fetch_add (1 , &prog->aux->error_count) >=BPF_PROG_ERROR_COUNT_MAX ) +return ; +bpf_prog_stderr_printk (prog,"ERROR: %s for bpf_res_spin_lock%s\n" , str, irqsave ?"_irqsave" :"" ); +bpf_prog_stderr_printk (prog,"Attempted lock = 0x%px\n" , lock); +bpf_prog_stderr_printk (prog,"Total held locks = %d\n" , rqh->cnt); +for (int i =0 ; i <min (RES_NR_HELD , rqh->cnt); i++) +bpf_prog_stderr_printk (prog,"Held lock[%2d] = 0x%px\n" , i, rqh->locks[i]); +bpf_prog_stderr_dump_stack (prog); +} + +#define REPORT_STR (ret) ({ (ret) == -ETIMEDOUT ? "Timeout detected" : "AA or ABBA deadlock detected"; }) + __bpf_kfunc int bpf_res_spin_lock(struct bpf_res_spin_lock *lock) {int ret;
@@ -676,6 +696,7 @@ __bpf_kfunc int bpf_res_spin_lock(struct bpf_res_spin_lock *lock)preempt_disable (); ret =res_spin_lock ((rqspinlock_t *)lock);if (unlikely (ret)) { +bpf_prog_report_rqspinlock_violation (REPORT_STR (ret), lock, false);preempt_enable ();return ret; }
@@ -698,6 +719,7 @@ __bpf_kfunc int bpf_res_spin_lock_irqsave(struct bpf_res_spin_lock *lock, unsignlocal_irq_save (flags); ret =res_spin_lock ((rqspinlock_t *)lock);if (unlikely (ret)) { +bpf_prog_report_rqspinlock_violation (REPORT_STR (ret), lock, true);local_irq_restore (flags);preempt_enable ();return ret;
Signature · 1 line
2.47.1
Kumar Kartikeya Dwivedi
maintainerRFC PATCH 11/13 bpf: Report arena faults to BPF stderr
In reply to Kumar Kartikeya DwivediBegin reporting arena page faults and the faulting address to BPF program's stderr, for now limited to x86, but arm64 support should be easy to add.
--- arch/x86/net/bpf_jit_comp.c | 21 ++++++++++++++++++--- include/linux/bpf.h | 1 + kernel/bpf/arena.c | 14 ++++++++++++++ 3 files changed, 33 insertions(+), 3 deletions(-)
arch/x86/net/bpf_jit_comp.c +18 -3
@@ -1384,15 +1384,27 @@ static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size, }#define DONT_CLEAR 1 +#define ARENA_FAULT (1 << 8)bool ex_handler_bpf (const struct exception_table_entry *x,struct pt_regs *regs) { -u32 reg = x->fixup >>8 ; +u32 arena_reg = (x->fixup >>8 ) &0xff ; +bool is_arena = !!arena_reg; +u32 reg = x->fixup >>16 ; +unsigned long addr; + +/* Read here, if src_reg is dst_reg for load, we'll write 0 to it. */ +if (is_arena) + addr = *(unsigned long *)((void *)regs + arena_reg);/* jump over faulting load and clear dest register */ if (reg !=DONT_CLEAR ) *(unsigned long *)((void *)regs + reg) =0 ; regs->ip += x->fixup &0xff ; + +if (is_arena) +bpf_prog_report_arena_violation (reg ==DONT_CLEAR , addr); +return true; }
@@ -2043,7 +2055,10 @@ st: if (is_imm8(insn->off)) ex->data =EX_TYPE_BPF ; ex->fixup = (prog - start_of_ldx) | - ((BPF_CLASS (insn->code) ==BPF_LDX ? reg2pt_regs[dst_reg] :DONT_CLEAR ) <<8 ); + ((BPF_CLASS (insn->code) ==BPF_LDX ? reg2pt_regs[dst_reg] :DONT_CLEAR ) <<16 ) + | ((BPF_CLASS (insn->code) ==BPF_LDX ? reg2pt_regs[src_reg] : reg2pt_regs[dst_reg])<<8 ); +/* Ensure src_reg offset fits in 1 byte. */ +BUILD_BUG_ON (sizeof (struct pt_regs ) >U8_MAX ); }break ;
@@ -2161,7 +2176,7 @@ st: if (is_imm8(insn->off))* End result: x86 insn "mov rbx, qword ptr [rax+0x14]" * of 4 bytes will be ignored and rbx will be zero inited. */ - ex->fixup = (prog - start_of_ldx) | (reg2pt_regs[dst_reg] <<8 ); + ex->fixup = (prog - start_of_ldx) | (reg2pt_regs[dst_reg] <<16 ); }break ;
include/linux/bpf.h +1 -0
@@ -3593,6 +3593,7 @@ __printf(2, 3)int bpf_prog_stderr_printk (struct bpf_prog *prog,const char *fmt, ...);struct bpf_prog *bpf_prog_find_from_stack (void );void bpf_prog_stderr_dump_stack (struct bpf_prog *prog); +void bpf_prog_report_arena_violation (bool write,unsigned long addr);#ifdef CONFIG_BPF_LSM void bpf_cgroup_atype_get (u32 attach_btf_id,int cgroup_atype);
kernel/bpf/arena.c +14 -0
@@ -590,3 +590,17 @@ static int __init kfunc_init(void)return register_btf_kfunc_id_set (BPF_PROG_TYPE_UNSPEC , &common_kfunc_set); }late_initcall (kfunc_init); + +void bpf_prog_report_arena_violation (bool write,unsigned long addr) +{ +struct bpf_prog *prog; + + prog =bpf_prog_find_from_stack (); +if (!prog) +return ; +if (atomic_long_fetch_add (1 , &prog->aux->error_count) >=BPF_PROG_ERROR_COUNT_MAX ) +return ; +bpf_prog_stderr_printk (prog,"ERROR: Arena %s access at unmapped address 0x%lx\n" , + write ?"WRITE" :"READ" , addr); +bpf_prog_stderr_dump_stack (prog); +}
Signature · 1 line
2.47.1
Kumar Kartikeya Dwivedi
maintainerRFC PATCH 12/13 bpftool: Add support for dumping streams
In reply to Kumar Kartikeya DwivediAdd bpftool support for dumping streams of a given BPF program.
TODO: JSON and filepath support.
--- tools/bpf/bpftool/Makefile | 2 +- tools/bpf/bpftool/prog.c | 71 +++++++++++++++++- tools/bpf/bpftool/skeleton/stream.bpf.c | 96 +++++++++++++++++++++++++ 3 files changed, 166 insertions(+), 3 deletions(-) create mode 100644 tools/bpf/bpftool/skeleton/stream.bpf.c
tools/bpf/bpftool/Makefile +1 -1
@@ -234,7 +234,7 @@ $(OUTPUT)%.bpf.o: skeleton/%.bpf.c $(OUTPUT)vmlinux.h $(LIBBPF_BOOTSTRAP) $(OUTPUT)%.skel.h: $(OUTPUT)%.bpf.o $(BPFTOOL_BOOTSTRAP) $(QUIET_GEN)$(BPFTOOL_BOOTSTRAP) gen skeleton $< > $@ -$(OUTPUT)prog.o: $(OUTPUT)profiler.skel.h +$(OUTPUT)prog.o: $(OUTPUT)profiler.skel.h $(OUTPUT)stream.skel.h $(OUTPUT)pids.o: $(OUTPUT)pid_iter.skel.h
tools/bpf/bpftool/prog.c +69 -2
120 lines not on this page · raw message
tools/bpf/bpftool/skeleton/stream.bpf.c new file +96 -0
96 lines not on this page · raw message
Signature · 1 line
2.47.1
Eduard Zingerman
maintainerRe: [RFC PATCH bpf-next/net v1 12/13] bpftool: Add support for dumping streams
When I add a test as at the bottom of the email I get the following output:
test_stream_cond_break:PASS:load 0 nsec
test_stream_cond_break:PASS:run_opts 0 nsec
test_stream_cond_break:PASS:info_by_fd 0 nsec
ERROR: Timeout detected for may_goto instruction
CPU: 6 UID: 0 PID: 206 Comm: test_progs
Call trace:
bpf_prog_stderr_dump_stack+0xde/0x160
bpf_check_timed_may_goto+0x5d/0x90
arch_bpf_timed_may_goto+0x21/0x40
bpf_prog_34056decf3b2fb2f_long_loop+0x49/0x57: [stream_cond_break.c:8]
bpf_prog_run_pin_on_cpu+0x5f/0x110
bpf_prog_test_run_syscall+0x205/0x320
bpf_prog_test_run+0x234/0x2a0
__sys_bpf+0x2d7/0x570
__x64_sys_bpf+0x7c/0x90
do_syscall_64+0x79/0x120
entry_SYSCALL_64_after_hwframe+0x76/0x7e
prog_dump_stream: ret==-22
test_stream_cond_break:FAIL:./tools/sbin/bpftool prog dump stderr id 8
unexpected error: 59904 (errno 95)Am I doing something wrong or EINVAL should not be returned from prog_dump_stream in this case?
---
tools/bpf/bpftool/prog.c +1 -0
@@ -747,6 +747,7 @@ prog_dump_stream(struct bpf_prog_info *info, enum dump_mode mode, const char *fi ret = -EINVAL ; end:stream_bpf__destroy (skel); +fprintf (stderr,"prog_dump_stream: ret==%d\n" , ret);return ret; }
tools/testing/selftests/bpf/prog_tests/stream_cond_break.c new file +42 -0
42 lines not on this page · raw message
tools/testing/selftests/bpf/progs/stream_cond_break.c new file +12 -0
@@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "bpf_experimental.h" + +SEC("syscall") +int long_loop (const void *ctx) +{ +for (;;) + cond_break; +return 0 ; +} + +char _license[] SEC("license") = "GPL";
Quentin Monnet
maintainerRe: [RFC PATCH bpf-next/net v1 12/13] bpftool: Add support for dumping streams
In reply to Kumar Kartikeya Dwivedi2025-04-14 09:14 UTC-0700 ~ Kumar Kartikeya Dwivedi <[email protected]>
Quoted 1 line
Add bpftool support for dumping streams of a given BPF program.
Thanks for adding bpftool support!
Quoted 1 line
TODO: JSON and filepath support.
... plus documentation (man page), and bash completion please.
Quoted 144 lines
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- tools/bpf/bpftool/Makefile | 2 +- tools/bpf/bpftool/prog.c | 71 +++++++++++++++++- tools/bpf/bpftool/skeleton/stream.bpf.c | 96 +++++++++++++++++++++++++ 3 files changed, 166 insertions(+), 3 deletions(-) create mode 100644 tools/bpf/bpftool/skeleton/stream.bpf.ctools/bpf/bpftool/Makefile +1
-1@@ -234,7 +234,7 @@ $(OUTPUT)%.bpf.o: skeleton/%.bpf.c $(OUTPUT)vmlinux.h $(LIBBPF_BOOTSTRAP) $(OUTPUT)%.skel.h: $(OUTPUT)%.bpf.o $(BPFTOOL_BOOTSTRAP) $(QUIET_GEN)$(BPFTOOL_BOOTSTRAP) gen skeleton $< > $@ -$(OUTPUT)prog.o: $(OUTPUT)profiler.skel.h +$(OUTPUT)prog.o: $(OUTPUT)profiler.skel.h $(OUTPUT)stream.skel.h $(OUTPUT)pids.o: $(OUTPUT)pid_iter.skel.htools/bpf/bpftool/prog.c +67
-2@@ -35,12 +35,16 @@#include "main.h" #include "xlated_dumper.h" +#include "stream.skel.h" +#define BPF_METADATA_PREFIX "bpf_metadata_"#define BPF_METADATA_PREFIX_LEN (sizeof(BPF_METADATA_PREFIX) - 1)enum dump_mode {DUMP_JITED ,DUMP_XLATED , +DUMP_STDOUT , +DUMP_STDERR , };static const bool attach_types[] = {@@ -697,6 +701,55 @@ static int do_show(int argc, char **argv)return err; } +static int process_stream_sample (void *ctx,void *data,size_t len) +{ +FILE *file = ctx; + +fprintf (file,"%s" , (char *)data); +fflush (file); +return 0 ; +} + +static int +prog_dump_stream (struct bpf_prog_info *info,enum dump_mode mode,const char *filepath) +{ +FILE *file = mode ==DUMP_STDOUT ? stdout : stderr; +LIBBPF_OPTS (bpf_test_run_opts, opts); +struct ring_buffer *ringbuf; +struct stream_bpf *skel; +int map_fd, ret =-1 ; + +__u32 prog_id = info->id; +__u32 stream_id = mode ==DUMP_STDOUT ?1 :2 ; + + skel =stream_bpf__open_and_load (); +if (!skel) +return -errno; + skel->bss->prog_id = prog_id; + skel->bss->stream_id = stream_id; + +//TODO(kkd): Filepath handling + map_fd =bpf_map__fd (skel->maps.ringbuf); + ringbuf =ring_buffer__new (map_fd, process_stream_sample, file,NULL ); +if (!ringbuf) { + ret = -errno; + goto end; + } +do { + skel->bss->written_count = skel->bss->written_size =0 ; + ret =bpf_prog_test_run_opts (bpf_program__fd (skel->progs.bpftool_dump_prog_stream), &opts); + ret = -EINVAL ; +if (ring_buffer__consume_n (ringbuf, skel->bss->written_count) != skel->bss->written_count) + goto end; + }while (!ret && opts.retval ==EAGAIN ); + +if (opts.retval !=0 ) + ret = -EINVAL ; +end: +stream_bpf__destroy (skel); +return ret; +} +static int prog_dump (struct bpf_prog_info *info,enum dump_mode mode,char *filepath,bool opcodes,bool visual,bool linum)@@ -719,13 +772,15 @@ prog_dump(struct bpf_prog_info *info, enum dump_mode mode, } buf =u64_to_ptr (info->jited_prog_insns); member_len = info->jited_prog_len; - }else {/* DUMP_XLATED */ + }else if (mode ==DUMP_XLATED ) {/* DUMP_XLATED */ if (info->xlated_prog_len ==0 || !info->xlated_prog_insns) {p_err ("error retrieving insn dump: kernel.kptr_restrict set?" );return -1 ; } buf =u64_to_ptr (info->xlated_prog_insns); member_len = info->xlated_prog_len; + }else if (mode ==DUMP_STDOUT || mode ==DUMP_STDERR ) { +return prog_dump_stream (info, mode, filepath); }if (info->btf_id) {@@ -898,8 +953,10 @@ static int do_dump(int argc, char **argv) mode =DUMP_JITED ; }else if (is_prefix (*argv,"xlated" )) { mode =DUMP_XLATED ; + }else if (is_prefix (*argv,"stdout" ) ||is_prefix (*argv,"stderr" )) { + mode =is_prefix (*argv,"stdout" ) ?DUMP_STDOUT :DUMP_STDERR ; }else { -p_err ("expected 'xlated' or 'jited', got: %s" , *argv); +p_err ("expected 'stdout', 'stderr', 'xlated' or 'jited', got: %s" , *argv);return -1 ; }NEXT_ARG ();@@ -950,6 +1007,14 @@ static int do_dump(int argc, char **argv) } } +if (mode ==DUMP_STDOUT || mode ==DUMP_STDERR ) { +if (opcodes || visual || linum) { +p_err ("'%s' is not compatible with 'opcodes', 'visual', or 'linum'" , + mode ==DUMP_STDOUT ?"stdout" :"stderr" ); + goto exit_close; + } + } +if (filepath && (opcodes || visual || linum)) {p_err ("'file' is not compatible with 'opcodes', 'visual', or 'linum'" ); goto exit_close;
tools/bpf/bpftool/prog.c static int do_help(int argc, char **argv) · line 2537
Quoted 5 lines
tools/bpf/bpftool/prog.c +2
-0@@ -2468,6 +2533,8 @@ static int do_help(int argc, char **argv)"Usage: %1$s %2$s { show | list } [PROG]\n" " %1$s %2$s dump xlated PROG [{ file FILE | [opcodes] [linum] [visual] }]\n" " %1$s %2$s dump jited PROG [{ file FILE | [opcodes] [linum] }]\n" +" %1$s %2$s dump stdout PROG [{ file FILE }]\n" +" %1$s %2$s dump stderr PROG [{ file FILE }]\n"
I'm not sure "prog dump" is the best subcommand for these features. The "dump" has been associated with printing the program itself, either translated or as JITed instructions. Here we display something else; it's closer to "prog tracelog", that we use to dump the trace pipe. And stdout/stderr are streams anyway, I'm not sure that "dumping" them is the most accurate term.
How about "prog trace (stdout|stderr)"? Bonus: you don't have to handle opcodes/linum/visual, and don't have to add support for "filepath".
tools/bpf/bpftool/skeleton/stream.bpf.c line 1
Quoted 10 lines
tools/bpf/bpftool/prog.c +0
-0" %1$s %2$s pin PROG FILE\n" " %1$s %2$s { load | loadall } OBJ PATH \\\n" " [type TYPE] [{ offload_dev | xdpmeta_dev } NAME] \\\n" tools/bpf/bpftool/skeleton/stream.bpf.c new file +1
-0@@ -0,0 +1,96 @@ +// SPDX-License-Identifier: GPL-2.0
Bpftool is dual-licensed (GPL-2.0-only OR BSD-2-Clause), please consider using the same here.
Quoted 31 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +31
-0+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> +#include <bpf/bpf_helpers.h> + +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 1024 * 1024); +} ringbuf SEC(".maps"); + +struct value { + struct bpf_stream_elem_batch __kptr *batch; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, int); + __type(value, struct value); + __uint(max_entries, 1); +} array SEC(".maps"); + +int written_size; +int written_count; +int stream_id; +int prog_id; + +#define ENOENT 2 +#define EAGAIN 11 +#define EFAULT 14 + +SEC ("syscall" )
tools/bpf/bpftool/skeleton/stream.bpf.c line 42
Quoted 10 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +10
-0+int bpftool_dump_prog_stream (void *ctx) +{ +struct bpf_stream_elem_batch *elem_batch; +struct bpf_stream_elem *elem; +struct bpf_stream *stream; +bool cont = false; +struct value *v; +bool ret =0 ; + + stream =bpf_prog_stream_get (BPF_STDERR , prog_id);
Calls to these new kfuncs will break compilation on older systems that don't support them yet (and don't have the definition in their vmlinux.h). We should provide fallback definitions to make sure that the program compiles.
Quoted 44 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +44
-0+if (!stream) +return ENOENT ; + + v =bpf_map_lookup_elem (&array, &(int ){0 }); + +if (v->batch) + elem_batch =bpf_kptr_xchg (&v->batch,NULL ); +else + elem_batch =bpf_stream_next_elem_batch (stream); +if (!elem_batch) + goto end; + +bpf_repeat (BPF_MAX_LOOPS ) { +struct bpf_dynptr dst_dptr, src_dptr; +int size; + + elem =bpf_stream_next_elem (elem_batch); +if (!elem) +break ; + size = elem->mem_slice.len; + +if (bpf_dynptr_from_mem_slice (&elem->mem_slice,0 , &src_dptr)) + ret =EFAULT ; +if (bpf_ringbuf_reserve_dynptr (&ringbuf, size,0 , &dst_dptr)) + ret =EFAULT ; +if (bpf_dynptr_copy (&dst_dptr,0 , &src_dptr,0 , size)) + ret =EFAULT ; +bpf_ringbuf_submit_dynptr (&dst_dptr,0 ); + + written_count++; + written_size += size; + +bpf_stream_free_elem (elem); + +/* Probe and exit if no more space, probe for twice the typical size.*/ +if (bpf_ringbuf_reserve_dynptr (&ringbuf,2048 ,0 , &dst_dptr)) + cont = true; +bpf_ringbuf_discard_dynptr (&dst_dptr,0 ); + +if (ret || cont) +break ; + } + +if (cont)
tools/bpf/bpftool/skeleton/stream.bpf.c line 96
Quoted 10 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +10
-0+ elem_batch =bpf_kptr_xchg (&v->batch, elem_batch); +if (elem_batch) +bpf_stream_free_elem_batch (elem_batch); +end: +bpf_prog_stream_put (stream); + +return ret ?: (cont ?EAGAIN :0 ); +} + +char _license[] SEC("license") = "GPL";
Same note on the license, other programs used by bpftool have license string "Dual BSD/GPL".
Kumar Kartikeya Dwivedi
maintainerRe: [RFC PATCH bpf-next/net v1 12/13] bpftool: Add support for dumping streams
Quoted 5 lines · Quentin Monnet
2025-04-14 09:14 UTC-0700 ~ Kumar Kartikeya Dwivedi <[email protected]>
Quoted 1 line
Add bpftool support for dumping streams of a given BPF program.
Quoted 8 lines
Thanks for adding bpftool support!
Quoted 1 line
TODO: JSON and filepath support.
... plus documentation (man page), and bash completion please.
Done.
Quoted 159 lines
Quoted 145 lines
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- tools/bpf/bpftool/Makefile | 2 +- tools/bpf/bpftool/prog.c | 71 +++++++++++++++++- tools/bpf/bpftool/skeleton/stream.bpf.c | 96 +++++++++++++++++++++++++ 3 files changed, 166 insertions(+), 3 deletions(-) create mode 100644 tools/bpf/bpftool/skeleton/stream.bpf.ctools/bpf/bpftool/Makefile +1
-1@@ -234,7 +234,7 @@ $(OUTPUT)%.bpf.o: skeleton/%.bpf.c $(OUTPUT)vmlinux.h $(LIBBPF_BOOTSTRAP) $(OUTPUT)%.skel.h: $(OUTPUT)%.bpf.o $(BPFTOOL_BOOTSTRAP) $(QUIET_GEN)$(BPFTOOL_BOOTSTRAP) gen skeleton $< > $@ -$(OUTPUT)prog.o: $(OUTPUT)profiler.skel.h +$(OUTPUT)prog.o: $(OUTPUT)profiler.skel.h $(OUTPUT)stream.skel.h $(OUTPUT)pids.o: $(OUTPUT)pid_iter.skel.htools/bpf/bpftool/prog.c +67
-2@@ -35,12 +35,16 @@#include "main.h" #include "xlated_dumper.h" +#include "stream.skel.h" +#define BPF_METADATA_PREFIX "bpf_metadata_"#define BPF_METADATA_PREFIX_LEN (sizeof(BPF_METADATA_PREFIX) - 1)enum dump_mode {DUMP_JITED ,DUMP_XLATED , +DUMP_STDOUT , +DUMP_STDERR , };static const bool attach_types[] = {@@ -697,6 +701,55 @@ static int do_show(int argc, char **argv)return err; } +static int process_stream_sample (void *ctx,void *data,size_t len) +{ +FILE *file = ctx; + +fprintf (file,"%s" , (char *)data); +fflush (file); +return 0 ; +} + +static int +prog_dump_stream (struct bpf_prog_info *info,enum dump_mode mode,const char *filepath) +{ +FILE *file = mode ==DUMP_STDOUT ? stdout : stderr; +LIBBPF_OPTS (bpf_test_run_opts, opts); +struct ring_buffer *ringbuf; +struct stream_bpf *skel; +int map_fd, ret =-1 ; + +__u32 prog_id = info->id; +__u32 stream_id = mode ==DUMP_STDOUT ?1 :2 ; + + skel =stream_bpf__open_and_load (); +if (!skel) +return -errno; + skel->bss->prog_id = prog_id; + skel->bss->stream_id = stream_id; + +//TODO(kkd): Filepath handling + map_fd =bpf_map__fd (skel->maps.ringbuf); + ringbuf =ring_buffer__new (map_fd, process_stream_sample, file,NULL ); +if (!ringbuf) { + ret = -errno; + goto end; + } +do { + skel->bss->written_count = skel->bss->written_size =0 ; + ret =bpf_prog_test_run_opts (bpf_program__fd (skel->progs.bpftool_dump_prog_stream), &opts); + ret = -EINVAL ; +if (ring_buffer__consume_n (ringbuf, skel->bss->written_count) != skel->bss->written_count) + goto end; + }while (!ret && opts.retval ==EAGAIN ); + +if (opts.retval !=0 ) + ret = -EINVAL ; +end: +stream_bpf__destroy (skel); +return ret; +} +static int prog_dump (struct bpf_prog_info *info,enum dump_mode mode,char *filepath,bool opcodes,bool visual,bool linum)@@ -719,13 +772,15 @@ prog_dump(struct bpf_prog_info *info, enum dump_mode mode, } buf =u64_to_ptr (info->jited_prog_insns); member_len = info->jited_prog_len; - }else {/* DUMP_XLATED */ + }else if (mode ==DUMP_XLATED ) {/* DUMP_XLATED */ if (info->xlated_prog_len ==0 || !info->xlated_prog_insns) {p_err ("error retrieving insn dump: kernel.kptr_restrict set?" );return -1 ; } buf =u64_to_ptr (info->xlated_prog_insns); member_len = info->xlated_prog_len; + }else if (mode ==DUMP_STDOUT || mode ==DUMP_STDERR ) { +return prog_dump_stream (info, mode, filepath); }if (info->btf_id) {@@ -898,8 +953,10 @@ static int do_dump(int argc, char **argv) mode =DUMP_JITED ; }else if (is_prefix (*argv,"xlated" )) { mode =DUMP_XLATED ; + }else if (is_prefix (*argv,"stdout" ) ||is_prefix (*argv,"stderr" )) { + mode =is_prefix (*argv,"stdout" ) ?DUMP_STDOUT :DUMP_STDERR ; }else { -p_err ("expected 'xlated' or 'jited', got: %s" , *argv); +p_err ("expected 'stdout', 'stderr', 'xlated' or 'jited', got: %s" , *argv);return -1 ; }NEXT_ARG ();@@ -950,6 +1007,14 @@ static int do_dump(int argc, char **argv) } } +if (mode ==DUMP_STDOUT || mode ==DUMP_STDERR ) { +if (opcodes || visual || linum) { +p_err ("'%s' is not compatible with 'opcodes', 'visual', or 'linum'" , + mode ==DUMP_STDOUT ?"stdout" :"stderr" ); + goto exit_close; + } + } +if (filepath && (opcodes || visual || linum)) {p_err ("'file' is not compatible with 'opcodes', 'visual', or 'linum'" ); goto exit_close;tools/bpf/bpftool/prog.c static int do_help(int argc, char **argv) · line 2537
Quoted 5 lines
tools/bpf/bpftool/prog.c +2
-0@@ -2468,6 +2533,8 @@ static int do_help(int argc, char **argv)"Usage: %1$s %2$s { show | list } [PROG]\n" " %1$s %2$s dump xlated PROG [{ file FILE | [opcodes] [linum] [visual] }]\n" " %1$s %2$s dump jited PROG [{ file FILE | [opcodes] [linum] }]\n" +" %1$s %2$s dump stdout PROG [{ file FILE }]\n" +" %1$s %2$s dump stderr PROG [{ file FILE }]\n" I'm not sure "prog dump" is the best subcommand for these features. The "dump" has been associated with printing the program itself, either translated or as JITed instructions. Here we display something else; it's closer to "prog tracelog", that we use to dump the trace pipe. And stdout/stderr are streams anyway, I'm not sure that "dumping" them is the most accurate term.
Quoted 2 lines
How about "prog trace (stdout|stderr)"? Bonus: you don't have to handle opcodes/linum/visual, and don't have to add support for "filepath".
I went with prog tracelog {stdout|stderr}, since trace probably feels a bit odd in this context.
Tracelog alone will print trace pipe, when passed stdout/stderr it gives the stream output.
Let me know what you think.
Quoted 16 lines
tools/bpf/bpftool/skeleton/stream.bpf.c line 1
Quoted 12 lines
tools/bpf/bpftool/prog.c +0
-0" %1$s %2$s pin PROG FILE\n" " %1$s %2$s { load | loadall } OBJ PATH \\\n" " [type TYPE] [{ offload_dev | xdpmeta_dev } NAME] \\\n" tools/bpf/bpftool/skeleton/stream.bpf.c new file +1
-0@@ -0,0 +1,96 @@ +// SPDX-License-Identifier: GPL-2.0 Bpftool is dual-licensed (GPL-2.0-only OR BSD-2-Clause), please consider using the same here.
Done.
Quoted 41 lines
Quoted 41 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +39
-0+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> +#include <bpf/bpf_helpers.h> + +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 1024 * 1024); +} ringbuf SEC(".maps"); + +struct value { + struct bpf_stream_elem_batch __kptr *batch; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, int); + __type(value, struct value); + __uint(max_entries, 1); +} array SEC(".maps"); + +int written_size; +int written_count; +int stream_id; +int prog_id; + +#define ENOENT 2 +#define EAGAIN 11 +#define EFAULT 14 + +SEC("syscall") +int bpftool_dump_prog_stream (void *ctx) +{ +struct bpf_stream_elem_batch *elem_batch; +struct bpf_stream_elem *elem; +struct bpf_stream *stream; +bool cont = false; +struct value *v; +bool ret =0 ;
Quoted 8 lines
tools/bpf/bpftool/skeleton/stream.bpf.c line 42
Quoted 2 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +2
-0+ + stream =bpf_prog_stream_get (BPF_STDERR , prog_id);Calls to these new kfuncs will break compilation on older systems that don't support them yet (and don't have the definition in their
vmlinux.h). We should provide fallback definitions to make sure that the program compiles.
This is the only thing I haven't yet addressed in v2, because it seemed a bit ugly.
I tried adding kfunc declarations, but those aren't enough.
We rely on structs introduced and read in this patch.
So I think vmlinux.h needs to be dropped, but it means adding a lot more than just the declarations, all types, plus any types they transitively depend on.
Maybe there is a better way (like detecting compilation failure and skipping?). But if not, I will address like above in v3.
Quoted 52 lines
Quoted 52 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +50
-0+if (!stream) +return ENOENT ; + + v =bpf_map_lookup_elem (&array, &(int ){0 }); + +if (v->batch) + elem_batch =bpf_kptr_xchg (&v->batch,NULL ); +else + elem_batch =bpf_stream_next_elem_batch (stream); +if (!elem_batch) + goto end; + +bpf_repeat (BPF_MAX_LOOPS ) { +struct bpf_dynptr dst_dptr, src_dptr; +int size; + + elem =bpf_stream_next_elem (elem_batch); +if (!elem) +break ; + size = elem->mem_slice.len; + +if (bpf_dynptr_from_mem_slice (&elem->mem_slice,0 , &src_dptr)) + ret =EFAULT ; +if (bpf_ringbuf_reserve_dynptr (&ringbuf, size,0 , &dst_dptr)) + ret =EFAULT ; +if (bpf_dynptr_copy (&dst_dptr,0 , &src_dptr,0 , size)) + ret =EFAULT ; +bpf_ringbuf_submit_dynptr (&dst_dptr,0 ); + + written_count++; + written_size += size; + +bpf_stream_free_elem (elem); + +/* Probe and exit if no more space, probe for twice the typical size.*/ +if (bpf_ringbuf_reserve_dynptr (&ringbuf,2048 ,0 , &dst_dptr)) + cont = true; +bpf_ringbuf_discard_dynptr (&dst_dptr,0 ); + +if (ret || cont) +break ; + } + +if (cont) + elem_batch =bpf_kptr_xchg (&v->batch, elem_batch); +if (elem_batch) +bpf_stream_free_elem_batch (elem_batch); +end: +bpf_prog_stream_put (stream); +
Quoted 8 lines
tools/bpf/bpftool/skeleton/stream.bpf.c line 96
Quoted 4 lines
tools/bpf/bpftool/skeleton/stream.bpf.c new file +4
-0+return ret ?: (cont ?EAGAIN :0 ); +} + +char _license[] SEC("license") = "GPL";Same note on the license, other programs used by bpftool have license string "Dual BSD/GPL".
Done, please take a look at the changes in v2.
Kumar Kartikeya Dwivedi
maintainerRFC PATCH 13/13 selftests/bpf: Add tests for prog streams
In reply to Kumar Kartikeya DwivediAdd selftests to stress test the various facets of the stream API, memory allocation pattern, and ensuring dumping support is tested and functional.
--- .../testing/selftests/bpf/prog_tests/stream.c | 57 +++++++ .../testing/selftests/bpf/progs/dynptr_fail.c | 28 ++++ tools/testing/selftests/bpf/progs/stream.c | 150 ++++++++++++++++++ .../selftests/bpf/progs/stream_bpftool.c | 142 +++++++++++++++++ .../testing/selftests/bpf/progs/stream_fail.c | 38 +++++ 5 files changed, 415 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/stream.c create mode 100644 tools/testing/selftests/bpf/progs/stream.c create mode 100644 tools/testing/selftests/bpf/progs/stream_bpftool.c create mode 100644 tools/testing/selftests/bpf/progs/stream_fail.c
tools/testing/selftests/bpf/prog_tests/stream.c new file +57 -0
@@ -0,0 +1,57 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#include <test_progs.h> +#include <sys/mman.h> + +#include "stream.skel.h" +#include "stream_fail.skel.h" + +#include "stream_bpftool.skel.h" + +void test_stream_failure (void ) +{ +RUN_TESTS (stream_fail); +} + +void test_stream_success (void ) +{ +RUN_TESTS (stream); +RUN_TESTS (stream_bpftool); +return ; +} + +static int process_sample (void *ctx,void *data,size_t len) +{ +fprintf (stderr,"%s" , (char *)data); +return 0 ; +} + +void test_stream_ringbuf_output (void ) +{ +LIBBPF_OPTS (bpf_test_run_opts, opts); +struct ring_buffer *ringbuf; +struct stream_bpftool *skel; +int fd, ret; + + skel =stream_bpftool__open_and_load (); +if (!ASSERT_OK_PTR (skel,"stream_bpftool_open_and_load" )) +return ; + + fd =bpf_map__fd (skel->maps.ringbuf); + + ringbuf =ring_buffer__new (fd, process_sample,NULL ,NULL ); +if (!ASSERT_OK_PTR (ringbuf,"ringbuf_new" )) + goto end; + +do { + skel->bss->written_count = skel->bss->written_size =0 ; + ret =bpf_prog_test_run_opts (bpf_program__fd (skel->progs.stream_bpftool_dump_prog_stream), &opts); +ASSERT_EQ (ring_buffer__consume_n (ringbuf, skel->bss->written_count), skel->bss->written_count,"consume" ); + }while (!ret && opts.retval ==EAGAIN ); + +ASSERT_OK (ret,"ret" ); +ASSERT_EQ (opts.retval,0 ,"retval" ); + +end: +stream_bpftool__destroy (skel); +}
tools/testing/selftests/bpf/progs/dynptr_fail.c +28 -0
@@ -1734,3 +1734,31 @@ int test_dynptr_reg_type(void *ctx)global_call_bpf_dynptr ((const struct bpf_dynptr *)current);return 0 ; } + +SEC("?syscall") +__failure __msg("Expected an initialized dynptr as arg #2") +int test_dynptr_source_release_btf(void *ctx) +{ +struct bpf_stream_elem_batch *elem_batch; +struct bpf_stream_elem *elem; +struct bpf_stream *stream; +struct bpf_dynptr dptr; +char buf[8 ]; + + stream =bpf_stream_get (BPF_STDERR ,NULL ); +if (!stream) +return 0 ; + elem_batch =bpf_stream_next_elem_batch (stream); +if (!elem_batch) +return 0 ; + elem =bpf_stream_next_elem (elem_batch); +if (!elem) { +bpf_stream_free_elem_batch (elem_batch); +return 0 ; + } +bpf_dynptr_from_mem_slice (&elem->mem_slice,0 , &dptr); +bpf_stream_free_elem (elem); + +bpf_dynptr_read (buf,sizeof (buf), &dptr,0 ,0 ); +return 0 ; +}
tools/testing/selftests/bpf/progs/stream.c new file +150 -0
150 lines not on this page · raw message
tools/testing/selftests/bpf/progs/stream_bpftool.c new file +142 -0
142 lines not on this page · raw message
tools/testing/selftests/bpf/progs/stream_fail.c new file +38 -0
@@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */ +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +struct map_value { + struct bpf_stream_elem_batch __kptr *batch; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, int); + __type(value, struct map_value); +} arrmap SEC(".maps"); + + +SEC("?tc") +__failure __msg("untrusted_ptr_bpf_stream_elem_batch()") +int stream_kptr_ptr_untrusted(struct __sk_buff *ctx) +{ +struct bpf_stream_elem_batch *batch; +struct map_value *v; +int key =0 ; + + v =bpf_map_lookup_elem (&arrmap, &key); +if (!v) +return 0 ; + batch = v->batch; +if (!batch) +return 0 ; + v->batch = (void *)batch->node->next->next->next; +return 0 ; +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.47.1
Eduard Zingerman
maintainerRe: [RFC PATCH bpf-next/net v1 13/13] selftests/bpf: Add tests for prog streams
Kumar Kartikeya Dwivedi <[email protected]> writes:
[...]
tools/testing/selftests/bpf/progs/stream_bpftool.c
Quoted 5 lines
tools/testing/selftests/bpf/progs/stream_bpftool.c new file +0
-0
This file is almost identical to ./tools/bpf/bpftool/skeleton/stream.bpf.c except it also adds a function foo that exhausts may goto budget, as far as I can tell. What is the point of adding it?
In general, I don't think we run bpftool tests on the CI, so it would be good to test stream.bpf.c itself e.g. by creating a symlink to it in selftests and adding a corresponding prog_tests/<smth>.c as you do here.
Also, it would be good if some of the tests checked the content read from the stream. I think existing tests only check the size.
[...]