Introduce typed pointer support in BPF maps
Subsystems (8)
Kumar Kartikeya Dwivedi
maintainerv6 cover Introduce typed pointer support in BPF maps
This set enables storing pointers of a certain type in BPF map, and extends the verifier to enforce type safety and lifetime correctness properties.
The infrastructure being added is generic enough for allowing storing any kind of pointers whose type is available using BTF (user or kernel) in the future (e.g. strongly typed memory allocation in BPF program), which are internally tracked in the verifier as PTR_TO_BTF_ID, but for now the series limits them to two kinds of pointers obtained from the kernel.
Obviously, use of this feature depends on map BTF.
1. Unreferenced kernel pointer
In this case, there are very few restrictions. The pointer type being stored must match the type declared in the map value. However, such a pointer when loaded from the map can only be dereferenced, but not passed to any in-kernel helpers or kernel functions available to the program. This is because while the verifier's exception handling mechanism coverts BPF_LDX to PROBE_MEM loads, which are then handled specially by the JIT implementation, the same liberty is not available to accesses inside the kernel. The pointer by the time it is passed into a helper has no lifetime related guarantees about the object it is pointing to, and may well be referencing invalid memory.
2. Referenced kernel pointer
This case imposes a lot of restrictions on the programmer, to ensure safety. To transfer the ownership of a reference in the BPF program to the map, the user must use the bpf_kptr_xchg helper, which returns the old pointer contained in the map, as an acquired reference, and releases verifier state for the referenced pointer being exchanged, as it moves into the map.
This a normal PTR_TO_BTF_ID that can be used with in-kernel helpers and kernel functions callable by the program.
However, if BPF_LDX is used to load a referenced pointer from the map, it is still not permitted to pass it to in-kernel helpers or kernel functions. To obtain a reference usable with helpers, the user must invoke a kfunc helper which returns a usable reference (which also must be eventually released before BPF_EXIT, or moved into a map).
Since the load of the pointer (preserving data dependency ordering) must happen inside the RCU read section, the kfunc helper will take a pointer to the map value, which must point to the actual pointer of the object whose reference is to be raised. The type will be verified from the BTF information of the kfunc, as the prototype must be:
T *func(T **, ... /* other arguments */);
Then, the verifier checks whether pointer at offset of the map value points to the type T, and permits the call.
This convention is followed so that such helpers may also be called from sleepable BPF programs, where RCU read lock is not necessarily held in the BPF program context, hence necessiating the need to pass in a pointer to the actual pointer to perform the load inside the RCU read section.
Notes
-----
* C selftests require https://reviews.llvm.org/D119799 to pass.
* Unlike BPF timers, kptr is not reset or freed on map_release_uref.
* Referenced kptr storage is always treated as unsigned long * on kernel side, as BPF side cannot mutate it. The storage (8 bytes) is sufficient for both 32-bit and 64-bit platforms.
* Use of WRITE_ONCE to reset unreferenced kptr on 32-bit systems is fine, as the actual pointer is always word sized, so the store tearing into two 32-bit stores won't be a problem as the other half is always zeroed out.
Changelog:
----------
v5 -> v6
v5: https://lore.kernel.org/bpf/[email protected]
* Address comments from Alexei
* Drop 'Revisit stack usage' comment
* Rename off_btf to kernel_btf
* Add comment about searching using type from map BTF
* Do kmemdup + btf_get instead of get + kmemdup + put
* Add comment for btf_struct_ids_match
* Add comment for assigning non-zero id for mark_ptr_or_null_reg
* Rename PTR_RELEASE to OBJ_RELEASE
* Rename BPF_MAP_OFF_DESC_TYPE_XXX_KPTR to BPF_KPTR_XXX
* Remove unneeded likely/unlikely in cold functions
* Fix other misc nits
* Keep release_regno instead of replacing with bool + regno
* Add a patch to prevent type match for first member when off == 0 for release functions (kfunc + BPF helpers)
* Guard kptr/kptr_ref definition in libbpf header with __has_attribute to prevent selftests compilation error with old clang not support type tags
v4 -> v5
v4: https://lore.kernel.org/bpf/[email protected]
* Address comments from Joanne
* Move __btf_member_bit_offset before strcmp
* Move strcmp conditional on name to unref kptr patch
* Directly return from btf_find_struct in patch 1
* Use enum btf_field_type vs int field_type
* Put btf and btf_id in off_desc in named struct 'kptr'
* Switch order for BTF_FIELD_IGNORE check
* Drop dead tab->nr_off = 0 store
* Use i instead of tab->nr_off to btf_put on failure
* Replace kzalloc + memcpy with kmemdup (kernel test robot)
* Reject both BPF_F_RDONLY_PROG and BPF_F_WRONLY_PROG
* Add logging statement for reject BPF_MODE(insn->code) != BPF_MEM
* Rename off_desc -> kptr_off_desc in check_mem_access
* Drop check for err, fallthrough to end of function
* Remove is_release_function, use meta.release_regno to detect release function, release reference state, and remove check_release_regno
* Drop off_desc->flags, use off_desc->type
* Update comment for ARG_PTR_TO_KPTR
* Distinguish between direct/indirect access to kptr
* Drop check_helper_mem_access from process_kptr_func, check_mem_reg in kptr_get
* Add verifier test for helper accessing kptr indirectly
* Fix other misc nits, add Acked-by for patch 2
v3 -> v4
v3: https://lore.kernel.org/bpf/[email protected]
* Use btf_parse_kptrs, plural kptrs naming (Joanne, Andrii)
* Remove unused parameters in check_map_kptr_access (Joanne)
* Handle idx < info_cnt kludge using tmp variable (Andrii)
* Validate tags always precede modifiers in BTF (Andrii)
* Split out into https://lore.kernel.org/bpf/[email protected]
* Store u32 type_id in btf_field_info (Andrii)
* Use base_type in map_kptr_match_type (Andrii)
* Free kptr_off_tab when not bpf_capable (Martin)
* Use PTR_RELEASE flag instead of bools in bpf_func_proto (Joanne)
* Drop extra reg->off and reg->ref_obj_id checks in map_kptr_match_type (Martin)
* Use separate u32 and u8 arrays for offs and sizes in off_arr (Andrii)
* Simplify and remove map->value_size sentinel in copy_map_value (Andrii)
* Use sort_r to keep both arrays in sync while sorting (Andrii)
* Rename check_and_free_timers_and_kptr to check_and_free_fields (Andrii)
* Move dtor prototype checks to registration phase (Alexei)
* Use ret variable for checking ASSERT_XXX, use shorter strings (Andrii)
* Fix missing checks for other maps (Jiri)
* Fix various other nits, and bugs noticed during self review
v2 -> v3
v2: https://lore.kernel.org/bpf/[email protected]
* Address comments from Alexei
* Set name, sz, align in btf_find_field
* Do idx >= info_cnt check in caller of btf_find_field_*
* Use extra element in the info_arr to make this safe
* Remove while loop, reject extra tags
* Remove cases of defensive programming
* Move bpf_capable() check to map_check_btf
* Put check_ptr_off_reg reordering hunk into separate patch
* Warn for ref_ptr once
* Make the meta.ref_obj_id == 0 case simpler to read
* Remove kptr_percpu and kptr_user support, remove their tests
* Store size of field at offset in off_arr
* Fix BPF_F_NO_PREALLOC set wrongly for hash map in C selftest
* Add missing check_mem_reg call for kptr_get kfunc arg#0 check
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
* Address comments from Alexei
* Rename bpf_btf_find_by_name_kind_all to bpf_find_btf_id
* Reduce indentation level in that function
* Always take reference regardless of module or vmlinux BTF
* Also made it the same for btf_get_module_btf
* Use kptr, kptr_ref, kptr_percpu, kptr_user type tags
* Don't reserve tag namespace
* Refactor btf_find_field to be side effect free, allocate and populate kptr_off_tab in caller
* Move module reference to dtor patch
* Remove support for BPF_XCHG, BPF_CMPXCHG insn
* Introduce bpf_kptr_xchg helper
* Embed offset array in struct bpf_map, populate and sort it once
* Adjust copy_map_value to memcpy directly using this offset array
* Removed size member from offset array to save space
* Fix some problems pointed out by kernel test robot
* Tidy selftests
* Lots of other minor fixes
Kumar Kartikeya Dwivedi (13):
bpf: Allow storing unreferenced kptr in map
bpf: Tag argument to be released in bpf_func_proto
bpf: Allow storing referenced kptr in map
bpf: Prevent escaping of kptr loaded from maps
bpf: Adapt copy_map_value for multiple offset case
bpf: Populate pairs of btf_id and destructor kfunc in btf
bpf: Wire up freeing of referenced kptr
bpf: Teach verifier about kptr_get kfunc helpers
bpf: Make BTF type match stricter for release arguments
libbpf: Add kptr type tag macros to bpf_helpers.h selftests/bpf: Add C tests for kptr selftests/bpf: Add verifier tests for kptr selftests/bpf: Add test for strict BTF type check
include/linux/bpf.h | 113 +++-- include/linux/bpf_verifier.h | 3 +- include/linux/btf.h | 23 + include/uapi/linux/bpf.h | 12 + kernel/bpf/arraymap.c | 18 +- kernel/bpf/btf.c | 460 +++++++++++++++-- kernel/bpf/hashtab.c | 64 ++- kernel/bpf/helpers.c | 21 + kernel/bpf/map_in_map.c | 5 +- kernel/bpf/ringbuf.c | 4 +- kernel/bpf/syscall.c | 234 ++++++++- kernel/bpf/verifier.c | 370 +++++++++++--- net/bpf/test_run.c | 67 ++- net/core/filter.c | 2 +- tools/include/uapi/linux/bpf.h | 12 + tools/lib/bpf/bpf_helpers.h | 7 + .../selftests/bpf/prog_tests/map_kptr.c | 37 ++ tools/testing/selftests/bpf/progs/map_kptr.c | 190 +++++++ tools/testing/selftests/bpf/test_verifier.c | 55 +- tools/testing/selftests/bpf/verifier/calls.c | 20 + .../testing/selftests/bpf/verifier/map_kptr.c | 469 ++++++++++++++++++ .../selftests/bpf/verifier/ref_tracking.c | 2 +- tools/testing/selftests/bpf/verifier/sock.c | 6 +- 23 files changed, 2032 insertions(+), 162 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/map_kptr.c create mode 100644 tools/testing/selftests/bpf/progs/map_kptr.c create mode 100644 tools/testing/selftests/bpf/verifier/map_kptr.c
Signature · 1 line
2.35.1
Kumar Kartikeya Dwivedi
maintainerPATCH v6 1/13 bpf: Allow storing unreferenced kptr in map
This commit introduces a new pointer type 'kptr' which can be embedded in a map value to hold a PTR_TO_BTF_ID stored by a BPF program during its invocation. When storing such a kptr, BPF program's PTR_TO_BTF_ID register must have the same type as in the map value's BTF, and loading a kptr marks the destination register as PTR_TO_BTF_ID with the correct kernel BTF and BTF ID.
Such kptr are unreferenced, i.e. by the time another invocation of the BPF program loads this pointer, the object which the pointer points to may not longer exist. Since PTR_TO_BTF_ID loads (using BPF_LDX) are patched to PROBE_MEM loads by the verifier, it would safe to allow user to still access such invalid pointer, but passing such pointers into BPF helpers and kfuncs should not be permitted. A future patch in this series will close this gap.
The flexibility offered by allowing programs to dereference such invalid pointers while being safe at runtime frees the verifier from doing complex lifetime tracking. As long as the user may ensure that the object remains valid, it can ensure data read by it from the kernel object is valid.
The user indicates that a certain pointer must be treated as kptr capable of accepting stores of PTR_TO_BTF_ID of a certain type, by using a BTF type tag 'kptr' on the pointed to type of the pointer. Then, this information is recorded in the object BTF which will be passed into the kernel by way of map's BTF information. The name and kind from the map value BTF is used to look up the in-kernel type, and the actual BTF and BTF ID is recorded in the map struct in a new kptr_off_tab member. For now, only storing pointers to structs is permitted.
An example of this specification is shown below:
#define __kptr __attribute__((btf_type_tag("kptr")))
struct map_value {
...
struct task_struct __kptr *task;
...
};Then, in a BPF program, user may store PTR_TO_BTF_ID with the type task_struct into the map, and then load it later.
Note that the destination register is marked PTR_TO_BTF_ID_OR_NULL, as the verifier cannot know whether the value is NULL or not statically, it must treat all potential loads at that map value offset as loading a possibly NULL pointer.
Only BPF_LDX, BPF_STX, and BPF_ST (with insn->imm = 0 to denote NULL) are allowed instructions that can access such a pointer. On BPF_LDX, the destination register is updated to be a PTR_TO_BTF_ID, and on BPF_STX, it is checked whether the source register type is a PTR_TO_BTF_ID with same BTF type as specified in the map BTF. The access size must always be BPF_DW.
For the map in map support, the kptr_off_tab for outer map is copied from the inner map's kptr_off_tab. It was chosen to do a deep copy instead of introducing a refcount to kptr_off_tab, because the copy only needs to be done when paramterizing using inner_map_fd in the map in map case, hence would be unnecessary for all other users.
It is not permitted to use MAP_FREEZE command and mmap for BPF map having kptrs, similar to the bpf_timer case. A kptr also requires that BPF program has both read and write access to the map (hence both BPF_F_RDONLY_PROG and BPF_F_WRONLY_PROG are disallowed).
Note that check_map_access must be called from both check_helper_mem_access and for the BPF instructions, hence the kptr check must distinguish between ACCESS_DIRECT and ACCESS_HELPER, and reject ACCESS_HELPER cases. We rename stack_access_src to bpf_access_src and reuse it for this purpose.
--- include/linux/bpf.h | 31 +++++++- include/linux/btf.h | 2 + kernel/bpf/btf.c | 167 +++++++++++++++++++++++++++++++++++----- kernel/bpf/map_in_map.c | 5 +- kernel/bpf/syscall.c | 103 ++++++++++++++++++++++++- kernel/bpf/verifier.c | 161 +++++++++++++++++++++++++++++++++++--- 6 files changed, 433 insertions(+), 36 deletions(-)
include/linux/bpf.h +30 -1
@@ -155,6 +155,24 @@ struct bpf_map_ops {const struct bpf_iter_seq_info *iter_seq_info; }; +enum { +/* Support at most 8 pointers in a BPF map value */ +BPF_MAP_VALUE_OFF_MAX =8 , +}; + +struct bpf_map_value_off_desc { +u32 offset; +struct { +struct btf *btf; +u32 btf_id; + } kptr; +}; + +struct bpf_map_value_off { +u32 nr_off; +struct bpf_map_value_off_desc off[]; +}; +struct bpf_map {/* The first two cachelines with read-mostly members of which some * are also accessed in fast-path (e.g. ops, max_entries).
@@ -171,6 +189,7 @@ struct bpf_map {u64 map_extra;/* any per-map-type extra fields */ u32 map_flags;int spin_lock_off;/* >=0 valid offset, <0 error */ +struct bpf_map_value_off *kptr_off_tab;int timer_off;/* >=0 valid offset, <0 error */ u32 id;int numa_node;
@@ -184,7 +203,7 @@ struct bpf_map {char name[BPF_OBJ_NAME_LEN ];bool bypass_spec_v1;bool frozen;/* write-once; write-protected by freeze_mutex */ -/* 14 bytes hole */ +/* 6 bytes hole */ /* The 3rd and 4th cacheline with misc members to avoid false sharing * particularly with refcounting.
@@ -217,6 +236,11 @@ static inline bool map_value_has_timer(const struct bpf_map *map)return map->timer_off >=0 ; } +static inline bool map_value_has_kptrs (const struct bpf_map *map) +{ +return !IS_ERR_OR_NULL (map->kptr_off_tab); +} +static inline void check_and_init_map_value (struct bpf_map *map,void *dst) {if (unlikely (map_value_has_spin_lock (map)))
@@ -1396,6 +1420,11 @@ void bpf_prog_put(struct bpf_prog *prog);void bpf_prog_free_id (struct bpf_prog *prog,bool do_idr_lock);void bpf_map_free_id (struct bpf_map *map,bool do_idr_lock); +struct bpf_map_value_off_desc *bpf_map_kptr_off_contains (struct bpf_map *map,u32 offset); +void bpf_map_free_kptr_off_tab (struct bpf_map *map); +struct bpf_map_value_off *bpf_map_copy_kptr_off_tab (const struct bpf_map *map); +bool bpf_map_equal_kptr_off_tab (const struct bpf_map *map_a,const struct bpf_map *map_b); +struct bpf_map *bpf_map_get (u32 ufd);struct bpf_map *bpf_map_get_with_uref (u32 ufd);struct bpf_map *__bpf_map_get (struct fd f);
include/linux/btf.h +2 -0
@@ -123,6 +123,8 @@ bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s, u32 expected_offset, u32 expected_size);int btf_find_spin_lock (const struct btf *btf,const struct btf_type *t);int btf_find_timer (const struct btf *btf,const struct btf_type *t); +struct bpf_map_value_off *btf_parse_kptrs (const struct btf *btf, +const struct btf_type *t);bool btf_type_is_void (const struct btf_type *t);s32 btf_find_by_name_kind (const struct btf *btf,const char *name,u8 kind); const struct btf_type *btf_type_skip_modifiers(const struct btf *btf,
kernel/bpf/btf.c +146 -21
268 lines not on this page · raw message
kernel/bpf/map_in_map.c +4 -1
@@ -52,6 +52,7 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) inner_map_meta->max_entries = inner_map->max_entries; inner_map_meta->spin_lock_off = inner_map->spin_lock_off; inner_map_meta->timer_off = inner_map->timer_off; + inner_map_meta->kptr_off_tab =bpf_map_copy_kptr_off_tab (inner_map);if (inner_map->btf) {btf_get (inner_map->btf); inner_map_meta->btf = inner_map->btf;
@@ -71,6 +72,7 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)void bpf_map_meta_free (struct bpf_map *map_meta) { +bpf_map_free_kptr_off_tab (map_meta);btf_put (map_meta->btf);kfree (map_meta); }
@@ -83,7 +85,8 @@ bool bpf_map_meta_equal(const struct bpf_map *meta0, meta0->key_size == meta1->key_size && meta0->value_size == meta1->value_size && meta0->timer_off == meta1->timer_off && - meta0->map_flags == meta1->map_flags; + meta0->map_flags == meta1->map_flags && +bpf_map_equal_kptr_off_tab (meta0, meta1); } void *bpf_map_fd_get_ptr(struct bpf_map *map,
kernel/bpf/syscall.c +100 -3
141 lines not on this page · raw message
kernel/bpf/verifier.c +151 -10
229 lines not on this page · raw message
Signature · 1 line
2.35.1
Kumar Kartikeya Dwivedi
maintainerPATCH v6 2/13 bpf: Tag argument to be released in bpf_func_proto
In reply to Kumar Kartikeya DwivediAdd a new type flag for bpf_arg_type that when set tells verifier that for a release function, that argument's register will be the one for which meta.ref_obj_id will be set, and which will then be released using release_reference. To capture the regno, introduce a new field release_regno in bpf_call_arg_meta.
This would be required in the next patch, where we may either pass NULL or a refcounted pointer as an argument to the release function bpf_kptr_xchg. Just releasing only when meta.ref_obj_id is set is not enough, as there is a case where the type of argument needed matches, but the ref_obj_id is set to 0. Hence, we must enforce that whenever meta.ref_obj_id is zero, the register that is to be released can only be NULL for a release function.
Since we now indicate whether an argument is to be released in bpf_func_proto itself, is_release_function helper has lost its utitlity, hence refactor code to work without it, and just rely on meta.release_regno to know when to release state for a ref_obj_id. Still, the restriction of one release argument and only one ref_obj_id passed to BPF helper or kfunc remains. This may be lifted in the future.
--- include/linux/bpf.h | 5 +- include/linux/bpf_verifier.h | 3 +- kernel/bpf/btf.c | 11 ++- kernel/bpf/ringbuf.c | 4 +- kernel/bpf/verifier.c | 76 +++++++++++-------- net/core/filter.c | 2 +- .../selftests/bpf/verifier/ref_tracking.c | 2 +- tools/testing/selftests/bpf/verifier/sock.c | 6 +- 8 files changed, 60 insertions(+), 49 deletions(-)
include/linux/bpf.h +4 -1
@@ -366,7 +366,10 @@ enum bpf_type_flag {*/ MEM_PERCPU =BIT (4 +BPF_BASE_TYPE_BITS ), - __BPF_TYPE_LAST_FLAG = MEM_PERCPU, +/* Indicates that the argument will be released. */ +OBJ_RELEASE =BIT (5 +BPF_BASE_TYPE_BITS ), + + __BPF_TYPE_LAST_FLAG = OBJ_RELEASE, };/* Max number of base types. */
include/linux/bpf_verifier.h +1 -2
@@ -523,8 +523,7 @@ int check_ptr_off_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, int regno);int check_func_arg_reg_off (struct bpf_verifier_env *env,const struct bpf_reg_state *reg,int regno, -enum bpf_arg_type arg_type, -bool is_release_func); +enum bpf_arg_type arg_type);int check_kfunc_mem_size_reg (struct bpf_verifier_env *env,struct bpf_reg_state *reg,u32 regno); int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
kernel/bpf/btf.c +5 -6
@@ -6047,6 +6047,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env,* verifier sees. */ for (i =0 ; i < nargs; i++) { +enum bpf_arg_type arg_type =ARG_DONTCARE ;u32 regno = i +1 ;struct bpf_reg_state *reg = ®s[regno];
@@ -6067,7 +6068,9 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, ref_t =btf_type_skip_modifiers (btf, t->type, &ref_id); ref_tname =btf_name_by_offset (btf, ref_t->name_off); - ret =check_func_arg_reg_off (env, reg, regno,ARG_DONTCARE , rel); +if (rel && reg->ref_obj_id) + arg_type |=OBJ_RELEASE ; + ret =check_func_arg_reg_off (env, reg, regno, arg_type);if (ret <0 )return ret;
@@ -6099,11 +6102,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env,if (reg->type ==PTR_TO_BTF_ID ) { reg_btf = reg->btf; reg_ref_id = reg->btf_id; -/* Ensure only one argument is referenced -* PTR_TO_BTF_ID, check_func_arg_reg_off relies -* on only one referenced register being allowed -* for kfuncs. -*/ +/* Ensure only one argument is referenced PTR_TO_BTF_ID */ if (reg->ref_obj_id) {if (ref_obj_id) { bpf_log(log, "verifier internal error: more than one arg with ref_obj_id R%d %u %u\n",
kernel/bpf/ringbuf.c +2 -2
@@ -404,7 +404,7 @@ BPF_CALL_2(bpf_ringbuf_submit, void *, sample, u64, flags)const struct bpf_func_proto bpf_ringbuf_submit_proto = { .func = bpf_ringbuf_submit, .ret_type =RET_VOID , - .arg1_type =ARG_PTR_TO_ALLOC_MEM , + .arg1_type =ARG_PTR_TO_ALLOC_MEM |OBJ_RELEASE , .arg2_type =ARG_ANYTHING , };
@@ -417,7 +417,7 @@ BPF_CALL_2(bpf_ringbuf_discard, void *, sample, u64, flags)const struct bpf_func_proto bpf_ringbuf_discard_proto = { .func = bpf_ringbuf_discard, .ret_type =RET_VOID , - .arg1_type =ARG_PTR_TO_ALLOC_MEM , + .arg1_type =ARG_PTR_TO_ALLOC_MEM |OBJ_RELEASE , .arg2_type =ARG_ANYTHING , };
kernel/bpf/verifier.c +43 -33
153 lines not on this page · raw message
net/core/filter.c +1 -1
@@ -6621,7 +6621,7 @@ static const struct bpf_func_proto bpf_sk_release_proto = { .func = bpf_sk_release, .gpl_only = false, .ret_type = RET_INTEGER, - .arg1_type = ARG_PTR_TO_BTF_ID_SOCK_COMMON, + .arg1_type = ARG_PTR_TO_BTF_ID_SOCK_COMMON | OBJ_RELEASE, }; BPF_CALL_5(bpf_xdp_sk_lookup_udp, struct xdp_buff *, ctx,
tools/testing/selftests/bpf/verifier/ref_tracking.c +1 -1
@@ -796,7 +796,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "reference has not been acquired before", + .errstr = "R1 must be referenced when passed to release function", }, {/* !bpf_sk_fullsock(sk) is checked but !bpf_tcp_sock(sk) is not checked */
tools/testing/selftests/bpf/verifier/sock.c +3 -3
@@ -417,7 +417,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "reference has not been acquired before", + .errstr = "R1 must be referenced when passed to release function", }, { "bpf_sk_release(bpf_sk_fullsock(skb->sk))",
@@ -436,7 +436,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "reference has not been acquired before", + .errstr = "R1 must be referenced when passed to release function", }, { "bpf_sk_release(bpf_tcp_sock(skb->sk))",
@@ -455,7 +455,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "reference has not been acquired before", + .errstr = "R1 must be referenced when passed to release function", }, { "sk_storage_get(map, skb->sk, NULL, 0): value == NULL",
Signature · 1 line
2.35.1
Kumar Kartikeya Dwivedi
maintainerPATCH v6 3/13 bpf: Allow storing referenced kptr in map
In reply to Kumar Kartikeya DwivediExtending the code in previous commits, introduce referenced kptr support, which needs to be tagged using 'kptr_ref' tag instead. Unlike unreferenced kptr, referenced kptr have a lot more restrictions. In addition to the type matching, only a newly introduced bpf_kptr_xchg helper is allowed to modify the map value at that offset. This transfers the referenced pointer being stored into the map, releasing the references state for the program, and returning the old value and creating new reference state for the returned pointer.
Similar to unreferenced pointer case, return value for this case will also be PTR_TO_BTF_ID_OR_NULL. The reference for the returned pointer must either be eventually released by calling the corresponding release function, otherwise it must be transferred into another map.
It is also allowed to call bpf_kptr_xchg with a NULL pointer, to clear the value, and obtain the old value if any.
BPF_LDX, BPF_STX, and BPF_ST cannot access referenced kptr. A future commit will permit using BPF_LDX for such pointers, but attempt at making it safe, since the lifetime of object won't be guaranteed.
There are valid reasons to enforce the restriction of permitting only bpf_kptr_xchg to operate on referenced kptr. The pointer value must be consistent in face of concurrent modification, and any prior values contained in the map must also be released before a new one is moved into the map. To ensure proper transfer of this ownership, bpf_kptr_xchg returns the old value, which the verifier would require the user to either free or move into another map, and releases the reference held for the pointer being moved in.
In the future, direct BPF_XCHG instruction may also be permitted to work like bpf_kptr_xchg helper.
Note that process_kptr_func doesn't have to call check_helper_mem_access, since we already disallow rdonly/wronly flags for map, which is what check_map_access_type checks, and we already ensure the PTR_TO_MAP_VALUE refers to kptr by obtaining its off_desc, so check_map_access is also not required.
--- include/linux/bpf.h | 8 +++ include/uapi/linux/bpf.h | 12 +++++ kernel/bpf/btf.c | 10 +++- kernel/bpf/helpers.c | 21 ++++++++ kernel/bpf/verifier.c | 98 +++++++++++++++++++++++++++++----- tools/include/uapi/linux/bpf.h | 12 +++++ 6 files changed, 148 insertions(+), 13 deletions(-)
include/linux/bpf.h +8 -0
@@ -160,8 +160,14 @@ enum { BPF_MAP_VALUE_OFF_MAX = 8, }; +enum bpf_kptr_type { +BPF_KPTR_UNREF , +BPF_KPTR_REF , +}; +struct bpf_map_value_off_desc {u32 offset; +enum bpf_kptr_type type;struct {struct btf *btf;u32 btf_id;
@@ -418,6 +424,7 @@ enum bpf_arg_type {ARG_PTR_TO_STACK ,/* pointer to stack */ ARG_PTR_TO_CONST_STR ,/* pointer to a null terminated read-only string */ ARG_PTR_TO_TIMER ,/* pointer to bpf_timer */ +ARG_PTR_TO_KPTR ,/* pointer to referenced kptr */ __BPF_ARG_TYPE_MAX,/* Extended arg_types. */
@@ -427,6 +434,7 @@ enum bpf_arg_type {ARG_PTR_TO_SOCKET_OR_NULL =PTR_MAYBE_NULL |ARG_PTR_TO_SOCKET ,ARG_PTR_TO_ALLOC_MEM_OR_NULL =PTR_MAYBE_NULL |ARG_PTR_TO_ALLOC_MEM ,ARG_PTR_TO_STACK_OR_NULL =PTR_MAYBE_NULL |ARG_PTR_TO_STACK , +ARG_PTR_TO_BTF_ID_OR_NULL =PTR_MAYBE_NULL |ARG_PTR_TO_BTF_ID ,/* This must be the last entry. Its purpose is to ensure the enum is * wide enough to hold the higher bits reserved for bpf_type_flag.
include/uapi/linux/bpf.h +12 -0
@@ -5143,6 +5143,17 @@ union bpf_attr {* The **hash_algo** is returned on success, * **-EOPNOTSUP** if the hash calculation failed or **-EINVAL** if * invalid arguments are passed. +* +* void *bpf_kptr_xchg(void *map_value, void *ptr) +* Description +* Exchange kptr at pointer *map_value* with *ptr*, and return the +* old value. *ptr* can be NULL, otherwise it must be a referenced +* pointer which will be released when this helper is called. +* Return +* The old value of kptr (which can be NULL). The returned pointer +* if not NULL, is a reference which must be released using its +* corresponding release function, or moved into a BPF map before +* program exit. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \
@@ -5339,6 +5350,7 @@ union bpf_attr {FN (copy_from_user_task), \FN (skb_set_tstamp), \FN (ima_file_hash), \ +FN (kptr_xchg), \/* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper
kernel/bpf/btf.c +9 -1
@@ -3177,6 +3177,7 @@ enum {struct btf_field_info {u32 type_id;u32 off; +enum bpf_kptr_type type; }; static int btf_find_struct(const struct btf *btf, const struct btf_type *t,
@@ -3193,6 +3194,7 @@ static int btf_find_struct(const struct btf *btf, const struct btf_type *t,static int btf_find_kptr (const struct btf *btf,const struct btf_type *t,u32 off,int sz,struct btf_field_info *info) { +enum bpf_kptr_type type;u32 res_id;/* For PTR, sz is always == 8 */
@@ -3205,7 +3207,11 @@ static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,/* Reject extra tags */ if (btf_type_is_type_tag (btf_type_by_id (btf, t->type)))return -EINVAL ; -if (strcmp ("kptr" ,__btf_name_by_offset (btf, t->name_off))) +if (!strcmp ("kptr" ,__btf_name_by_offset (btf, t->name_off))) + type =BPF_KPTR_UNREF ; +else if (!strcmp ("kptr_ref" ,__btf_name_by_offset (btf, t->name_off))) + type =BPF_KPTR_REF ; +else return -EINVAL ;/* Get the base type */
@@ -3216,6 +3222,7 @@ static int btf_find_kptr(const struct btf *btf, const struct btf_type *t, info->type_id = res_id; info->off = off; + info->type = type;return BTF_FIELD_FOUND ; }
@@ -3420,6 +3427,7 @@ struct bpf_map_value_off *btf_parse_kptrs(const struct btf *btf, } tab->off[i].offset = info_arr[i].off; + tab->off[i].type = info_arr[i].type; tab->off[i].kptr.btf_id = id; tab->off[i].kptr.btf = kernel_btf; }
kernel/bpf/helpers.c +21 -0
@@ -1374,6 +1374,25 @@ void bpf_timer_cancel_and_free(void *val)kfree (t); } +BPF_CALL_2(bpf_kptr_xchg, void *, map_value, void *, ptr) +{ +unsigned long *kptr = map_value; + +return xchg (kptr, (unsigned long )ptr); +} + +static u32 bpf_kptr_xchg_btf_id; + +const struct bpf_func_proto bpf_kptr_xchg_proto = { + .func = bpf_kptr_xchg, + .gpl_only = false, + .ret_type =RET_PTR_TO_BTF_ID_OR_NULL , + .ret_btf_id = &bpf_kptr_xchg_btf_id, + .arg1_type =ARG_PTR_TO_KPTR , + .arg2_type =ARG_PTR_TO_BTF_ID_OR_NULL |OBJ_RELEASE , + .arg2_btf_id = &bpf_kptr_xchg_btf_id, +}; +const struct bpf_func_proto bpf_get_current_task_proto__weak ;const struct bpf_func_proto bpf_get_current_task_btf_proto__weak ;const struct bpf_func_proto bpf_probe_read_user_proto__weak ;
@@ -1452,6 +1471,8 @@ bpf_base_func_proto(enum bpf_func_id func_id)return &bpf_timer_start_proto;case BPF_FUNC_timer_cancel:return &bpf_timer_cancel_proto; +case BPF_FUNC_kptr_xchg: +return &bpf_kptr_xchg_proto;default :break ; }
kernel/bpf/verifier.c +86 -12
186 lines not on this page · raw message
tools/include/uapi/linux/bpf.h +12 -0
@@ -5143,6 +5143,17 @@ union bpf_attr {* The **hash_algo** is returned on success, * **-EOPNOTSUP** if the hash calculation failed or **-EINVAL** if * invalid arguments are passed. +* +* void *bpf_kptr_xchg(void *map_value, void *ptr) +* Description +* Exchange kptr at pointer *map_value* with *ptr*, and return the +* old value. *ptr* can be NULL, otherwise it must be a referenced +* pointer which will be released when this helper is called. +* Return +* The old value of kptr (which can be NULL). The returned pointer +* if not NULL, is a reference which must be released using its +* corresponding release function, or moved into a BPF map before +* program exit. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \
@@ -5339,6 +5350,7 @@ union bpf_attr {FN (copy_from_user_task), \FN (skb_set_tstamp), \FN (ima_file_hash), \ +FN (kptr_xchg), \/* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper
Signature · 1 line
2.35.1
Re: [PATCH bpf-next v6 03/13] bpf: Allow storing referenced kptr in map
kernel/bpf/helpers.c void bpf_timer_cancel_and_free(void *val) · line 1394
Quoted 12 lines · Kumar Kartikeya Dwivedi
kernel/bpf/helpers.c +12
-0+ +static u32 bpf_kptr_xchg_btf_id; + +const struct bpf_func_proto bpf_kptr_xchg_proto = { + .func = bpf_kptr_xchg, + .gpl_only = false, + .ret_type =RET_PTR_TO_BTF_ID_OR_NULL , + .ret_btf_id = &bpf_kptr_xchg_btf_id, + .arg1_type =ARG_PTR_TO_KPTR , + .arg2_type =ARG_PTR_TO_BTF_ID_OR_NULL |OBJ_RELEASE , + .arg2_btf_id = &bpf_kptr_xchg_btf_id, +};
I've tweaked above with:
+/* Unlike other PTR_TO_BTF_ID helpers the btf_id in bpf_kptr_xchg()
+ * helper is determined dynamically by the verifier.
+ */
+#define BPF_PTR_POISON ((void *)((0xeB9FUL << 2) + POISON_POINTER_DELTA)) +
+const struct bpf_func_proto bpf_kptr_xchg_proto = {
+ .func = bpf_kptr_xchg, + .gpl_only = false, + .ret_type = RET_PTR_TO_BTF_ID_OR_NULL, + .ret_btf_id = BPF_PTR_POISON, + .arg1_type = ARG_PTR_TO_KPTR, + .arg2_type = ARG_PTR_TO_BTF_ID_OR_NULL | OBJ_RELEASE, + .arg2_btf_id = BPF_PTR_POISON,
Kumar Kartikeya Dwivedi
maintainerPATCH v6 4/13 bpf: Prevent escaping of kptr loaded from maps
In reply to Kumar Kartikeya DwivediWhile we can guarantee that even for unreferenced kptr, the object pointer points to being freed etc. can be handled by the verifier's exception handling (normal load patching to PROBE_MEM loads), we still cannot allow the user to pass these pointers to BPF helpers and kfunc, because the same exception handling won't be done for accesses inside the kernel. The same is true if a referenced pointer is loaded using normal load instruction. Since the reference is not guaranteed to be held while the pointer is used, it must be marked as untrusted.
Hence introduce a new type flag, PTR_UNTRUSTED, which is used to mark all registers loading unreferenced and referenced kptr from BPF maps, and ensure they can never escape the BPF program and into the kernel by way of calling stable/unstable helpers.
In check_ptr_to_btf_access, the !type_may_be_null check to reject type flags is still correct, as apart from PTR_MAYBE_NULL, only MEM_USER, MEM_PERCPU, and PTR_UNTRUSTED may be set for PTR_TO_BTF_ID. The first two are checked inside the function and rejected using a proper error message, but we still want to allow dereference of untrusted case.
Also, we make sure to inherit PTR_UNTRUSTED when chain of pointers are walked, so that this flag is never dropped once it has been set on a PTR_TO_BTF_ID (i.e. trusted to untrusted transition can only be in one direction).
In convert_ctx_accesses, extend the switch case to consider untrusted PTR_TO_BTF_ID in addition to normal PTR_TO_BTF_ID for PROBE_MEM conversion for BPF_LDX.
--- include/linux/bpf.h | 10 +++++++++- kernel/bpf/verifier.c | 35 ++++++++++++++++++++++++++++------- 2 files changed, 37 insertions(+), 8 deletions(-)
include/linux/bpf.h +9 -1
@@ -375,7 +375,15 @@ enum bpf_type_flag {/* Indicates that the argument will be released. */ OBJ_RELEASE =BIT (5 +BPF_BASE_TYPE_BITS ), - __BPF_TYPE_LAST_FLAG = OBJ_RELEASE, +/* PTR is not trusted. This is only used with PTR_TO_BTF_ID, to mark +* unreferenced and referenced kptr loaded from map value using a load +* instruction, so that they can only be dereferenced but not escape the +* BPF program into the kernel (i.e. cannot be passed as arguments to +* kfunc or bpf helpers). +*/ +PTR_UNTRUSTED =BIT (6 +BPF_BASE_TYPE_BITS ), + + __BPF_TYPE_LAST_FLAG = PTR_UNTRUSTED, };/* Max number of base types. */
kernel/bpf/verifier.c +28 -7
@@ -567,6 +567,8 @@ static const char *reg_type_str(struct bpf_verifier_env *env,strncpy (prefix,"user_" ,32 );if (type &MEM_PERCPU )strncpy (prefix,"percpu_" ,32 ); +if (type &PTR_UNTRUSTED ) +strncpy (prefix,"untrusted_" ,32 );snprintf (env->type_str_buf,TYPE_STR_BUF_LEN ,"%s%s%s" , prefix, str[base_type (type)], postfix);
@@ -3504,9 +3506,14 @@ static int map_kptr_match_type(struct bpf_verifier_env *env, struct bpf_reg_state *reg, u32 regno) {const char *targ_name =kernel_type_name (off_desc->kptr.btf, off_desc->kptr.btf_id); +int perm_flags =PTR_MAYBE_NULL ;const char *reg_name ="" ; -if (base_type (reg->type) !=PTR_TO_BTF_ID ||type_flag (reg->type) !=PTR_MAYBE_NULL ) +/* Only unreferenced case accepts untrusted pointers */ +if (off_desc->type ==BPF_KPTR_UNREF ) + perm_flags |=PTR_UNTRUSTED ; + +if (base_type (reg->type) !=PTR_TO_BTF_ID || (type_flag (reg->type) & ~perm_flags)) goto bad_type;if (!btf_is_kernel (reg->btf)) {
@@ -3553,7 +3560,12 @@ static int map_kptr_match_type(struct bpf_verifier_env *env, bad_type:verbose (env,"invalid kptr access, R%d type=%s%s " , regno,reg_type_str (env, reg->type), reg_name); -verbose (env,"expected=%s%s\n" ,reg_type_str (env,PTR_TO_BTF_ID ), targ_name); +verbose (env,"expected=%s%s" ,reg_type_str (env,PTR_TO_BTF_ID ), targ_name); +if (off_desc->type ==BPF_KPTR_UNREF ) +verbose (env," or %s%s\n" ,reg_type_str (env,PTR_TO_BTF_ID |PTR_UNTRUSTED ), + targ_name); +else +verbose (env,"\n" );return -EINVAL ; }
@@ -3577,9 +3589,11 @@ static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno,return -EACCES ; } -/* We cannot directly access kptr_ref */ -if (off_desc->type ==BPF_KPTR_REF ) { -verbose (env,"accessing referenced kptr disallowed\n" ); +/* We only allow loading referenced kptr, since it will be marked as +* untrusted, similar to unreferenced kptr. +*/ +if (class !=BPF_LDX && off_desc->type ==BPF_KPTR_REF ) { +verbose (env,"store to referenced kptr disallowed\n" );return -EACCES ; }
@@ -3589,7 +3603,7 @@ static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno,* value from map as PTR_TO_BTF_ID, with the correct type. */ mark_btf_ld_reg (env,cur_regs (env), value_regno,PTR_TO_BTF_ID , off_desc->kptr.btf, - off_desc->kptr.btf_id,PTR_MAYBE_NULL ); + off_desc->kptr.btf_id,PTR_MAYBE_NULL |PTR_UNTRUSTED );/* For mark_ptr_or_null_reg */ val_reg->id = ++env->id_gen; }else if (class ==BPF_STX ) {
@@ -4358,6 +4372,12 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,if (ret <0 )return ret; +/* If this is an untrusted pointer, all pointers formed by walking it +* also inherit the untrusted flag. +*/ +if (type_flag (reg->type) &PTR_UNTRUSTED ) + flag |=PTR_UNTRUSTED ; +if (atype ==BPF_READ && value_regno >=0 )mark_btf_ld_reg (env, regs, value_regno, ret, reg->btf, btf_id, flag);
@@ -13076,7 +13096,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)if (!ctx_access)continue ; -switch (env->insn_aux_data[i + delta].ptr_type) { +switch ((int )env->insn_aux_data[i + delta].ptr_type) {case PTR_TO_CTX :if (!ops->convert_ctx_access)continue ;
@@ -13093,6 +13113,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) convert_ctx_access = bpf_xdp_sock_convert_ctx_access;break ;case PTR_TO_BTF_ID : +case PTR_TO_BTF_ID |PTR_UNTRUSTED :if (type ==BPF_READ ) { insn->code =BPF_LDX |BPF_PROBE_MEM |BPF_SIZE ((insn)->code);
Signature · 1 line
2.35.1
Kumar Kartikeya Dwivedi
maintainerPATCH v6 5/13 bpf: Adapt copy_map_value for multiple offset case
In reply to Kumar Kartikeya DwivediSince now there might be at most 10 offsets that need handling in copy_map_value, the manual shuffling and special case is no longer going to work. Hence, let's generalise the copy_map_value function by using a sorted array of offsets to skip regions that must be avoided while copying into and out of a map value.
When the map is created, we populate the offset array in struct map, Then, copy_map_value uses this sorted offset array is used to memcpy while skipping timer, spin lock, and kptr. The array is allocated as in most cases none of these special fields would be present in map value, hence we can save on space for the common case by not embedding the entire object inside bpf_map struct.
--- include/linux/bpf.h | 56 +++++++++++++++------------- kernel/bpf/syscall.c | 88 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 117 insertions(+), 27 deletions(-)
include/linux/bpf.h +30 -26
@@ -158,6 +158,9 @@ struct bpf_map_ops {enum {/* Support at most 8 pointers in a BPF map value */ BPF_MAP_VALUE_OFF_MAX =8 , +BPF_MAP_OFF_ARR_MAX =BPF_MAP_VALUE_OFF_MAX + +1 +/* for bpf_spin_lock */ +1 ,/* for bpf_timer */ };enum bpf_kptr_type {
@@ -179,6 +182,12 @@ struct bpf_map_value_off {struct bpf_map_value_off_desc off[]; }; +struct bpf_map_off_arr { +u32 cnt; +u32 field_off[BPF_MAP_OFF_ARR_MAX ]; +u8 field_sz[BPF_MAP_OFF_ARR_MAX ]; +}; +struct bpf_map {/* The first two cachelines with read-mostly members of which some * are also accessed in fast-path (e.g. ops, max_entries).
@@ -207,10 +216,7 @@ struct bpf_map {struct mem_cgroup *memcg;#endif char name[BPF_OBJ_NAME_LEN ]; -bool bypass_spec_v1; -bool frozen;/* write-once; write-protected by freeze_mutex */ -/* 6 bytes hole */ - +struct bpf_map_off_arr *off_arr;/* The 3rd and 4th cacheline with misc members to avoid false sharing * particularly with refcounting. */
@@ -230,6 +236,8 @@ struct bpf_map {bool jited;bool xdp_has_frags; } owner; +bool bypass_spec_v1; +bool frozen;/* write-once; write-protected by freeze_mutex */ };static inline bool map_value_has_spin_lock (const struct bpf_map *map)
@@ -253,37 +261,33 @@ static inline void check_and_init_map_value(struct bpf_map *map, void *dst)memset (dst + map->spin_lock_off,0 ,sizeof (struct bpf_spin_lock ));if (unlikely (map_value_has_timer (map)))memset (dst + map->timer_off,0 ,sizeof (struct bpf_timer )); +if (unlikely (map_value_has_kptrs (map))) { +struct bpf_map_value_off *tab = map->kptr_off_tab; +int i; + +for (i =0 ; i < tab->nr_off; i++) + *(u64 *)(dst + tab->off[i].offset) =0 ; + } }/* copy everything but bpf_spin_lock and bpf_timer. There could be one of each. */ static inline void copy_map_value (struct bpf_map *map,void *dst,void *src) { -u32 s_off =0 , s_sz =0 , t_off =0 , t_sz =0 ; +u32 curr_off =0 ; +int i; -if (unlikely (map_value_has_spin_lock (map))) { - s_off = map->spin_lock_off; - s_sz =sizeof (struct bpf_spin_lock ); - } -if (unlikely (map_value_has_timer (map))) { - t_off = map->timer_off; - t_sz =sizeof (struct bpf_timer ); +if (likely (!map->off_arr)) { +memcpy (dst, src, map->value_size); +return ; } -if (unlikely (s_sz || t_sz)) { -if (s_off < t_off || !s_sz) { -swap (s_off, t_off); -swap (s_sz, t_sz); - } -memcpy (dst, src, t_off); -memcpy (dst + t_off + t_sz, - src + t_off + t_sz, - s_off - t_off - t_sz); -memcpy (dst + s_off + s_sz, - src + s_off + s_sz, - map->value_size - s_off - s_sz); - }else { -memcpy (dst, src, map->value_size); +for (i =0 ; i < map->off_arr->cnt; i++) { +u32 next_off = map->off_arr->field_off[i]; + +memcpy (dst + curr_off, src + curr_off, next_off - curr_off); + curr_off += map->off_arr->field_sz[i]; } +memcpy (dst + curr_off, src + curr_off, map->value_size - curr_off); }void copy_map_value_locked (struct bpf_map *map,void *dst,void *src,bool lock_src);
kernel/bpf/syscall.c +87 -1
121 lines not on this page · raw message
Signature · 1 line
2.35.1
Kumar Kartikeya Dwivedi
maintainerPATCH v6 6/13 bpf: Populate pairs of btf_id and destructor kfunc in btf
In reply to Kumar Kartikeya DwivediTo support storing referenced PTR_TO_BTF_ID in maps, we require associating a specific BTF ID with a 'destructor' kfunc. This is because we need to release a live referenced pointer at a certain offset in map value from the map destruction path, otherwise we end up leaking resources.
Hence, introduce support for passing an array of btf_id, kfunc_btf_id pairs that denote a BTF ID and its associated release function. Then, add an accessor 'btf_find_dtor_kfunc' which can be used to look up the destructor kfunc of a certain BTF ID. If found, we can use it to free the object from the map free path.
The registration of these pairs also serve as a whitelist of structures which are allowed as referenced PTR_TO_BTF_ID in a BPF map, because without finding the destructor kfunc, we will bail and return an error.
--- include/linux/btf.h | 17 +++++++ kernel/bpf/btf.c | 108 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+)
include/linux/btf.h +17 -0
@@ -40,6 +40,11 @@ struct btf_kfunc_id_set { }; }; +struct btf_id_dtor_kfunc { +u32 btf_id; +u32 kfunc_btf_id; +}; +extern const struct file_operations btf_fops;void btf_get (struct btf *btf);
@@ -346,6 +351,9 @@ bool btf_kfunc_id_set_contains(const struct btf *btf, enum btf_kfunc_type type, u32 kfunc_btf_id);int register_btf_kfunc_id_set (enum bpf_prog_type prog_type,const struct btf_kfunc_id_set *s); +s32 btf_find_dtor_kfunc (struct btf *btf,u32 btf_id); +int register_btf_id_dtor_kfuncs (const struct btf_id_dtor_kfunc *dtors,u32 add_cnt, +struct module *owner);#else static inline const struct btf_type *btf_type_by_id (const struct btf *btf,u32 type_id)
@@ -369,6 +377,15 @@ static inline int register_btf_kfunc_id_set(enum bpf_prog_type prog_type, {return 0 ; } +static inline s32 btf_find_dtor_kfunc (struct btf *btf,u32 btf_id) +{ +return -ENOENT ; +} +static inline int register_btf_id_dtor_kfuncs (const struct btf_id_dtor_kfunc *dtors, +u32 add_cnt,struct module *owner) +{ +return 0 ; +}#endif #endif
kernel/bpf/btf.c +108 -0
140 lines not on this page · raw message
Signature · 1 line
2.35.1
Kumar Kartikeya Dwivedi
maintainerPATCH v6 7/13 bpf: Wire up freeing of referenced kptr
In reply to Kumar Kartikeya DwivediA destructor kfunc can be defined as void func(type *), where type may be void or any other pointer type as per convenience.
In this patch, we ensure that the type is sane and capture the function pointer into off_desc of ptr_off_tab for the specific pointer offset, with the invariant that the dtor pointer is always set when 'kptr_ref' tag is applied to the pointer's pointee type, which is indicated by the flag BPF_MAP_VALUE_OFF_F_REF.
Note that only BTF IDs whose destructor kfunc is registered, thus become the allowed BTF IDs for embedding as referenced kptr. Hence it serves the purpose of finding dtor kfunc BTF ID, as well acting as a check against the whitelist of allowed BTF IDs for this purpose.
Finally, wire up the actual freeing of the referenced pointer if any at all available offsets, so that no references are leaked after the BPF map goes away and the BPF program previously moved the ownership a referenced pointer into it.
The behavior is similar to BPF timers, where bpf_map_{update,delete}_elem will free any existing referenced kptr. The same case is with LRU map's bpf_lru_push_free/htab_lru_push_free functions, which are extended to reset unreferenced and free referenced kptr.
Note that unlike BPF timers, kptr is not reset or freed when map uref drops to zero.
--- include/linux/bpf.h | 4 ++ include/linux/btf.h | 2 + kernel/bpf/arraymap.c | 18 ++++++-- kernel/bpf/btf.c | 98 ++++++++++++++++++++++++++++++++++++++++++- kernel/bpf/hashtab.c | 64 +++++++++++++++++++++------- kernel/bpf/syscall.c | 49 ++++++++++++++++++++-- 6 files changed, 210 insertions(+), 25 deletions(-)
include/linux/bpf.h +4 -0
@@ -23,6 +23,7 @@#include <linux/slab.h> #include <linux/percpu-refcount.h> #include <linux/bpfptr.h> +#include <linux/btf.h> struct bpf_verifier_env ;struct bpf_verifier_log ;
@@ -173,6 +174,8 @@ struct bpf_map_value_off_desc {enum bpf_kptr_type type;struct {struct btf *btf; +struct module *module; +btf_dtor_kfunc_t dtor;u32 btf_id; } kptr; };
@@ -1447,6 +1450,7 @@ struct bpf_map_value_off_desc *bpf_map_kptr_off_contains(struct bpf_map *map, u3void bpf_map_free_kptr_off_tab (struct bpf_map *map);struct bpf_map_value_off *bpf_map_copy_kptr_off_tab (const struct bpf_map *map);bool bpf_map_equal_kptr_off_tab (const struct bpf_map *map_a,const struct bpf_map *map_b); +void bpf_map_free_kptrs (struct bpf_map *map,void *map_value);struct bpf_map *bpf_map_get (u32 ufd);struct bpf_map *bpf_map_get_with_uref (u32 ufd);
include/linux/btf.h +2 -0
@@ -45,6 +45,8 @@ struct btf_id_dtor_kfunc {u32 kfunc_btf_id; }; +typedef void (*btf_dtor_kfunc_t )(void *); +extern const struct file_operations btf_fops;void btf_get (struct btf *btf);
kernel/bpf/arraymap.c +14 -4
@@ -287,10 +287,12 @@ static int array_map_get_next_key(struct bpf_map *map, void *key, void *next_keyreturn 0 ; } -static void check_and_free_timer_in_array (struct bpf_array *arr,void *val) +static void check_and_free_fields (struct bpf_array *arr,void *val) { -if (unlikely (map_value_has_timer (&arr->map))) +if (map_value_has_timer (&arr->map))bpf_timer_cancel_and_free (val + arr->map.timer_off); +if (map_value_has_kptrs (&arr->map)) +bpf_map_free_kptrs (&arr->map, val); }/* Called from syscall or from eBPF program */
@@ -327,7 +329,7 @@ static int array_map_update_elem(struct bpf_map *map, void *key, void *value,copy_map_value_locked (map, val, value, false); elsecopy_map_value (map ,val ,value ); -check_and_free_timer_in_array (array, val); +check_and_free_fields (array, val); }return 0 ; }
@@ -386,7 +388,8 @@ static void array_map_free_timers(struct bpf_map *map) struct bpf_array *array = container_of(map, struct bpf_array, map);int i; -if (likely (!map_value_has_timer (map))) +/* We don't reset or free kptr on uref dropping to zero. */ +if (!map_value_has_timer (map))return ;for (i =0 ; i < array->map.max_entries; i++)
@@ -398,6 +401,13 @@ static void array_map_free_timers(struct bpf_map *map)static void array_map_free (struct bpf_map *map) { struct bpf_array *array = container_of(map, struct bpf_array, map); +int i; + +if (map_value_has_kptrs (map)) { +for (i =0 ; i < array->map.max_entries; i++) +bpf_map_free_kptrs (map, array->value + array->elem_size * i); +bpf_map_free_kptr_off_tab (map); + }if (array->map.map_type ==BPF_MAP_TYPE_PERCPU_ARRAY )bpf_array_free_percpu (array);
kernel/bpf/btf.c +97 -1
131 lines not on this page · raw message
kernel/bpf/hashtab.c +48 -16
134 lines not on this page · raw message
kernel/bpf/syscall.c +45 -4
76 lines not on this page · raw message
Signature · 1 line
2.35.1
Kumar Kartikeya Dwivedi
maintainerPATCH v6 8/13 bpf: Teach verifier about kptr_get kfunc helpers
In reply to Kumar Kartikeya DwivediWe introduce a new style of kfunc helpers, namely *_kptr_get, where they take pointer to the map value which points to a referenced kernel pointer contained in the map. Since this is referenced, only bpf_kptr_xchg from BPF side and xchg from kernel side is allowed to change the current value, and each pointer that resides in that location would be referenced, and RCU protected (this must be kept in mind while adding kernel types embeddable as reference kptr in BPF maps).
This means that if do the load of the pointer value in an RCU read section, and find a live pointer, then as long as we hold RCU read lock, it won't be freed by a parallel xchg + release operation. This allows us to implement a safe refcount increment scheme. Hence, enforce that first argument of all such kfunc is a proper PTR_TO_MAP_VALUE pointing at the right offset to referenced pointer.
For the rest of the arguments, they are subjected to typical kfunc argument checks, hence allowing some flexibility in passing more intent into how the reference should be taken.
For instance, in case of struct nf_conn, it is not freed until RCU grace period ends, but can still be reused for another tuple once refcount has dropped to zero. Hence, a bpf_ct_kptr_get helper not only needs to call refcount_inc_not_zero, but also do a tuple match after incrementing the reference, and when it fails to match it, put the reference again and return NULL.
This can be implemented easily if we allow passing additional parameters to the bpf_ct_kptr_get kfunc, like a struct bpf_sock_tuple * and a tuple__sz pair.
--- include/linux/btf.h | 2 ++ kernel/bpf/btf.c | 58 +++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 55 insertions(+), 5 deletions(-)
include/linux/btf.h +2 -0
@@ -17,6 +17,7 @@ enum btf_kfunc_type {BTF_KFUNC_TYPE_ACQUIRE ,BTF_KFUNC_TYPE_RELEASE ,BTF_KFUNC_TYPE_RET_NULL , +BTF_KFUNC_TYPE_KPTR_ACQUIRE , BTF_KFUNC_TYPE_MAX, };
@@ -35,6 +36,7 @@ struct btf_kfunc_id_set {struct btf_id_set *acquire_set;struct btf_id_set *release_set;struct btf_id_set *ret_null_set; +struct btf_id_set *kptr_acquire_set; };struct btf_id_set *sets[BTF_KFUNC_TYPE_MAX ]; };
kernel/bpf/btf.c +53 -5
82 lines not on this page · raw message
Signature · 1 line
2.35.1
Kumar Kartikeya Dwivedi
maintainerPATCH v6 9/13 bpf: Make BTF type match stricter for release arguments
In reply to Kumar Kartikeya DwivediThe current of behavior of btf_struct_ids_match for release arguments is that when type match fails, it retries with first member type again (recursively). Since the offset is already 0, this is akin to just casting the pointer in normal C, since if type matches it was just embedded inside parent sturct as an object. However, we want to reject cases for release function type matching, be it kfunc or BPF helpers.
An example is the following:
struct foo {
struct bar b;
};struct foo *v = acq_foo();rel_bar(&v->b); // btf_struct_ids_match fails btf_types_are_same, then
// retries with first member type and succeeds, while // it should fail.
Hence, don't walk the struct and only rely on btf_types_are_same for strict mode. All users of strict mode must be dealing with zero offset anyway, since otherwise they would want the struct to be walked.
--- include/linux/bpf.h | 3 ++- kernel/bpf/btf.c | 14 ++++++++++---- kernel/bpf/verifier.c | 18 +++++++++++++++--- 3 files changed, 27 insertions(+), 8 deletions(-)
include/linux/bpf.h +2 -1
@@ -1748,7 +1748,8 @@ int btf_struct_access(struct bpf_verifier_log *log, const struct btf *btf, u32 *next_btf_id, enum bpf_type_flag *flag);bool btf_struct_ids_match (struct bpf_verifier_log *log,const struct btf *btf,u32 id,int off, -const struct btf *need_btf,u32 need_type_id); +const struct btf *need_btf,u32 need_type_id, +bool strict); int btf_distill_func_proto(struct bpf_verifier_log *log, struct btf *btf,
kernel/bpf/btf.c +10 -4
@@ -5746,7 +5746,8 @@ static bool btf_types_are_same(const struct btf *btf1, u32 id1,bool btf_struct_ids_match (struct bpf_verifier_log *log,const struct btf *btf,u32 id,int off, -const struct btf *need_btf,u32 need_type_id) +const struct btf *need_btf,u32 need_type_id, +bool strict) {const struct btf_type *type;enum bpf_type_flag flag;
@@ -5755,7 +5756,12 @@ bool btf_struct_ids_match(struct bpf_verifier_log *log,/* Are we already done? */ if (off ==0 &&btf_types_are_same (btf, id, need_btf, need_type_id))return true; - +/* In case of strict type match, we do not walk struct, the top level +* type match must succeed. When strict is true, off should have already +* been 0. +*/ +if (strict) +return false; again: type =btf_type_by_id (btf, id);if (!type)
@@ -6197,7 +6203,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env,return -EINVAL ; }if (!btf_struct_ids_match (log, btf, ref_id,0 , off_desc->kptr.btf, - off_desc->kptr.btf_id)) { + off_desc->kptr.btf_id, true)) {bpf_log (log,"kernel function %s args#%d expected pointer to %s %s\n" , func_name, i,btf_type_str (ref_t), ref_tname);return -EINVAL ;
@@ -6250,7 +6256,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, reg_ref_tname =btf_name_by_offset (reg_btf, reg_ref_t->name_off);if (!btf_struct_ids_match (log, reg_btf, reg_ref_id, - reg->off, btf, ref_id)) { + reg->off, btf, ref_id, rel && reg->ref_obj_id)) { bpf_log(log, "kernel function %s args#%d expected pointer to %s %s but R%d has a pointer to %s %s\n", func_name, i, btf_type_str(ref_t), ref_tname,
kernel/bpf/verifier.c +15 -3
@@ -3551,10 +3551,14 @@ static int map_kptr_match_type(struct bpf_verifier_env *env,* // to match type * * In the kptr_ref case, check_func_arg_reg_off already ensures reg->off -* is zero. +* is zero. We must also ensure that btf_struct_ids_match does not walk +* the struct to match type against first member of struct, i.e. reject +* second case from above. Hence, when type is BPF_KPTR_REF, we set +* strict mode to true for type match. */ if (!btf_struct_ids_match (&env->log, reg->btf, reg->btf_id, reg->off, - off_desc->kptr.btf, off_desc->kptr.btf_id)) + off_desc->kptr.btf, off_desc->kptr.btf_id, + off_desc->type ==BPF_KPTR_REF )) goto bad_type;return 0 ; bad_type:
@@ -5593,6 +5597,13 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno, found:if (reg->type ==PTR_TO_BTF_ID ) { +/* For bpf_sk_release, it needs to match against first member +* 'struct sock_common', hence make an exception for it. This +* allows bpf_sk_release to work for multiple socket types. +*/ +bool strict_type_match =arg_type_is_release (arg_type) && + meta->func_id != BPF_FUNC_sk_release; +if (!arg_btf_id) {if (!compatible->btf_id) {verbose (env,"verifier internal error: missing arg compatible BTF ID\n" );
@@ -5605,7 +5616,8 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno,if (map_kptr_match_type (env, meta->kptr_off_desc, reg, regno))return -EACCES ; }else if (!btf_struct_ids_match (&env->log, reg->btf, reg->btf_id, reg->off, - btf_vmlinux, *arg_btf_id)) { + btf_vmlinux, *arg_btf_id, + strict_type_match)) {verbose (env,"R%d is of type %s but %s is expected\n" , regno,kernel_type_name (reg->btf, reg->btf_id),kernel_type_name (btf_vmlinux, *arg_btf_id));
Signature · 1 line
2.35.1
Kumar Kartikeya Dwivedi
maintainerPATCH v6 10/13 libbpf: Add kptr type tag macros to bpf_helpers.h
In reply to Kumar Kartikeya DwivediInclude convenience definitions:__kptr: Unreferenced kptr__kptr_ref: Referenced kptr
Users can use them to tag the pointer type meant to be used with the new support directly in the map value definition.
--- tools/lib/bpf/bpf_helpers.h | 7 +++++++ 1 file changed, 7 insertions(+)
tools/lib/bpf/bpf_helpers.h +7 -0
@@ -149,6 +149,13 @@ enum libbpf_tristate {#define __kconfig __attribute__((section(".kconfig")))#define __ksym __attribute__((section(".ksyms"))) +#if __has_attribute (btf_type_tag) +#define __kptr __attribute__((btf_type_tag("kptr"))) +#define __kptr_ref __attribute__((btf_type_tag("kptr_ref"))) +#else +#define __kptr +#define __kptr_ref +#endif #ifndef ___bpf_concat#define ___bpf_concat (a, b) a ## b
Signature · 1 line
2.35.1
Kumar Kartikeya Dwivedi
maintainerPATCH v6 11/13 selftests/bpf: Add C tests for kptr
In reply to Kumar Kartikeya DwivediThis uses the __kptr and __kptr_ref macros as well, and tries to test the stuff that is supposed to work, since we have negative tests in test_verifier suite. Also include some code to test map-in-map support, such that the inner_map_meta matches the kptr_off_tab of map added as element.
--- .../selftests/bpf/prog_tests/map_kptr.c | 37 ++++ tools/testing/selftests/bpf/progs/map_kptr.c | 190 ++++++++++++++++++ 2 files changed, 227 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/map_kptr.c create mode 100644 tools/testing/selftests/bpf/progs/map_kptr.c
tools/testing/selftests/bpf/prog_tests/map_kptr.c new file +37 -0
@@ -0,0 +1,37 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <test_progs.h> + +#include "map_kptr.skel.h" + +void test_map_kptr (void ) +{ +struct map_kptr *skel; +int key =0 , ret; +char buf[24 ]; + + skel =map_kptr__open_and_load (); +if (!ASSERT_OK_PTR (skel,"map_kptr__open_and_load" )) +return ; + + ret =bpf_map_update_elem (bpf_map__fd (skel->maps.array_map), &key, buf,0 ); +ASSERT_OK (ret,"array_map update" ); + ret =bpf_map_update_elem (bpf_map__fd (skel->maps.array_map), &key, buf,0 ); +ASSERT_OK (ret,"array_map update2" ); + + ret =bpf_map_update_elem (bpf_map__fd (skel->maps.hash_map), &key, buf,0 ); +ASSERT_OK (ret,"hash_map update" ); + ret =bpf_map_delete_elem (bpf_map__fd (skel->maps.hash_map), &key); +ASSERT_OK (ret,"hash_map delete" ); + + ret =bpf_map_update_elem (bpf_map__fd (skel->maps.hash_malloc_map), &key, buf,0 ); +ASSERT_OK (ret,"hash_malloc_map update" ); + ret =bpf_map_delete_elem (bpf_map__fd (skel->maps.hash_malloc_map), &key); +ASSERT_OK (ret,"hash_malloc_map delete" ); + + ret =bpf_map_update_elem (bpf_map__fd (skel->maps.lru_hash_map), &key, buf,0 ); +ASSERT_OK (ret,"lru_hash_map update" ); + ret =bpf_map_delete_elem (bpf_map__fd (skel->maps.lru_hash_map), &key); +ASSERT_OK (ret,"lru_hash_map delete" ); + +map_kptr__destroy (skel); +}
tools/testing/selftests/bpf/progs/map_kptr.c new file +190 -0
190 lines not on this page · raw message
Signature · 1 line
2.35.1
Kumar Kartikeya Dwivedi
maintainerPATCH v6 12/13 selftests/bpf: Add verifier tests for kptr
In reply to Kumar Kartikeya DwivediReuse bpf_prog_test functions to test the support for PTR_TO_BTF_ID in BPF map case, including some tests that verify implementation sanity and corner cases.
--- net/bpf/test_run.c | 45 +- tools/testing/selftests/bpf/test_verifier.c | 55 +- .../testing/selftests/bpf/verifier/map_kptr.c | 469 ++++++++++++++++++ 3 files changed, 562 insertions(+), 7 deletions(-) create mode 100644 tools/testing/selftests/bpf/verifier/map_kptr.c
net/bpf/test_run.c +40 -5
@@ -584,6 +584,12 @@ noinline void bpf_kfunc_call_memb_release(struct prog_test_member *p) { } +noinline struct prog_test_ref_kfunc * +bpf_kfunc_call_test_kptr_get (struct prog_test_ref_kfunc **p,int a,int b) +{ +return &prog_test_struct; +} +struct prog_test_pass1 {int x0;struct {
@@ -669,6 +675,7 @@ BTF_ID(func, bpf_kfunc_call_test3) BTF_ID(func, bpf_kfunc_call_test_acquire) BTF_ID(func, bpf_kfunc_call_test_release) BTF_ID(func, bpf_kfunc_call_memb_release) +BTF_ID(func, bpf_kfunc_call_test_kptr_get) BTF_ID(func, bpf_kfunc_call_test_pass_ctx) BTF_ID(func, bpf_kfunc_call_test_pass1)BTF_ID (func, bpf_kfunc_call_test_pass2)
@@ -682,6 +689,7 @@ BTF_SET_END(test_sk_check_kfunc_ids) BTF_SET_START(test_sk_acquire_kfunc_ids) BTF_ID(func, bpf_kfunc_call_test_acquire) +BTF_ID(func, bpf_kfunc_call_test_kptr_get) BTF_SET_END(test_sk_acquire_kfunc_ids) BTF_SET_START(test_sk_release_kfunc_ids)
@@ -691,8 +699,13 @@ BTF_SET_END(test_sk_release_kfunc_ids)BTF_SET_START (test_sk_ret_null_kfunc_ids)BTF_ID (func ,bpf_kfunc_call_test_acquire ) +BTF_ID (func, bpf_kfunc_call_test_kptr_get)BTF_SET_END (test_sk_ret_null_kfunc_ids) +BTF_SET_START (test_sk_kptr_acquire_kfunc_ids) +BTF_ID (func, bpf_kfunc_call_test_kptr_get) +BTF_SET_END (test_sk_kptr_acquire_kfunc_ids) + static void *bpf_test_init(const union bpf_attr *kattr, u32 user_size, u32 size, u32 headroom, u32 tailroom) {
@@ -1579,14 +1592,36 @@ int bpf_prog_test_run_syscall(struct bpf_prog *prog,static const struct btf_kfunc_id_set bpf_prog_test_kfunc_set = { .owner =THIS_MODULE , - .check_set = &test_sk_check_kfunc_ids, - .acquire_set = &test_sk_acquire_kfunc_ids, - .release_set = &test_sk_release_kfunc_ids, - .ret_null_set = &test_sk_ret_null_kfunc_ids, + .check_set = &test_sk_check_kfunc_ids, + .acquire_set = &test_sk_acquire_kfunc_ids, + .release_set = &test_sk_release_kfunc_ids, + .ret_null_set = &test_sk_ret_null_kfunc_ids, + .kptr_acquire_set = &test_sk_kptr_acquire_kfunc_ids }; +BTF_ID_LIST (bpf_prog_test_dtor_kfunc_ids) +BTF_ID(struct, prog_test_ref_kfunc) +BTF_ID (func, bpf_kfunc_call_test_release) +BTF_ID (struct, prog_test_member) +BTF_ID (func, bpf_kfunc_call_memb_release) + static int__init bpf_prog_test_run_init (void) { -return register_btf_kfunc_id_set (BPF_PROG_TYPE_SCHED_CLS , &bpf_prog_test_kfunc_set); +const struct btf_id_dtor_kfunc bpf_prog_test_dtor_kfunc[] = { + { + .btf_id = bpf_prog_test_dtor_kfunc_ids[0 ], + .kfunc_btf_id = bpf_prog_test_dtor_kfunc_ids[1 ] + }, + { + .btf_id = bpf_prog_test_dtor_kfunc_ids[2 ], + .kfunc_btf_id = bpf_prog_test_dtor_kfunc_ids[3 ], + }, + }; +int ret; + + ret =register_btf_kfunc_id_set (BPF_PROG_TYPE_SCHED_CLS , &bpf_prog_test_kfunc_set); +return ret ?:register_btf_id_dtor_kfuncs (bpf_prog_test_dtor_kfunc, +ARRAY_SIZE (bpf_prog_test_dtor_kfunc), +THIS_MODULE ); }late_initcall (bpf_prog_test_run_init);
tools/testing/selftests/bpf/test_verifier.c +53 -2
98 lines not on this page · raw message
tools/testing/selftests/bpf/verifier/map_kptr.c new file +469 -0
469 lines not on this page · raw message
Signature · 1 line
2.35.1
Re: [PATCH bpf-next v6 12/13] selftests/bpf: Add verifier tests for kptr
Quoted 63 lines · Kumar Kartikeya Dwivedi
Reuse
bpf_prog_testfunctions to test the support forPTR_TO_BTF_IDin BPF map case, including some tests that verify implementation sanity and corner cases.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- net/bpf/test_run.c | 45 +- tools/testing/selftests/bpf/test_verifier.c | 55 +- .../testing/selftests/bpf/verifier/map_kptr.c | 469 ++++++++++++++++++ 3 files changed, 562 insertions(+), 7 deletions(-) create mode 100644 tools/testing/selftests/bpf/verifier/map_kptr.cnet/bpf/test_run.c +13
-0@@ -584,6 +584,12 @@ noinline void bpf_kfunc_call_memb_release(struct prog_test_member *p) { } +noinline struct prog_test_ref_kfunc * +bpf_kfunc_call_test_kptr_get (struct prog_test_ref_kfunc **p,int a,int b) +{ +return &prog_test_struct; +} +struct prog_test_pass1 {int x0;struct {@@ -669,6 +675,7 @@ BTF_ID(func, bpf_kfunc_call_test3) BTF_ID(func, bpf_kfunc_call_test_acquire) BTF_ID(func, bpf_kfunc_call_test_release) BTF_ID(func, bpf_kfunc_call_memb_release) +BTF_ID(func, bpf_kfunc_call_test_kptr_get) BTF_ID(func, bpf_kfunc_call_test_pass_ctx) BTF_ID(func, bpf_kfunc_call_test_pass1)BTF_ID (func, bpf_kfunc_call_test_pass2)@@ -682,6 +689,7 @@ BTF_SET_END(test_sk_check_kfunc_ids) BTF_SET_START(test_sk_acquire_kfunc_ids) BTF_ID(func, bpf_kfunc_call_test_acquire) +BTF_ID(func, bpf_kfunc_call_test_kptr_get) BTF_SET_END(test_sk_acquire_kfunc_ids) BTF_SET_START(test_sk_release_kfunc_ids)@@ -691,8 +699,13 @@ BTF_SET_END(test_sk_release_kfunc_ids)BTF_SET_START (test_sk_ret_null_kfunc_ids)BTF_ID (func ,bpf_kfunc_call_test_acquire ) +BTF_ID (func, bpf_kfunc_call_test_kptr_get)BTF_SET_END (test_sk_ret_null_kfunc_ids) +BTF_SET_START (test_sk_kptr_acquire_kfunc_ids) +BTF_ID (func, bpf_kfunc_call_test_kptr_get) +BTF_SET_END (test_sk_kptr_acquire_kfunc_ids) + static void *bpf_test_init(const union bpf_attr *kattr, u32 user_size, u32 size, u32 headroom, u32 tailroom) {@@ -1579,14 +1592,36 @@ int bpf_prog_test_run_syscall(struct bpf_prog *prog,static const struct btf_kfunc_id_set bpf_prog_test_kfunc_set = { .owner =THIS_MODULE ,
net/bpf/test_run.c int bpf_prog_test_run_syscall(struct bpf_prog *prog, · line 1600
Quoted 10 lines
net/bpf/test_run.c +5
-4- .check_set = &test_sk_check_kfunc_ids, - .acquire_set = &test_sk_acquire_kfunc_ids, - .release_set = &test_sk_release_kfunc_ids, - .ret_null_set = &test_sk_ret_null_kfunc_ids, + .check_set = &test_sk_check_kfunc_ids, + .acquire_set = &test_sk_acquire_kfunc_ids, + .release_set = &test_sk_release_kfunc_ids, + .ret_null_set = &test_sk_ret_null_kfunc_ids, + .kptr_acquire_set = &test_sk_kptr_acquire_kfunc_ids };
This hunk probably should have been in the previous patch, but since it's not affecting bisect I left it as-is.
net/bpf/test_run.c int bpf_prog_test_run_syscall(struct bpf_prog *prog, · line 1606
Quoted 5 lines
net/bpf/test_run.c +5
-0+BTF_ID_LIST(bpf_prog_test_dtor_kfunc_ids) +BTF_ID(struct, prog_test_ref_kfunc) +BTF_ID(func, bpf_kfunc_call_test_release) +BTF_ID(struct, prog_test_member) +BTF_ID(func, bpf_kfunc_call_memb_release)
dtor of prog_test_member doesn't seem to be used ?
Please improve dtor and kptr_get test methods for
struct prog_test_ref_kfunc and prog_test_member to do the real refcnting. Empty methods are not testing things fully.
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v6 12/13] selftests/bpf: Add verifier tests for kptr
Quoted 78 lines · Alexei Starovoitov
Quoted 63 lines · Kumar Kartikeya Dwivedi
Reuse
bpf_prog_testfunctions to test the support forPTR_TO_BTF_IDin BPF map case, including some tests that verify implementation sanity and corner cases.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- net/bpf/test_run.c | 45 +- tools/testing/selftests/bpf/test_verifier.c | 55 +- .../testing/selftests/bpf/verifier/map_kptr.c | 469 ++++++++++++++++++ 3 files changed, 562 insertions(+), 7 deletions(-) create mode 100644 tools/testing/selftests/bpf/verifier/map_kptr.cnet/bpf/test_run.c +13
-0@@ -584,6 +584,12 @@ noinline void bpf_kfunc_call_memb_release(struct prog_test_member *p) { } +noinline struct prog_test_ref_kfunc * +bpf_kfunc_call_test_kptr_get (struct prog_test_ref_kfunc **p,int a,int b) +{ +return &prog_test_struct; +} +struct prog_test_pass1 {int x0;struct {@@ -669,6 +675,7 @@ BTF_ID(func, bpf_kfunc_call_test3) BTF_ID(func, bpf_kfunc_call_test_acquire) BTF_ID(func, bpf_kfunc_call_test_release) BTF_ID(func, bpf_kfunc_call_memb_release) +BTF_ID(func, bpf_kfunc_call_test_kptr_get) BTF_ID(func, bpf_kfunc_call_test_pass_ctx) BTF_ID(func, bpf_kfunc_call_test_pass1)BTF_ID (func, bpf_kfunc_call_test_pass2)@@ -682,6 +689,7 @@ BTF_SET_END(test_sk_check_kfunc_ids) BTF_SET_START(test_sk_acquire_kfunc_ids) BTF_ID(func, bpf_kfunc_call_test_acquire) +BTF_ID(func, bpf_kfunc_call_test_kptr_get) BTF_SET_END(test_sk_acquire_kfunc_ids) BTF_SET_START(test_sk_release_kfunc_ids)@@ -691,8 +699,13 @@ BTF_SET_END(test_sk_release_kfunc_ids)BTF_SET_START (test_sk_ret_null_kfunc_ids)BTF_ID (func ,bpf_kfunc_call_test_acquire ) +BTF_ID (func, bpf_kfunc_call_test_kptr_get)BTF_SET_END (test_sk_ret_null_kfunc_ids) +BTF_SET_START (test_sk_kptr_acquire_kfunc_ids) +BTF_ID (func, bpf_kfunc_call_test_kptr_get) +BTF_SET_END (test_sk_kptr_acquire_kfunc_ids) + static void *bpf_test_init(const union bpf_attr *kattr, u32 user_size, u32 size, u32 headroom, u32 tailroom) {@@ -1579,14 +1592,36 @@ int bpf_prog_test_run_syscall(struct bpf_prog *prog,static const struct btf_kfunc_id_set bpf_prog_test_kfunc_set = { .owner =THIS_MODULE ,net/bpf/test_run.c int bpf_prog_test_run_syscall(struct bpf_prog *prog, · line 1600
Quoted 10 lines
net/bpf/test_run.c +5
-4- .check_set = &test_sk_check_kfunc_ids, - .acquire_set = &test_sk_acquire_kfunc_ids, - .release_set = &test_sk_release_kfunc_ids, - .ret_null_set = &test_sk_ret_null_kfunc_ids, + .check_set = &test_sk_check_kfunc_ids, + .acquire_set = &test_sk_acquire_kfunc_ids, + .release_set = &test_sk_release_kfunc_ids, + .ret_null_set = &test_sk_ret_null_kfunc_ids, + .kptr_acquire_set = &test_sk_kptr_acquire_kfunc_ids };This hunk probably should have been in the previous patch, but since it's not affecting bisect I left it as-is.
Quoted 8 lines
net/bpf/test_run.c int bpf_prog_test_run_syscall(struct bpf_prog *prog, · line 1606
Quoted 5 lines
net/bpf/test_run.c +5
-0+BTF_ID_LIST(bpf_prog_test_dtor_kfunc_ids) +BTF_ID(struct, prog_test_ref_kfunc) +BTF_ID(func, bpf_kfunc_call_test_release) +BTF_ID(struct, prog_test_member) +BTF_ID(func, bpf_kfunc_call_memb_release)dtor of
prog_test_memberdoesn't seem to be used ?
It is necessary to be able to embed prog_test_member struct as a referenced kptr in a map value, so it is still called by the map implementation's free path. But since this is just used for verifier tests related to type matching when non-zero offset is involved, both acquire and release are dummy functions.
Quoted 3 lines
Please improve dtor and
kptr_gettest methods for
structprog_test_ref_kfuncandprog_test_memberto do the real refcnting. Empty methods are not testing things fully.
I will leave out prog_test_member since it is not meant to be used at runtime, but I will add refcount_t to prog_test_ref_kfunc, and include tests for it.
--
Kartikeya
Kumar Kartikeya Dwivedi
maintainerPATCH v6 13/13 selftests/bpf: Add test for strict BTF type check
In reply to Kumar Kartikeya DwivediEnsure that the edge case where first member type was matched successfully even if it didn't match BTF type of register is caught and rejected by the verifier.
--- net/bpf/test_run.c | 22 +++++++++++++++++++- tools/testing/selftests/bpf/verifier/calls.c | 20 ++++++++++++++++++ 2 files changed, 41 insertions(+), 1 deletion(-)
net/bpf/test_run.c +21 -1
@@ -550,8 +550,13 @@ struct sock * noinline bpf_kfunc_call_test3(struct sock *sk)return sk; } +struct prog_test_member1 { +int a; +}; +struct prog_test_member { -u64 c; +struct prog_test_member1 m; +int c; };struct prog_test_ref_kfunc {
@@ -576,6 +581,12 @@ bpf_kfunc_call_test_acquire(unsigned long *scalar_ptr)return &prog_test_struct; } +noinline struct prog_test_member * +bpf_kfunc_call_memb_acquire (void ) +{ +return &prog_test_struct.memb; +} + noinline void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) { }
@@ -584,6 +595,10 @@ noinline void bpf_kfunc_call_memb_release(struct prog_test_member *p) { } +noinline void bpf_kfunc_call_memb1_release(struct prog_test_member1 *p) +{ +} + noinline struct prog_test_ref_kfunc *bpf_kfunc_call_test_kptr_get (struct prog_test_ref_kfunc **p,int a,int b) {
@@ -673,8 +688,10 @@ BTF_ID(func, bpf_kfunc_call_test1) BTF_ID(func, bpf_kfunc_call_test2) BTF_ID(func, bpf_kfunc_call_test3) BTF_ID(func, bpf_kfunc_call_test_acquire) +BTF_ID(func, bpf_kfunc_call_memb_acquire) BTF_ID(func, bpf_kfunc_call_test_release) BTF_ID(func, bpf_kfunc_call_memb_release) +BTF_ID(func, bpf_kfunc_call_memb1_release) BTF_ID(func, bpf_kfunc_call_test_kptr_get) BTF_ID(func, bpf_kfunc_call_test_pass_ctx)BTF_ID (func, bpf_kfunc_call_test_pass1)
@@ -689,16 +706,19 @@ BTF_SET_END(test_sk_check_kfunc_ids) BTF_SET_START(test_sk_acquire_kfunc_ids) BTF_ID(func, bpf_kfunc_call_test_acquire) +BTF_ID(func, bpf_kfunc_call_memb_acquire) BTF_ID(func, bpf_kfunc_call_test_kptr_get) BTF_SET_END(test_sk_acquire_kfunc_ids) BTF_SET_START(test_sk_release_kfunc_ids) BTF_ID(func, bpf_kfunc_call_test_release) BTF_ID(func, bpf_kfunc_call_memb_release) +BTF_ID(func, bpf_kfunc_call_memb1_release) BTF_SET_END(test_sk_release_kfunc_ids) BTF_SET_START(test_sk_ret_null_kfunc_ids) BTF_ID(func, bpf_kfunc_call_test_acquire) +BTF_ID(func, bpf_kfunc_call_memb_acquire) BTF_ID(func, bpf_kfunc_call_test_kptr_get) BTF_SET_END(test_sk_ret_null_kfunc_ids)
tools/testing/selftests/bpf/verifier/calls.c +20 -0
@@ -138,6 +138,26 @@ { "bpf_kfunc_call_memb_release", 8 }, }, }, +{ +"calls: invalid kfunc call: don't match first member type when passed to release kfunc" , + .insns = { +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,BPF_PSEUDO_KFUNC_CALL ,0 ,0 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,BPF_PSEUDO_KFUNC_CALL ,0 ,0 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = REJECT, + .errstr = "kernel function bpf_kfunc_call_memb1_release args#0 expected pointer", + .fixup_kfunc_btf_id = { + { "bpf_kfunc_call_memb_acquire", 1 }, + { "bpf_kfunc_call_memb1_release", 5 }, + }, +}, {"calls: invalid kfunc call: PTR_TO_BTF_ID with negative offset" , .insns = {
Signature · 1 line
2.35.1
Re: [PATCH bpf-next v6 13/13] selftests/bpf: Add test for strict BTF type check
Quoted 19 lines · Kumar Kartikeya Dwivedi
tools/testing/selftests/bpf/verifier/calls.c +10
-0@@ -138,6 +138,26 @@ { "bpf_kfunc_call_memb_release", 8 }, }, }, +{ +"calls: invalid kfunc call: don't match first member type when passed to release kfunc" , + .insns = { +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,BPF_PSEUDO_KFUNC_CALL ,0 ,0 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,BPF_PSEUDO_KFUNC_CALL ,0 ,0 ), + BPF_MOV64_IMM(BPF_REG_0, 0),
tools/testing/selftests/bpf/verifier/calls.c line 160
Quoted 10 lines
tools/testing/selftests/bpf/verifier/calls.c +10
-0+ BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .result = REJECT, + .errstr = "kernel function bpf_kfunc_call_memb1_release args#0 expected pointer", + .fixup_kfunc_btf_id = { + { "bpf_kfunc_call_memb_acquire", 1 }, + { "bpf_kfunc_call_memb1_release", 5 }, + }, +},
Please add negative C tests as well.
Consider using SEC("?tc") logic added by commit 0d7fefebea552 and put a bunch of bpf progs that should fail to load in one .c
Re: [PATCH bpf-next v6 13/13] selftests/bpf: Add test for strict BTF type check
Quoted 27 lines · Alexei Starovoitov
Quoted 25 lines · Kumar Kartikeya Dwivedi
tools/testing/selftests/bpf/verifier/calls.c +16
-0@@ -138,6 +138,26 @@ { "bpf_kfunc_call_memb_release", 8 }, }, }, +{ +"calls: invalid kfunc call: don't match first member type when passed to release kfunc" , + .insns = { +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,BPF_PSEUDO_KFUNC_CALL ,0 ,0 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,BPF_PSEUDO_KFUNC_CALL ,0 ,0 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = REJECT, + .errstr = "kernel function bpf_kfunc_call_memb1_release args#0 expected pointer", + .fixup_kfunc_btf_id = {
Quoted 8 lines
tools/testing/selftests/bpf/verifier/calls.c line 160
Quoted 4 lines
tools/testing/selftests/bpf/verifier/calls.c +4
-0+ { "bpf_kfunc_call_memb_acquire", 1 }, + { "bpf_kfunc_call_memb1_release", 5 }, + }, +},Please add negative C tests as well.
Consider usingSEC("?tc")logic added by commit0d7fefebea552and put a bunch of bpf progs that should fail to load in one .c
Kumar,
ping?
Are you still working on the follow up?
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v6 13/13] selftests/bpf: Add test for strict BTF type check
Quoted 38 lines · Alexei Starovoitov
Quoted 27 lines · Alexei Starovoitov
Quoted 25 lines · Kumar Kartikeya Dwivedi
tools/testing/selftests/bpf/verifier/calls.c +16
-0@@ -138,6 +138,26 @@ { "bpf_kfunc_call_memb_release", 8 }, }, }, +{ +"calls: invalid kfunc call: don't match first member type when passed to release kfunc" , + .insns = { +BPF_MOV64_IMM (BPF_REG_0 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,BPF_PSEUDO_KFUNC_CALL ,0 ,0 ), +BPF_JMP_IMM (BPF_JNE ,BPF_REG_0 ,0 ,1 ), +BPF_EXIT_INSN (), +BPF_MOV64_REG (BPF_REG_1 ,BPF_REG_0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,BPF_PSEUDO_KFUNC_CALL ,0 ,0 ), +BPF_MOV64_IMM (BPF_REG_0 ,0 ), + BPF_EXIT_INSN(), + }, + .prog_type =BPF_PROG_TYPE_SCHED_CLS , + .result = REJECT, + .errstr = "kernel function bpf_kfunc_call_memb1_release args#0 expected pointer", + .fixup_kfunc_btf_id = {Quoted 8 lines
tools/testing/selftests/bpf/verifier/calls.c line 160
Quoted 4 lines
tools/testing/selftests/bpf/verifier/calls.c +4
-0+ { "bpf_kfunc_call_memb_acquire", 1 }, + { "bpf_kfunc_call_memb1_release", 5 }, + }, +},Please add negative C tests as well.
Consider usingSEC("?tc")logic added by commit0d7fefebea552and put a bunch of bpf progs that should fail to load in one .c
Quoted 4 lines
Kumar,
ping?
Are you still working on the follow up?
Yes, just got a bit stalled due to some IRL stuff... I'll post the follow up set tomorrow.
--
Kartikeya
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 10 lines · On Mon, 25 Apr 2022 03:18:48 +0530 you wrote:
This set enables storing pointers of a certain type in BPF map, and extends the verifier to enforce type safety and lifetime correctness properties.
The infrastructure being added is generic enough for allowing storing any kind of pointers whose type is available using BTF (user or kernel) in the future (e.g. strongly typed memory allocation in BPF program), which are internally tracked in the verifier as
PTR_TO_BTF_ID, but for now the series limits them to two kinds of pointers obtained from the kernel.[...]
Here is the summary with links:
- [bpf-next,v6,01/13] bpf: Allow storing unreferenced kptr in map
https://git.kernel.org/bpf/bpf-next/c/61df10c7799e - [bpf-next,v6,02/13] bpf: Tag argument to be released in bpf_func_proto https://git.kernel.org/bpf/bpf-next/c/8f14852e8911 - [bpf-next,v6,03/13] bpf: Allow storing referenced kptr in map https://git.kernel.org/bpf/bpf-next/c/c0a5a21c25f3 - [bpf-next,v6,04/13] bpf: Prevent escaping of kptr loaded from maps https://git.kernel.org/bpf/bpf-next/c/6efe152d4061 - [bpf-next,v6,05/13] bpf: Adapt copy_map_value for multiple offset case https://git.kernel.org/bpf/bpf-next/c/4d7d7f69f4b1 - [bpf-next,v6,06/13] bpf: Populate pairs of btf_id and destructor kfunc in btf https://git.kernel.org/bpf/bpf-next/c/5ce937d613a4 - [bpf-next,v6,07/13] bpf: Wire up freeing of referenced kptr https://git.kernel.org/bpf/bpf-next/c/14a324f6a67e - [bpf-next,v6,08/13] bpf: Teach verifier about kptr_get kfunc helpers https://git.kernel.org/bpf/bpf-next/c/a1ef19599652 - [bpf-next,v6,09/13] bpf: Make BTF type match stricter for release arguments https://git.kernel.org/bpf/bpf-next/c/2ab3b3808eb1 - [bpf-next,v6,10/13] libbpf: Add kptr type tag macros to bpf_helpers.h https://git.kernel.org/bpf/bpf-next/c/ef89654f2bc7 - [bpf-next,v6,11/13] selftests/bpf: Add C tests for kptr https://git.kernel.org/bpf/bpf-next/c/2cbc469a6fc3 - [bpf-next,v6,12/13] selftests/bpf: Add verifier tests for kptr https://git.kernel.org/bpf/bpf-next/c/05a945deefaa - [bpf-next,v6,13/13] selftests/bpf: Add test for strict BTF type check https://git.kernel.org/bpf/bpf-next/c/792c0a345f0e
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html