Local kptrs, BPF linked lists
Subsystems (9)
Kumar Kartikeya Dwivedi
maintainerv5 cover Local kptrs, BPF linked lists
This series introduces user defined BPF objects, by introducing local kptrs. These are kptrs (strongly typed pointers) that refer to objects of a user defined type, hence called "local" kptrs. This allows BPF programs to allocate their own objects, build their own object hierarchies, and use the basic building blocks provided by BPF runtime to build their own data structures flexibly.
Then, we introduce the support for single ownership BPF linked lists, which can be put inside BPF maps, or local kptrs, and hold such allocated local kptrs as elements. It works as an instrusive collection, which is done to allow making local kptrs part of multiple data structures at the same time in the future.
The eventual goal of this and future patches is to allow one to do some limited form of kernel style programming in BPF C, and allow programmers to build their own complex data structures flexibly out of basic building blocks.
The key difference will be that such programs are verified to be safe, preserve runtime integrity of the system, and are proven to be bug free as far as the invariants of BPF specific APIs are concerned.
One immediate use case that will be using the entire infrastructure this series is introducing will be managing percpu NMI safe linked lists inside BPF programs.
The other use case this will serve in the near future will be linking kernel structures like XDP frame and sk_buff directly into user data structures (rbtree, pifomap, etc.) for packet queueing. This will follow single ownership concept included in this series.
The user has complete control of the internal locking, and hence also the batching of operations for each critical section.
The features are:
- Local kptrs - User defined kernel objects.
- bpf_obj_new, bpf_obj_drop to allocate and free them.
- Single ownership BPF linked lists.
- Support for them in BPF maps.
- Support for them in local kptrs.
- Global spin locks.
- Spin locks inside local kptrs.
Some other notable things:
- Completely static verification of locking.
- Kfunc argument handling has been completely reworked.
- Argument rewriting support for kfuncs.
- Search pruning now understands non-size precise registers.
- A new bpf_experimental.h header as a dumping ground for these APIs.
Any functionality exposed in this series is NOT part of UAPI. It is only available through use of kfuncs, and structs that can be added to map value may also change their size or name in the future. Hence, every feature in this series must be considered experimental.
Follow-ups:
-----------
* Support for kptrs (local and kernel) in local storage and percpu maps + kptr tests
* Fixes for helper access checks rebasing on top of this series
Next steps:
-----------
* NMI safe percpu single ownership linked lists (using local_t protection).
* Lockless linked lists.
* Allow RCU protected local kptrs. This then allows RCU protected list lookups, since spinlock protection for readers does not scale.
* Introduce bpf_refcount for local kptrs, shared ownership.
* Introduce shared ownership linked lists.
* Documentation.
Changelog:
----------
v4 -> v5
v4: https://lore.kernel.org/bpf/[email protected]
* Add a lot more selftests (failure, success, runtime, BTF)
* Make sure series is bisect friendly
* Move list draining out of spin lock
* This exposed an issue where bpf_mem_free can now be called in map_free path without migrate_disable, also fixed that.
* Rename MEM_ALLOC -> MEM_RINGBUF, MEM_TYPE_LOCAL -> MEM_ALLOC (Alexei)
* Group lock identity into a struct active_lock { ptr, id } (Dave)
* Split set_release_on_unlock logic into separate patch (Alexei)
v3 -> v4
v3: https://lore.kernel.org/bpf/[email protected]
* Fix compiler error for !CONFIG_BPF_SYSCALL (Kernel Test Robot)
* Fix error due to BUILD_BUG_ON on 32-bit platforms (Kernel Test Robot)
v2 -> v3
v2: https://lore.kernel.org/bpf/[email protected]
* Add ack from Dave for patch 5
* Rename btf_type_fields -> btf_record, btf_type_fields_off -> btf_field_offs, rename functions similarly (Alexei)
* Remove 'kind' component from contains declaration tag (Alexei)
* Move bpf_list_head, bpf_list_node definitions to UAPI bpf.h (Alexei)
* Add note in commit log about modifying btf_struct_access API (Dave)
* Downgrade WARN_ON_ONCE to verbose(env, "...") and return -EFAULT (Dave)
* Add type_is_local_kptr wrapper to avoid noisy checks (Dave)
* Remove unused flags parameter from bpf_kptr_new (Alexei)
* Rename bpf_kptr_new -> bpf_obj_new, bpf_kptr_drop -> bpf_obj_drop (Alexei)
* Reword comment in ref_obj_id_set_release_on_unlock (Dave)
* Fix return type of ref_obj_id_set_release_on_unlock (Dave)
* Introduce is_bpf_list_api_kfunc to dedup checks (Dave)
* Disallow BPF_WRITE to untrusted local kptrs
* Add details about soundness of check_reg_allocation_locked logic
* List untrusted local kptrs for PROBE_MEM handling
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
* Rebase on bpf-next to resolve merge conflict in DENYLIST.s390x
* Fix a couple of mental lapses in bpf_list_head_free
RFC v1 -> v1
RFC v1: https://lore.kernel.org/bpf/[email protected]
* Mostly a complete rewrite of BTF parsing, refactor existing code (Kartikeya)
* Rebase kfunc rewrite for bpf-next, add support for more changes
* Cache type metadata in BTF to avoid recomputation inside verifier (Kartikeya)
* Remove __kernel tag, make things similar to map values, reserve bpf_ prefix
* bpf_kptr_new, bpf_kptr_drop
* Rename precision state enum values (Alexei)
* Drop explicit constructor/destructor support (Alexei)
* Rewrite code for constructing/destructing objects and offload to runtime
* Minimize duplication in bpf_map_value_off_desc handling (Alexei)
* Expose global memory allocator (Alexei)
* Address other nits from Alexei
* Split out local kptrs in maps, more kptrs in maps support into a follow up
Links:
------
* Dave's BPF RB-Tree RFC series
v1 (Discussion thread)
https://lore.kernel.org/bpf/[email protected] v2 (With support for static locks) https://lore.kernel.org/bpf/[email protected]
* BPF Linked Lists Discussion
https://lore.kernel.org/bpf/CAP01T74U30+yeBHEgmgzTJ-XYxZ0zj71kqCDJtTH9YQNfTK+Xw@mail.gmail.com
* BPF Memory Allocator from Alexei
https://lore.kernel.org/bpf/[email protected]
* BPF Memory Allocator UAPI Discussion
https://lore.kernel.org/bpf/[email protected]
Kumar Kartikeya Dwivedi (25):
bpf: Remove BPF_MAP_OFF_ARR_MAX
bpf: Fix copy_map_value, zero_map_value
bpf: Support bpf_list_head in map values
bpf: Rename RET_PTR_TO_ALLOC_MEM
bpf: Rename MEM_ALLOC to MEM_RINGBUF
bpf: Introduce local kptrs
bpf: Recognize bpf_{spin_lock,list_head,list_node} in local kptrs
bpf: Verify ownership relationships for user BTF types
bpf: Allow locking bpf_spin_lock in local kptr
bpf: Allow locking bpf_spin_lock global variables
bpf: Allow locking bpf_spin_lock in inner map values
bpf: Rewrite kfunc argument handling
bpf: Drop kfunc bits from btf_check_func_arg_match
bpf: Support constant scalar arguments for kfuncs
bpf: Teach verifier about non-size constant arguments
bpf: Introduce bpf_obj_new
bpf: Introduce bpf_obj_drop
bpf: Permit NULL checking pointer with non-zero fixed offset
bpf: Introduce single ownership BPF linked list API
bpf: Add 'release on unlock' logic for bpf_list_push_{front,back} selftests/bpf: Add __contains macro to bpf_experimental.h selftests/bpf: Update spinlock selftest
selftests/bpf: Add failure test cases for spin lock pairing selftests/bpf: Add BPF linked list API tests selftests/bpf: Add BTF sanity tests
Documentation/bpf/kfuncs.rst | 30 + include/linux/bpf.h | 113 +- include/linux/bpf_verifier.h | 24 +- include/linux/btf.h | 67 +- include/linux/filter.h | 8 +- include/uapi/linux/bpf.h | 10 + kernel/bpf/btf.c | 808 +++++----- kernel/bpf/core.c | 16 + kernel/bpf/helpers.c | 143 +- kernel/bpf/map_in_map.c | 5 - kernel/bpf/ringbuf.c | 6 +- kernel/bpf/syscall.c | 30 +- kernel/bpf/verifier.c | 1346 +++++++++++++++-- net/bpf/bpf_dummy_struct_ops.c | 14 +- net/core/filter.c | 34 +- net/ipv4/bpf_tcp_ca.c | 13 +- net/netfilter/nf_conntrack_bpf.c | 17 +- tools/include/uapi/linux/bpf.h | 10 + tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../testing/selftests/bpf/bpf_experimental.h | 68 + .../testing/selftests/bpf/prog_tests/dynptr.c | 2 +- .../bpf/prog_tests/kfunc_dynptr_param.c | 2 +- .../selftests/bpf/prog_tests/linked_list.c | 524 +++++++ .../selftests/bpf/prog_tests/spin_lock.c | 133 ++ .../selftests/bpf/prog_tests/spinlock.c | 45 - .../testing/selftests/bpf/progs/linked_list.c | 370 +++++ .../testing/selftests/bpf/progs/linked_list.h | 56 + .../selftests/bpf/progs/linked_list_fail.c | 581 +++++++ .../selftests/bpf/progs/test_spin_lock.c | 4 +- .../selftests/bpf/progs/test_spin_lock_fail.c | 204 +++ tools/testing/selftests/bpf/verifier/calls.c | 2 +- .../selftests/bpf/verifier/ref_tracking.c | 4 +- .../testing/selftests/bpf/verifier/ringbuf.c | 2 +- .../selftests/bpf/verifier/spill_fill.c | 2 +- 34 files changed, 4012 insertions(+), 682 deletions(-) create mode 100644 tools/testing/selftests/bpf/bpf_experimental.h create mode 100644 tools/testing/selftests/bpf/prog_tests/linked_list.c create mode 100644 tools/testing/selftests/bpf/prog_tests/spin_lock.c delete mode 100644 tools/testing/selftests/bpf/prog_tests/spinlock.c create mode 100644 tools/testing/selftests/bpf/progs/linked_list.c create mode 100644 tools/testing/selftests/bpf/progs/linked_list.h create mode 100644 tools/testing/selftests/bpf/progs/linked_list_fail.c create mode 100644 tools/testing/selftests/bpf/progs/test_spin_lock_fail.c
base-commit: 25906092edb4bcf94cb669bd1ed03a0ef2f4120c
prerequisite-patch-id: c763e9eecf8258840e7db20e4475c26f4c7800bc
prerequisite-patch-id: 553e0a3f3035d36a96bdcc798793cbecc519d3e7
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 1/25 bpf: Remove BPF_MAP_OFF_ARR_MAX
In f71b2f64177a ("bpf: Refactor map->off_arr handling"), map->off_arr was refactored to be btf_field_offs. The number of field offsets is equal to maximum possible fields limited by BTF_FIELDS_MAX. Hence, reuse BTF_FIELDS_MAX as spin_lock and timer no longer are to be handled specially for offset sorting, fix the comment, and remove incorrect WARN_ON as its rec->cnt can never exceed this value. The reason to keep separate constant was the it was always more 2 more than total kptrs. This is no longer the case.
--- include/linux/bpf.h | 9 ++++----- kernel/bpf/btf.c | 2 +- 2 files changed, 5 insertions(+), 6 deletions(-)
include/linux/bpf.h +4 -5
@@ -165,9 +165,8 @@ struct bpf_map_ops { };enum { -/* Support at most 8 pointers in a BTF type */ -BTF_FIELDS_MAX =10 , -BPF_MAP_OFF_ARR_MAX =BTF_FIELDS_MAX , +/* Support at most 10 fields in a BTF type */ +BTF_FIELDS_MAX =10 , };enum btf_field_type {
@@ -203,8 +202,8 @@ struct btf_record {struct btf_field_offs {u32 cnt; -u32 field_off[BPF_MAP_OFF_ARR_MAX ]; -u8 field_sz[BPF_MAP_OFF_ARR_MAX ]; +u32 field_off[BTF_FIELDS_MAX ]; +u8 field_sz[BTF_FIELDS_MAX ]; };struct bpf_map {
kernel/bpf/btf.c +1 -1
@@ -3584,7 +3584,7 @@ struct btf_field_offs *btf_parse_field_offs(struct btf_record *rec)u8 *sz;BUILD_BUG_ON (ARRAY_SIZE (foffs->field_off) !=ARRAY_SIZE (foffs->field_sz)); -if (IS_ERR_OR_NULL (rec) ||WARN_ON_ONCE (rec->cnt >sizeof (foffs->field_off))) +if (IS_ERR_OR_NULL (rec))return NULL ; foffs =kzalloc (sizeof (*foffs),GFP_KERNEL | __GFP_NOWARN);
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 2/25 bpf: Fix copy_map_value, zero_map_value
In reply to Kumar Kartikeya DwivediThe current offset needs to also skip over the already copied region in addition to the size of the next field. This case manifests where there are gaps between adjacent special fields.
It was observed that for a map value with size 48, having fields at:
off: 0, 16, 32 size: 4, 16, 16
The current code does:
memcpy(dst + 0, src + 0, 0)memcpy(dst + 4, src + 4, 12)memcpy(dst + 20, src + 20, 12)memcpy(dst + 36, src + 36, 12)
With the fix, it is done correctly as:
memcpy(dst + 0, src + 0, 0)memcpy(dst + 4, src + 4, 12)memcpy(dst + 32, src + 32, 0)memcpy(dst + 48, src + 48, 0)
--- include/linux/bpf.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/bpf.h +2 -2
@@ -360,7 +360,7 @@ static inline void bpf_obj_memcpy(struct btf_field_offs *foffs,u32 sz = next_off - curr_off;memcpy (dst + curr_off, src + curr_off, sz); - curr_off += foffs->field_sz[i]; + curr_off += foffs->field_sz[i] + sz; }memcpy (dst + curr_off, src + curr_off, size - curr_off); }
@@ -390,7 +390,7 @@ static inline void bpf_obj_memzero(struct btf_field_offs *foffs, void *dst, u32u32 sz = next_off - curr_off;memset (dst + curr_off,0 , sz); - curr_off += foffs->field_sz[i]; + curr_off += foffs->field_sz[i] + sz; }memset (dst + curr_off,0 , size - curr_off); }
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 3/25 bpf: Support bpf_list_head in map values
In reply to Kumar Kartikeya DwivediAdd the support on the map side to parse, recognize, verify, and build metadata table for a new special field of the type struct bpf_list_head. To parameterize the bpf_list_head for a certain value type and the list_node member it will accept in that value type, we use BTF declaration tags.
The definition of bpf_list_head in a map value will be done as follows:
struct foo {
struct bpf_list_node node;
int data;
};struct map_value {
struct bpf_list_head head __contains(foo, node);
};Then, the bpf_list_head only allows adding to the list 'head' using the bpf_list_node 'node' for the type struct foo.
The 'contains' annotation is a BTF declaration tag composed of four parts, "contains:name:node" where the name is then used to look up the type in the map BTF, with its kind hardcoded to BTF_KIND_STRUCT during the lookup. The node defines name of the member in this type that has the type struct bpf_list_node, which is actually used for linking into the linked list. For now, 'kind' part is hardcoded as struct.
This allows building intrusive linked lists in BPF, using container_of to obtain pointer to entry, while being completely type safe from the perspective of the verifier. The verifier knows exactly the type of the nodes, and knows that list helpers return that type at some fixed offset where the bpf_list_node member used for this list exists. The verifier also uses this information to disallow adding types that are not accepted by a certain list.
For now, no elements can be added to such lists. Support for that is coming in future patches, hence draining and freeing items is done with a TODO that will be resolved in a future patch.
Note that the bpf_list_head_free function moves the list out to a local variable under the lock and releases it, doing the actual draining of the list items outside the lock. While this helps with not holding the lock for too long pessimizing other concurrent list operations, it is also necessary for deadlock prevention: unless every function called in the critical section would be notrace, a fentry/fexit program could attach and call bpf_map_update_elem again on the map, leading to the same lock being acquired if the key matches and lead to a deadlock. While this requires some special effort on part of the BPF programmer to trigger and is highly unlikely to occur in practice, it is always better if we can avoid such a condition.
While notrace would prevent this, doing the draining outside the lock has advantages of its own, hence it is used to also fix the deadlock related problem.
--- include/linux/bpf.h | 17 ++++ include/uapi/linux/bpf.h | 10 +++ kernel/bpf/btf.c | 143 ++++++++++++++++++++++++++++++++- kernel/bpf/helpers.c | 32 ++++++++ kernel/bpf/syscall.c | 22 ++++- kernel/bpf/verifier.c | 7 ++ tools/include/uapi/linux/bpf.h | 10 +++ 7 files changed, 237 insertions(+), 4 deletions(-)
include/linux/bpf.h +17 -0
@@ -175,6 +175,7 @@ enum btf_field_type {BPF_KPTR_UNREF = (1 <<2 ),BPF_KPTR_REF = (1 <<3 ),BPF_KPTR =BPF_KPTR_UNREF |BPF_KPTR_REF , + BPF_LIST_HEAD = (1 << 4), }; struct btf_field_kptr {
@@ -184,11 +185,18 @@ struct btf_field_kptr {u32 btf_id; }; +struct btf_field_list_head { +struct btf *btf; +u32 value_btf_id; +u32 node_offset; +}; +struct btf_field {u32 offset;enum btf_field_type type;union {struct btf_field_kptr kptr; +struct btf_field_list_head list_head; }; };
@@ -266,6 +274,8 @@ static inline const char *btf_field_type_name(enum btf_field_type type)case BPF_KPTR_UNREF :case BPF_KPTR_REF :return "kptr" ; +case BPF_LIST_HEAD : +return "bpf_list_head" ;default :WARN_ON_ONCE (1 );return "unknown" ;
@@ -282,6 +292,8 @@ static inline u32 btf_field_type_size(enum btf_field_type type)case BPF_KPTR_UNREF :case BPF_KPTR_REF :return sizeof (u64); +case BPF_LIST_HEAD : +return sizeof (struct bpf_list_head );default :WARN_ON_ONCE (1 );return 0 ;
@@ -298,6 +310,8 @@ static inline u32 btf_field_type_align(enum btf_field_type type)case BPF_KPTR_UNREF :case BPF_KPTR_REF :return __alignof__(u64 ); +case BPF_LIST_HEAD : +return __alignof__(struct bpf_list_head );default :WARN_ON_ONCE (1 );return 0 ;
@@ -403,6 +417,9 @@ static inline void zero_map_value(struct bpf_map *map, void *dst)void copy_map_value_locked (struct bpf_map *map,void *dst,void *src,bool lock_src);void bpf_timer_cancel_and_free (void *timer); +void bpf_list_head_free (const struct btf_field *field,void *list_head, +struct bpf_spin_lock *spin_lock); +int bpf_obj_name_cpy (char *dst,const char *src,unsigned int size);struct bpf_offload_dev ;
include/uapi/linux/bpf.h +10 -0
@@ -6887,6 +6887,16 @@ struct bpf_dynptr { __u64 :64 ; } __attribute__((aligned(8))); +struct bpf_list_head { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 ))); + +struct bpf_list_node { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 ))); +struct bpf_sysctl {__u32 write;/* Sysctl is being read (= 0) or written (= 1). * Allows 1,2,4-byte read, but no write.
kernel/bpf/btf.c +140 -3
203 lines not on this page · raw message
kernel/bpf/helpers.c +32 -0
@@ -1706,6 +1706,38 @@ bpf_base_func_proto(enum bpf_func_id func_id) } } +void bpf_list_head_free (const struct btf_field *field,void *list_head, +struct bpf_spin_lock *spin_lock) +{ +struct list_head *head = list_head, *orig_head = list_head; + +BUILD_BUG_ON (sizeof (struct list_head ) >sizeof (struct bpf_list_head )); +BUILD_BUG_ON (__alignof__(struct list_head ) > __alignof__(struct bpf_list_head )); + +/* Do the actual list draining outside the lock to not hold the lock for +* too long, and also prevent deadlocks if tracing programs end up +* executing on entry/exit of functions called inside the critical +* section, and end up doing map ops that call bpf_list_head_free for +* the same map value again. +*/ +__bpf_spin_lock_irqsave (spin_lock); +if (!head->next ||list_empty (head)) + goto unlock; + head = head->next; +unlock: +INIT_LIST_HEAD (orig_head); +__bpf_spin_unlock_irqrestore (spin_lock); + +while (head != orig_head) { +void *obj = head; + + obj -= field->list_head.node_offset; + head = head->next; +/* TODO: Rework later */ +kfree (obj); + } +} + BTF_SET8_START(tracing_btf_ids)#ifdef CONFIG_KEXEC_CORE BTF_ID_FLAGS (func, crash_kexec,KF_DESTRUCTIVE )
kernel/bpf/syscall.c +21 -1
@@ -536,6 +536,9 @@ void btf_record_free(struct btf_record *rec)module_put (rec->fields[i].kptr.module);btf_put (rec->fields[i].kptr.btf);break ; +case BPF_LIST_HEAD : +/* Nothing to release for bpf_list_head */ +break ;default :WARN_ON_ONCE (1 );continue ;
@@ -578,6 +581,9 @@ struct btf_record *btf_record_dup(const struct btf_record *rec) goto free; }break ; +case BPF_LIST_HEAD : +/* Nothing to acquire for bpf_list_head */ +break ;default : ret = -EFAULT ;WARN_ON_ONCE (1 );
@@ -637,6 +643,11 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj)case BPF_KPTR_REF : field->kptr.dtor ((void *)xchg ((unsigned long *)field_ptr,0 ));break ; +case BPF_LIST_HEAD : +if (WARN_ON_ONCE (rec->spin_lock_off <0 )) +continue ; +bpf_list_head_free (field, field_ptr, obj + rec->spin_lock_off); +break ;default :WARN_ON_ONCE (1 );continue ;
@@ -965,7 +976,8 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf,if (!value_type || value_size != map->value_size)return -EINVAL ; - map->record =btf_parse_fields (btf, value_type,BPF_SPIN_LOCK |BPF_TIMER |BPF_KPTR , + map->record =btf_parse_fields (btf, value_type, +BPF_SPIN_LOCK |BPF_TIMER |BPF_KPTR |BPF_LIST_HEAD , map->value_size);if (!IS_ERR_OR_NULL (map->record)) {int i;
@@ -1012,6 +1024,14 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, goto free_map_tab; }break ; +case BPF_LIST_HEAD : +if (map->map_type !=BPF_MAP_TYPE_HASH && + map->map_type !=BPF_MAP_TYPE_LRU_HASH && + map->map_type !=BPF_MAP_TYPE_ARRAY ) { + ret = -EOPNOTSUPP ; + goto free_map_tab; + } +break ;default :/* Fail if map_type checks are missing for a field type */ ret = -EOPNOTSUPP ;
kernel/bpf/verifier.c +7 -0
@@ -12805,6 +12805,13 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, {enum bpf_prog_type prog_type =resolve_prog_type (prog); +if (btf_record_has_field (map->record,BPF_LIST_HEAD )) { +if (is_tracing_prog_type (prog_type)) { +verbose (env,"tracing progs cannot use bpf_list_head yet\n" ); +return -EINVAL ; + } + } +if (btf_record_has_field (map->record,BPF_SPIN_LOCK )) {if (prog_type ==BPF_PROG_TYPE_SOCKET_FILTER ) {verbose (env,"socket filter progs cannot use bpf_spin_lock yet\n" );
tools/include/uapi/linux/bpf.h +10 -0
@@ -6887,6 +6887,16 @@ struct bpf_dynptr { __u64 :64 ; } __attribute__((aligned(8))); +struct bpf_list_head { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 ))); + +struct bpf_list_node { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 ))); +struct bpf_sysctl {__u32 write;/* Sysctl is being read (= 0) or written (= 1). * Allows 1,2,4-byte read, but no write.
Signature · 1 line
2.38.1
Re: [PATCH bpf-next v5 03/25] bpf: Support bpf_list_head in map values
Quoted 67 lines · Kumar Kartikeya Dwivedi
Add the support on the map side to parse, recognize, verify, and build metadata table for a new special field of the type struct
bpf_list_head. To parameterize thebpf_list_headfor a certain value type and thelist_nodemember it will accept in that value type, we use BTF declaration tags.The definition of
bpf_list_headin a map value will be done as follows:struct foo { struct bpf_list_node node; int data; };struct map_value { struct bpf_list_head head __contains(foo, node); };Then, the
bpf_list_headonly allows adding to the list 'head' using thebpf_list_node'node' for the type struct foo.The 'contains' annotation is a BTF declaration tag composed of four parts, "contains:name:node" where the name is then used to look up the type in the map BTF, with its kind hardcoded to
BTF_KIND_STRUCTduring the lookup. The node defines name of the member in this type that has the type structbpf_list_node, which is actually used for linking into the linked list. For now, 'kind' part is hardcoded as struct.This allows building intrusive linked lists in BPF, using
container_ofto obtain pointer to entry, while being completely type safe from the perspective of the verifier. The verifier knows exactly the type of the nodes, and knows that list helpers return that type at some fixed offset where thebpf_list_nodemember used for this list exists. The verifier also uses this information to disallow adding types that are not accepted by a certain list.For now, no elements can be added to such lists. Support for that is coming in future patches, hence draining and freeing items is done with a TODO that will be resolved in a future patch.
Note that the
bpf_list_head_freefunction moves the list out to a local variable under the lock and releases it, doing the actual draining of the list items outside the lock. While this helps with not holding the lock for too long pessimizing other concurrent list operations, it is also necessary for deadlock prevention: unless every function called in the critical section would be notrace, a fentry/fexit program could attach and callbpf_map_update_elemagain on the map, leading to the same lock being acquired if the key matches and lead to a deadlock. While this requires some special effort on part of the BPF programmer to trigger and is highly unlikely to occur in practice, it is always better if we can avoid such a condition.While notrace would prevent this, doing the draining outside the lock has advantages of its own, hence it is used to also fix the deadlock related problem.
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 17 ++++ include/uapi/linux/bpf.h | 10 +++ kernel/bpf/btf.c | 143 ++++++++++++++++++++++++++++++++- kernel/bpf/helpers.c | 32 ++++++++ kernel/bpf/syscall.c | 22 ++++- kernel/bpf/verifier.c | 7 ++ tools/include/uapi/linux/bpf.h | 10 +++ 7 files changed, 237 insertions(+), 4 deletions(-)
[...]
Quoted 8 lines
struct
bpf_offload_dev;include/uapi/linux/bpf.h +0
-0@@ -6887,6 +6887,16 @@ struct bpf_dynptr { __u64 :64 ; } __attribute__((aligned(8)));
include/uapi/linux/bpf.h struct bpf_dynptr { · line 6898
Quoted 10 lines
include/uapi/linux/bpf.h +9
-0+struct bpf_list_head { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 ))); + +struct bpf_list_node { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 )));
Dave mentioned that this __u64 :64 trick makes vmlinux.h lose the alignment information, as the struct itself is empty, and so there is nothing indicating that it has to be 8-byte aligned.
So what if we have
struct bpf_list_node {
__u64 __opaque[2];
} __attribute__((aligned(8)));?
Quoted 5 lines
include/uapi/linux/bpf.h +1
-0+struct bpf_sysctl {__u32 write;/* Sysctl is being read (= 0) or written (= 1). * Allows 1,2,4-byte read, but no write. kernel/bpf/btf.c +0
-0
[...]
Quoted 14 lines
kernel/bpf/btf.c +6
-0@@ -3284,6 +3347,12 @@ static int btf_get_field_type(const char *name, u32 field_mask, u32 *seen_mask, goto end; } } +if (field_mask &BPF_LIST_HEAD ) { +if (!strcmp (name,"bpf_list_head" )) { + type =BPF_LIST_HEAD ; + goto end; + } + }/* Only return BPF_KPTR when all other types with matchable names fail */ if (field_mask &BPF_KPTR ) { type =BPF_KPTR_REF ;
kernel/bpf/btf.c static int btf_find_struct_field(const struct btf *btf, · line 3390
Quoted 5 lines
kernel/bpf/btf.c +2
-0@@ -3317,6 +3386,8 @@ static int btf_find_struct_field(const struct btf *btf,return field_type; off =__btf_member_bit_offset (t, member); +if (i && !off) +return -EFAULT ;
why? why can't my struct has zero-sized field in the beginning? This seems like a very incomplete and unnecessary check to me.
Quoted 17 lines
kernel/bpf/btf.c +6
-0if (off %8 )/* valid C code cannot generate such BTF */ return -EINVAL ;@@ -3339,6 +3410,12 @@ static int btf_find_struct_field(const struct btf *btf,if (ret <0 )return ret;break ; +case BPF_LIST_HEAD : + ret =btf_find_list_head (btf, t, member_type, i, off, sz, + idx < info_cnt ? &info[idx] : &tmp); +if (ret <0 ) +return ret; +break ;default :return -EFAULT ; }
kernel/bpf/btf.c static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, · line 3454
Quoted 5 lines
kernel/bpf/btf.c +2
-0@@ -3373,6 +3450,8 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,return field_type; off = vsi->offset; +if (i && !off) +return -EFAULT ;
similarly, I'd say that either we'd need to calculate the exact expected offset, or just not do anything here?
Quoted 16 lines
kernel/bpf/btf.c +6
-0if (vsi->size != sz)continue ;if (off % align)@@ -3393,6 +3472,12 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,if (ret <0 )return ret;break ; +case BPF_LIST_HEAD : + ret =btf_find_list_head (btf, var, var_type,-1 , off, sz, + idx < info_cnt ? &info[idx] : &tmp); +if (ret <0 ) +return ret; +break ;default :return -EFAULT ; }
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v5 03/25] bpf: Support bpf_list_head in map values
Quoted 94 lines · Andrii Nakryiko
Quoted 67 lines · Kumar Kartikeya Dwivedi
Add the support on the map side to parse, recognize, verify, and build metadata table for a new special field of the type struct
bpf_list_head. To parameterize thebpf_list_headfor a certain value type and thelist_nodemember it will accept in that value type, we use BTF declaration tags.The definition of
bpf_list_headin a map value will be done as follows:struct foo { struct bpf_list_node node; int data; };struct map_value { struct bpf_list_head head __contains(foo, node); };Then, the
bpf_list_headonly allows adding to the list 'head' using thebpf_list_node'node' for the type struct foo.The 'contains' annotation is a BTF declaration tag composed of four parts, "contains:name:node" where the name is then used to look up the type in the map BTF, with its kind hardcoded to
BTF_KIND_STRUCTduring the lookup. The node defines name of the member in this type that has the type structbpf_list_node, which is actually used for linking into the linked list. For now, 'kind' part is hardcoded as struct.This allows building intrusive linked lists in BPF, using
container_ofto obtain pointer to entry, while being completely type safe from the perspective of the verifier. The verifier knows exactly the type of the nodes, and knows that list helpers return that type at some fixed offset where thebpf_list_nodemember used for this list exists. The verifier also uses this information to disallow adding types that are not accepted by a certain list.For now, no elements can be added to such lists. Support for that is coming in future patches, hence draining and freeing items is done with a TODO that will be resolved in a future patch.
Note that the
bpf_list_head_freefunction moves the list out to a local variable under the lock and releases it, doing the actual draining of the list items outside the lock. While this helps with not holding the lock for too long pessimizing other concurrent list operations, it is also necessary for deadlock prevention: unless every function called in the critical section would be notrace, a fentry/fexit program could attach and callbpf_map_update_elemagain on the map, leading to the same lock being acquired if the key matches and lead to a deadlock. While this requires some special effort on part of the BPF programmer to trigger and is highly unlikely to occur in practice, it is always better if we can avoid such a condition.While notrace would prevent this, doing the draining outside the lock has advantages of its own, hence it is used to also fix the deadlock related problem.
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 17 ++++ include/uapi/linux/bpf.h | 10 +++ kernel/bpf/btf.c | 143 ++++++++++++++++++++++++++++++++- kernel/bpf/helpers.c | 32 ++++++++ kernel/bpf/syscall.c | 22 ++++- kernel/bpf/verifier.c | 7 ++ tools/include/uapi/linux/bpf.h | 10 +++ 7 files changed, 237 insertions(+), 4 deletions(-)[...]
Quoted 8 lines
struct
bpf_offload_dev;include/uapi/linux/bpf.h +0
-0@@ -6887,6 +6887,16 @@ struct bpf_dynptr { __u64 :64 ; } __attribute__((aligned(8)));include/uapi/linux/bpf.h struct bpf_dynptr { · line 6898
Quoted 10 lines
include/uapi/linux/bpf.h +9
-0+struct bpf_list_head { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 ))); + +struct bpf_list_node { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 )));Dave mentioned that this
__u64 :64trick makesvmlinux.hlose the alignment information, as the struct itself is empty, and so there is nothing indicating that it has to be 8-byte aligned.
Quoted 8 lines
So what if we have
struct bpf_list_node { __u64 __opaque[2]; } __attribute__((aligned(8)));?
Yep, can do that. Note that it's also potentially an issue for existing cases, like bpf_spin_lock, bpf_timer, bpf_dynptr, etc. Not completely sure if changing things now is possible, but if it is, we should probably make it for all of them?
Quoted 24 lines
Quoted 5 lines
include/uapi/linux/bpf.h +1
-0+struct bpf_sysctl {__u32 write;/* Sysctl is being read (= 0) or written (= 1). * Allows 1,2,4-byte read, but no write. kernel/bpf/btf.c +0
-0[...]
Quoted 16 lines
kernel/bpf/btf.c +6
-0@@ -3284,6 +3347,12 @@ static int btf_get_field_type(const char *name, u32 field_mask, u32 *seen_mask, goto end; } } +if (field_mask &BPF_LIST_HEAD ) { +if (!strcmp (name,"bpf_list_head" )) { + type =BPF_LIST_HEAD ; + goto end; + } + }/* Only return BPF_KPTR when all other types with matchable names fail */ if (field_mask &BPF_KPTR ) { type =BPF_KPTR_REF ;@@ -3317,6 +3386,8 @@ static int btf_find_struct_field(const struct btf *btf,return field_type;
Quoted 7 lines
kernel/bpf/btf.c static int btf_find_struct_field(const struct btf *btf, · line 3390
Quoted 3 lines
kernel/bpf/btf.c +2
-0off =__btf_member_bit_offset (t, member); +if (i && !off) +return -EFAULT ;why? why can't my struct has zero-sized field in the beginning? This seems like a very incomplete and unnecessary check to me.
Right, I will drop it for the struct case.
Quoted 19 lines
Quoted 19 lines
kernel/bpf/btf.c +6
-0if (off %8 )/* valid C code cannot generate such BTF */ return -EINVAL ;@@ -3339,6 +3410,12 @@ static int btf_find_struct_field(const struct btf *btf,if (ret <0 )return ret;break ; +case BPF_LIST_HEAD : + ret =btf_find_list_head (btf, t, member_type, i, off, sz, + idx < info_cnt ? &info[idx] : &tmp); +if (ret <0 ) +return ret; +break ;default :return -EFAULT ; }@@ -3373,6 +3450,8 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,return field_type;
Quoted 7 lines
kernel/bpf/btf.c static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, · line 3454
Quoted 3 lines
kernel/bpf/btf.c +2
-0off = vsi->offset; +if (i && !off) +return -EFAULT ;similarly, I'd say that either we'd need to calculate the exact expected offset, or just not do anything here?
This thread is actually what prompted this check:
https://lore.kernel.org/bpf/CAEf4Bza7ga2hEQ4J7EtgRHz49p1vZtaT4d2RDiyGOKGK41Nt=Q@mail.gmail.com
Unless loaded using libbpf all offsets are zero. So I think we need to reject it here, but I think the same zero sized field might be an issue for this as well, so maybe we remember the last field size and check whether it was zero or not?
I'll also include some more tests for these cases.
Re: [PATCH bpf-next v5 03/25] bpf: Support bpf_list_head in map values
Quoted 102 lines · Kumar Kartikeya Dwivedi
Quoted 100 lines · Andrii Nakryiko
Quoted 67 lines · Kumar Kartikeya Dwivedi
Add the support on the map side to parse, recognize, verify, and build metadata table for a new special field of the type struct
bpf_list_head. To parameterize thebpf_list_headfor a certain value type and thelist_nodemember it will accept in that value type, we use BTF declaration tags.The definition of
bpf_list_headin a map value will be done as follows:struct foo { struct bpf_list_node node; int data; };struct map_value { struct bpf_list_head head __contains(foo, node); };Then, the
bpf_list_headonly allows adding to the list 'head' using thebpf_list_node'node' for the type struct foo.The 'contains' annotation is a BTF declaration tag composed of four parts, "contains:name:node" where the name is then used to look up the type in the map BTF, with its kind hardcoded to
BTF_KIND_STRUCTduring the lookup. The node defines name of the member in this type that has the type structbpf_list_node, which is actually used for linking into the linked list. For now, 'kind' part is hardcoded as struct.This allows building intrusive linked lists in BPF, using
container_ofto obtain pointer to entry, while being completely type safe from the perspective of the verifier. The verifier knows exactly the type of the nodes, and knows that list helpers return that type at some fixed offset where thebpf_list_nodemember used for this list exists. The verifier also uses this information to disallow adding types that are not accepted by a certain list.For now, no elements can be added to such lists. Support for that is coming in future patches, hence draining and freeing items is done with a TODO that will be resolved in a future patch.
Note that the
bpf_list_head_freefunction moves the list out to a local variable under the lock and releases it, doing the actual draining of the list items outside the lock. While this helps with not holding the lock for too long pessimizing other concurrent list operations, it is also necessary for deadlock prevention: unless every function called in the critical section would be notrace, a fentry/fexit program could attach and callbpf_map_update_elemagain on the map, leading to the same lock being acquired if the key matches and lead to a deadlock. While this requires some special effort on part of the BPF programmer to trigger and is highly unlikely to occur in practice, it is always better if we can avoid such a condition.While notrace would prevent this, doing the draining outside the lock has advantages of its own, hence it is used to also fix the deadlock related problem.
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 17 ++++ include/uapi/linux/bpf.h | 10 +++ kernel/bpf/btf.c | 143 ++++++++++++++++++++++++++++++++- kernel/bpf/helpers.c | 32 ++++++++ kernel/bpf/syscall.c | 22 ++++- kernel/bpf/verifier.c | 7 ++ tools/include/uapi/linux/bpf.h | 10 +++ 7 files changed, 237 insertions(+), 4 deletions(-)[...]
Quoted 8 lines
struct
bpf_offload_dev;include/uapi/linux/bpf.h +0
-0@@ -6887,6 +6887,16 @@ struct bpf_dynptr { __u64 :64 ; } __attribute__((aligned(8)));include/uapi/linux/bpf.h struct bpf_dynptr { · line 6898
Quoted 10 lines
include/uapi/linux/bpf.h +9
-0+struct bpf_list_head { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 ))); + +struct bpf_list_node { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 )));Dave mentioned that this
__u64 :64trick makesvmlinux.hlose the alignment information, as the struct itself is empty, and so there is nothing indicating that it has to be 8-byte aligned.So what if we have
struct bpf_list_node { __u64 __opaque[2]; } __attribute__((aligned(8)));
Quoted 7 lines
Quoted 2 lines
?
Yep, can do that. Note that it's also potentially an issue for existing cases, like
bpf_spin_lock,bpf_timer,bpf_dynptr, etc. Not completely sure if changing things now is possible, but if it is, we should probably make it for all of them?
Why not? We are not removing anything or changing sizes, so it's backwards compatible. But I have a suspicion Alexei might not like this __opaque field, so let's see what he says.
Quoted 62 lines
Quoted 25 lines
Quoted 6 lines
include/uapi/linux/bpf.h +1
-0+struct bpf_sysctl {__u32 write;/* Sysctl is being read (= 0) or written (= 1). * Allows 1,2,4-byte read, but no write. kernel/bpf/btf.c +0
-0[...]
Quoted 16 lines
kernel/bpf/btf.c +6
-0@@ -3284,6 +3347,12 @@ static int btf_get_field_type(const char *name, u32 field_mask, u32 *seen_mask, goto end; } } +if (field_mask &BPF_LIST_HEAD ) { +if (!strcmp (name,"bpf_list_head" )) { + type =BPF_LIST_HEAD ; + goto end; + } + }/* Only return BPF_KPTR when all other types with matchable names fail */ if (field_mask &BPF_KPTR ) { type =BPF_KPTR_REF ;@@ -3317,6 +3386,8 @@ static int btf_find_struct_field(const struct btf *btf,return field_type;Quoted 7 lines
kernel/bpf/btf.c static int btf_find_struct_field(const struct btf *btf, · line 3390
Quoted 3 lines
kernel/bpf/btf.c +2
-0off =__btf_member_bit_offset (t, member); +if (i && !off) +return -EFAULT ;why? why can't my struct has zero-sized field in the beginning? This seems like a very incomplete and unnecessary check to me.
Right, I will drop it for the struct case.
Quoted 19 lines
Quoted 19 lines
kernel/bpf/btf.c +6
-0if (off %8 )/* valid C code cannot generate such BTF */ return -EINVAL ;@@ -3339,6 +3410,12 @@ static int btf_find_struct_field(const struct btf *btf,if (ret <0 )return ret;break ; +case BPF_LIST_HEAD : + ret =btf_find_list_head (btf, t, member_type, i, off, sz, + idx < info_cnt ? &info[idx] : &tmp); +if (ret <0 ) +return ret; +break ;default :return -EFAULT ; }@@ -3373,6 +3450,8 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,return field_type;Quoted 7 lines
kernel/bpf/btf.c static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, · line 3454
Quoted 3 lines
kernel/bpf/btf.c +2
-0off = vsi->offset; +if (i && !off) +return -EFAULT ;similarly, I'd say that either we'd need to calculate the exact expected offset, or just not do anything here?
Quoted 8 lines
This thread is actually what prompted this check:
https://lore.kernel.org/bpf/CAEf4Bza7ga2hEQ4J7EtgRHz49p1vZtaT4d2RDiyGOKGK41Nt=Q@mail.gmail.comUnless loaded using libbpf all offsets are zero. So I think we need to reject it here, but I think the same zero sized field might be an issue for this as well, so maybe we remember the last field size and check whether it was zero or not?
I'll also include some more tests for these cases.
The question is whether this affects correctness from the verifier standpoint? If it does, there must be some other place where this will cause problem and should be caught and reported.
My main objection is that it's half a check, we check that it's non-zero, but we don't check that it is correct in stricter sense. So I'd rather drop it altogether, or go all the way to check that it is correct offset (taking into account sizes of previous vars).
Re: [PATCH bpf-next v5 03/25] bpf: Support bpf_list_head in map values
Quoted 9 lines · Andrii Nakryiko
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
struct
bpf_offload_dev;include/uapi/linux/bpf.h +0
-0@@ -6887,6 +6887,16 @@ struct bpf_dynptr { __u64 :64 ; } __attribute__((aligned(8)));
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
include/uapi/linux/bpf.h +5
-0+struct bpf_list_head { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 ))); +Quoted 8 lines
Quoted 8 lines
include/uapi/linux/bpf.h struct bpf_dynptr { · line 6898
Quoted 4 lines
include/uapi/linux/bpf.h +4
-0+struct bpf_list_node { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 )));Dave mentioned that this
__u64 :64trick makesvmlinux.hlose the alignment information, as the struct itself is empty, and so there is nothing indicating that it has to be 8-byte aligned.
Since it's not a new issue let's fix it for all. Whether it's a combination of pahole + bpftool or just pahole.
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
So what if we have
struct bpf_list_node { __u64 __opaque[2]; } __attribute__((aligned(8)));Quoted 7 lines
Quoted 2 lines
?
Yep, can do that. Note that it's also potentially an issue for existing cases, like
bpf_spin_lock,bpf_timer,bpf_dynptr, etc. Not completely sure if changing things now is possible, but if it is, we should probably make it for all of them?
Quoted 4 lines
Why not? We are not removing anything or changing sizes, so it's backwards compatible.
But I have a suspicion Alexei might not like
this__opaquefield, so let's see what he says.
I prefer to fix them all at once without adding a name.
Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
include/uapi/linux/bpf.h struct bpf_dynptr { · line 6901
Quoted 4 lines
include/uapi/linux/bpf.h +2
-0off = vsi->offset; +if (i && !off) +return -EFAULT ;similarly, I'd say that either we'd need to calculate the exact expected offset, or just not do anything here?
This thread is actually what prompted this check:
https://lore.kernel.org/bpf/CAEf4Bza7ga2hEQ4J7EtgRHz49p1vZtaT4d2RDiyGOKGK41Nt=Q@mail.gmail.comUnless loaded using libbpf all offsets are zero. So I think we need to reject it here, but I think the same zero sized field might be an issue for this as well, so maybe we remember the last field size and check whether it was zero or not?
Quoted 5 lines
Quoted 1 line
I'll also include some more tests for these cases.
The question is whether this affects correctness from the verifier standpoint? If it does, there must be some other place where this will cause problem and should be caught and reported.
If it's an issue with BTF then we should probably check it during generic datasec verification.
Here it's kinda late to warn.
Quoted 4 lines
My main objection is that it's half a check, we check that it's non-zero, but we don't check that it is correct in stricter sense. So I'd rather drop it altogether, or go all the way to check that it is correct offset (taking into account sizes of previous vars).
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v5 03/25] bpf: Support bpf_list_head in map values
Quoted 25 lines · Alexei Starovoitov
Quoted 9 lines · Andrii Nakryiko
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
struct
bpf_offload_dev;include/uapi/linux/bpf.h +0
-0@@ -6887,6 +6887,16 @@ struct bpf_dynptr { __u64 :64 ; } __attribute__((aligned(8)));Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
include/uapi/linux/bpf.h +5
-0+struct bpf_list_head { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 ))); +Quoted 8 lines
Quoted 8 lines
include/uapi/linux/bpf.h struct bpf_dynptr { · line 6898
Quoted 4 lines
include/uapi/linux/bpf.h +4
-0+struct bpf_list_node { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 )));Dave mentioned that this
__u64 :64trick makesvmlinux.hlose the alignment information, as the struct itself is empty, and so there is nothing indicating that it has to be 8-byte aligned.
Quoted 3 lines
Since it's not a new issue let's fix it for all. Whether it's a combination of pahole + bpftool or just pahole.
Would it make sense to do that as a follow up? The selftest does work right now because I specified __attribute__((aligned(8))) manually for the variables.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 9 lines
So what if we have
struct bpf_list_node { __u64 __opaque[2]; } __attribute__((aligned(8)));?
Yep, can do that. Note that it's also potentially an issue for existing cases, like
bpf_spin_lock,bpf_timer,bpf_dynptr, etc. Not completely sure if changing things now is possible, but if it is, we should probably make it for all of them?
Quoted 28 lines
Quoted 4 lines
Why not? We are not removing anything or changing sizes, so it's backwards compatible.
But I have a suspicion Alexei might not like
this__opaquefield, so let's see what he says.I prefer to fix them all at once without adding a name.
Quoted 21 lines
Quoted 9 lines
Quoted 8 lines
include/uapi/linux/bpf.h struct bpf_dynptr { · line 6901
Quoted 4 lines
include/uapi/linux/bpf.h +2
-0off = vsi->offset; +if (i && !off) +return -EFAULT ;similarly, I'd say that either we'd need to calculate the exact expected offset, or just not do anything here?
Quoted 8 lines
This thread is actually what prompted this check:
https://lore.kernel.org/bpf/CAEf4Bza7ga2hEQ4J7EtgRHz49p1vZtaT4d2RDiyGOKGK41Nt=Q@mail.gmail.comUnless loaded using libbpf all offsets are zero. So I think we need to reject it here, but I think the same zero sized field might be an issue for this as well, so maybe we remember the last field size and check whether it was zero or not?
I'll also include some more tests for these cases.
The question is whether this affects correctness from the verifier standpoint? If it does, there must be some other place where this will cause problem and should be caught and reported.
The problem here is that if the BTF is incorrect like this, where you have same off for multiple items (bpf_spin_lock, bpf_list_head, etc.) like off=0 here, they essentially get the same offset in our btf_record array.
I can check for it when appending items to the array (i.e. next offset must be atleast prev_off + prev_sz at the very minimum).
Quoted 5 lines
If it's an issue with BTF then we should probably check it during generic datasec verification.
Here it's kinda late to warn.
There's also a concern that clang produces this BTF by default. If you're not using libbpf as a loader, BTF that loaded previously will fail now (since DATASEC var offs are always 0 and we will complain during validation and return an error). Not sure what the impact will be, but just putting it out there.
Let me know what should be better. In either case I'll add a test case.
Re: [PATCH bpf-next v5 03/25] bpf: Support bpf_list_head in map values
Quoted 78 lines · Kumar Kartikeya Dwivedi
Quoted 25 lines · Alexei Starovoitov
Quoted 9 lines · Andrii Nakryiko
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
struct
bpf_offload_dev;include/uapi/linux/bpf.h +0
-0@@ -6887,6 +6887,16 @@ struct bpf_dynptr { __u64 :64 ; } __attribute__((aligned(8)));Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
include/uapi/linux/bpf.h +5
-0+struct bpf_list_head { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 ))); +Quoted 8 lines
Quoted 8 lines
include/uapi/linux/bpf.h struct bpf_dynptr { · line 6898
Quoted 4 lines
include/uapi/linux/bpf.h +4
-0+struct bpf_list_node { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 )));Dave mentioned that this
__u64 :64trick makesvmlinux.hlose the alignment information, as the struct itself is empty, and so there is nothing indicating that it has to be 8-byte aligned.Quoted 3 lines
Since it's not a new issue let's fix it for all. Whether it's a combination of pahole + bpftool or just pahole.
Would it make sense to do that as a follow up? The selftest does work right now because I specified __attribute__((aligned(8))) manually for the variables.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 9 lines
So what if we have
struct bpf_list_node { __u64 __opaque[2]; } __attribute__((aligned(8)));?
Yep, can do that. Note that it's also potentially an issue for existing cases, like
bpf_spin_lock,bpf_timer,bpf_dynptr, etc. Not completely sure if changing things now is possible, but if it is, we should probably make it for all of them?Quoted 28 lines
Quoted 4 lines
Why not? We are not removing anything or changing sizes, so it's backwards compatible.
But I have a suspicion Alexei might not like
this__opaquefield, so let's see what he says.I prefer to fix them all at once without adding a name.
Quoted 21 lines
Quoted 9 lines
Quoted 8 lines
include/uapi/linux/bpf.h struct bpf_dynptr { · line 6901
Quoted 4 lines
include/uapi/linux/bpf.h +2
-0off = vsi->offset; +if (i && !off) +return -EFAULT ;similarly, I'd say that either we'd need to calculate the exact expected offset, or just not do anything here?
Quoted 8 lines
This thread is actually what prompted this check:
https://lore.kernel.org/bpf/CAEf4Bza7ga2hEQ4J7EtgRHz49p1vZtaT4d2RDiyGOKGK41Nt=Q@mail.gmail.comUnless loaded using libbpf all offsets are zero. So I think we need to reject it here, but I think the same zero sized field might be an issue for this as well, so maybe we remember the last field size and check whether it was zero or not?
I'll also include some more tests for these cases.
The question is whether this affects correctness from the verifier standpoint? If it does, there must be some other place where this will cause problem and should be caught and reported.
Quoted 6 lines
The problem here is that if the BTF is incorrect like this, where you have same off for multiple items (
bpf_spin_lock,bpf_list_head, etc.) like off=0 here, they essentially get the same offset in ourbtf_recordarray.I can check for it when appending items to the array (i.e. next offset must be atleast
prev_off+prev_szat the very minimum).
yes, that would be more correct, because you can still generate bad BTF where you have different non-zero offsets, but overlapping variables. Such a situation should be prevented anyways, right?
Quoted 7 lines
Quoted 6 lines
If it's an issue with BTF then we should probably check it during generic datasec verification.
Here it's kinda late to warn.
Quoted 6 lines
There's also a concern that clang produces this BTF by default. If you're not using libbpf as a loader, BTF that loaded previously will fail now (since DATASEC var offs are always 0 and we will complain during validation and return an error). Not sure what the impact will be, but just putting it out there.
Let me know what should be better. In either case I'll add a test case.
So on one hand all BPF loaders like libbpf should be adjusting and fixing such incomplete BTFs coming from Clang. On the other hand, we can add such enforcements only if such special global variables are present. I.e., if we care about DATASEC layout for correctness.
Re: [PATCH bpf-next v5 03/25] bpf: Support bpf_list_head in map values
In reply to Alexei StarovoitovQuoted 26 lines · Alexei Starovoitov
Quoted 9 lines · Andrii Nakryiko
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
struct
bpf_offload_dev;include/uapi/linux/bpf.h +0
-0@@ -6887,6 +6887,16 @@ struct bpf_dynptr { __u64 :64 ; } __attribute__((aligned(8)));Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
include/uapi/linux/bpf.h +5
-0+struct bpf_list_head { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 ))); +Quoted 8 lines
Quoted 8 lines
include/uapi/linux/bpf.h struct bpf_dynptr { · line 6898
Quoted 4 lines
include/uapi/linux/bpf.h +4
-0+struct bpf_list_node { + __u64 :64; + __u64 :64; +} __attribute__((aligned (8 )));Dave mentioned that this
__u64 :64trick makesvmlinux.hlose the alignment information, as the struct itself is empty, and so there is nothing indicating that it has to be 8-byte aligned.
Quoted 2 lines
Since it's not a new issue let's fix it for all. Whether it's a combination of pahole + bpftool or just pahole.
So yeah, I was expecting if we do this, we'd do this for all opaque BPF UAPI structs like this (bpf_spin_lock and others), of course.
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 10 lines
So what if we have
struct bpf_list_node { __u64 __opaque[2]; } __attribute__((aligned(8)));?
Yep, can do that. Note that it's also potentially an issue for existing cases, like
bpf_spin_lock,bpf_timer,bpf_dynptr, etc. Not completely sure if changing things now is possible, but if it is, we should probably make it for all of them?
Quoted 7 lines
Quoted 4 lines
Why not? We are not removing anything or changing sizes, so it's backwards compatible.
But I have a suspicion Alexei might not like
this__opaquefield, so let's see what he says.I prefer to fix them all at once without adding a name.
This is not an issue with BTF per se.
struct blah {
u64: 64
};
is just an empty struct blah with 8-byte size. Both BTF and DWARF will record it as just
struct blah {
}
and record it's size as 8 bytes.
With that, there is nothing to suggest that this struct has to have 8-byte alignment.
If we mark explicitly __attribute__((aligned(8))) then DWARF will additionally record alignment=8 for such struct. BTF doesn't record alignment, though.
adding u64 fields internally will make libbpf recognize that struct needs at least 8-byte alignment, which is what I propose as a simple solution.
Alternatives are:
- extend BTF to record struct/union alignments in BTF_KIND_{STRUCT,UNION}
- record __attribute__((aligned(8))) as a new KIND (BTF_KIND_ATTRIBUTE)
Both seem like a bit of an overkill, given the work around is to have u64 __opaque[] fields, which we won't have to remove or rename ever (because those fields are not used).
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 8 lines
include/uapi/linux/bpf.h struct bpf_dynptr { · line 6901
Quoted 4 lines
include/uapi/linux/bpf.h +2
-0off = vsi->offset; +if (i && !off) +return -EFAULT ;similarly, I'd say that either we'd need to calculate the exact expected offset, or just not do anything here?
This thread is actually what prompted this check:
https://lore.kernel.org/bpf/CAEf4Bza7ga2hEQ4J7EtgRHz49p1vZtaT4d2RDiyGOKGK41Nt=Q@mail.gmail.comUnless loaded using libbpf all offsets are zero. So I think we need to reject it here, but I think the same zero sized field might be an issue for this as well, so maybe we remember the last field size and check whether it was zero or not?
I'll also include some more tests for these cases.
Quoted 7 lines
Quoted 3 lines
The question is whether this affects correctness from the verifier standpoint? If it does, there must be some other place where this will cause problem and should be caught and reported.
If it's an issue with BTF then we should probably check it during generic datasec verification.
Here it's kinda late to warn.
+1 and do it more properly than forcing it to be non-zero
Quoted 5 lines
Quoted 5 lines
My main objection is that it's half a check, we check that it's non-zero, but we don't check that it is correct in stricter sense. So I'd rather drop it altogether, or go all the way to check that it is correct offset (taking into account sizes of previous vars).
Re: [PATCH bpf-next v5 03/25] bpf: Support bpf_list_head in map values
Quoted 46 lines · Andrii Nakryiko
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 11 lines
So what if we have
struct bpf_list_node { __u64 __opaque[2]; } __attribute__((aligned(8)));?
Yep, can do that. Note that it's also potentially an issue for existing cases, like
bpf_spin_lock,bpf_timer,bpf_dynptr, etc. Not completely sure if changing things now is possible, but if it is, we should probably make it for all of them?Quoted 7 lines
Quoted 4 lines
Why not? We are not removing anything or changing sizes, so it's backwards compatible.
But I have a suspicion Alexei might not like
this__opaquefield, so let's see what he says.I prefer to fix them all at once without adding a name.
This is not an issue with BTF per se.
struct blah {
u64: 64
};is just an empty struct blah with 8-byte size. Both BTF and DWARF will record it as just
struct blah {
}and record it's size as 8 bytes.
With that, there is nothing to suggest that this struct has to have 8-byte alignment.
If we mark explicitly __attribute__((aligned(8))) then DWARF will additionally record alignment=8 for such struct. BTF doesn't record alignment, though.
Quoted 7 lines
adding u64 fields internally will make libbpf recognize that struct needs at least 8-byte alignment, which is what I propose as a simple solution.
Alternatives are:
- extend BTF to record struct/union alignments in BTF_KIND_{STRUCT,UNION}
- record __attribute__((aligned(8))) as a new KIND (BTF_KIND_ATTRIBUTE)
imo above two options are way better than adding __opaque which is nothing but a workaround for a missing feature in BTF.
Quoted 3 lines
Both seem like a bit of an overkill, given the work around is to have u64 __opaque[] fields, which we won't have to remove or rename ever (because those fields are not used).
There is no rush to do this workaround.
As Kumar says we can land the existing patch as-is and add BTF_KIND_ATTR (or anything else) in the followup.
Adding __opaque now won't work, since we won't be able to remove it later due to backward compat.
Kumar Kartikeya Dwivedi
maintainerPATCH v5 4/25 bpf: Rename RET_PTR_TO_ALLOC_MEM
In reply to Kumar Kartikeya DwivediCurrently, the verifier has two return types, RET_PTR_TO_ALLOC_MEM, and RET_PTR_TO_ALLOC_MEM_OR_NULL, however the former is confusingly named to imply that it carries MEM_ALLOC, while only the latter does. This causes confusion during code review leading to conclusions like that the return value of RET_PTR_TO_DYNPTR_MEM_OR_NULL (which is RET_PTR_TO_ALLOC_MEM | PTR_MAYBE_NULL) may be consumable by bpf_ringbuf_{submit,commit}.
Rename it to make it clear MEM_ALLOC needs to be tacked on top of RET_PTR_TO_MEM.
--- include/linux/bpf.h | 6 +++--- kernel/bpf/verifier.c | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-)
include/linux/bpf.h +3 -3
@@ -607,7 +607,7 @@ enum bpf_return_type {RET_PTR_TO_SOCKET ,/* returns a pointer to a socket */ RET_PTR_TO_TCP_SOCK ,/* returns a pointer to a tcp_sock */ RET_PTR_TO_SOCK_COMMON ,/* returns a pointer to a sock_common */ -RET_PTR_TO_ALLOC_MEM ,/* returns a pointer to dynamically allocated memory */ +RET_PTR_TO_MEM ,/* returns a pointer to memory */ RET_PTR_TO_MEM_OR_BTF_ID ,/* returns a pointer to a valid memory or a btf_id */ RET_PTR_TO_BTF_ID ,/* returns a pointer to a btf_id */ __BPF_RET_TYPE_MAX,
@@ -617,8 +617,8 @@ enum bpf_return_type {RET_PTR_TO_SOCKET_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_SOCKET ,RET_PTR_TO_TCP_SOCK_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_TCP_SOCK ,RET_PTR_TO_SOCK_COMMON_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_SOCK_COMMON , -RET_PTR_TO_ALLOC_MEM_OR_NULL =PTR_MAYBE_NULL |MEM_ALLOC |RET_PTR_TO_ALLOC_MEM , -RET_PTR_TO_DYNPTR_MEM_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_ALLOC_MEM , +RET_PTR_TO_ALLOC_MEM_OR_NULL =PTR_MAYBE_NULL |MEM_ALLOC |RET_PTR_TO_MEM , +RET_PTR_TO_DYNPTR_MEM_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_MEM ,RET_PTR_TO_BTF_ID_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_BTF_ID ,/* This must be the last entry. Its purpose is to ensure the enum is
kernel/bpf/verifier.c +1 -1
@@ -7621,7 +7621,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insnmark_reg_known_zero (env, regs,BPF_REG_0 ); regs[BPF_REG_0 ].type =PTR_TO_TCP_SOCK | ret_flag;break ; -case RET_PTR_TO_ALLOC_MEM : +case RET_PTR_TO_MEM :mark_reg_known_zero (env, regs,BPF_REG_0 ); regs[BPF_REG_0 ].type =PTR_TO_MEM | ret_flag; regs[BPF_REG_0 ].mem_size = meta.mem_size;
Signature · 1 line
2.38.1
Re: [PATCH bpf-next v5 04/25] bpf: Rename RET_PTR_TO_ALLOC_MEM
Quoted 8 lines · Kumar Kartikeya Dwivedi
Currently, the verifier has two return types,
RET_PTR_TO_ALLOC_MEM, andRET_PTR_TO_ALLOC_MEM_OR_NULL, however the former is confusingly named to imply that it carriesMEM_ALLOC, while only the latter does. This causes confusion during code review leading to conclusions like that the return value ofRET_PTR_TO_DYNPTR_MEM_OR_NULL(which isRET_PTR_TO_ALLOC_MEM|PTR_MAYBE_NULL) may be consumable by bpf_ringbuf_{submit,commit}.
Quoted 5 lines
Rename it to make it clear
MEM_ALLOCneeds to be tacked on top ofRET_PTR_TO_MEM.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
The whole MEM_ALLOC as related to ringbuf is so confusing. Why can't be just call it for what it is: RET_PTR_TO_RINGBUF_MEM_OR_NULL, ARG_PTR_TO_RINGBUF_MEM, PTR_TO_RINGBUF_MEM ?
It would be also much easier to make sure (by looking at the code) that ringbuf invariants are properly checked.
Quoted 44 lines
include/linux/bpf.h | 6 +++--- kernel/bpf/verifier.c | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-)include/linux/bpf.h +3
-3@@ -607,7 +607,7 @@ enum bpf_return_type {RET_PTR_TO_SOCKET ,/* returns a pointer to a socket */ RET_PTR_TO_TCP_SOCK ,/* returns a pointer to a tcp_sock */ RET_PTR_TO_SOCK_COMMON ,/* returns a pointer to a sock_common */ -RET_PTR_TO_ALLOC_MEM ,/* returns a pointer to dynamically allocated memory */ +RET_PTR_TO_MEM ,/* returns a pointer to memory */ RET_PTR_TO_MEM_OR_BTF_ID ,/* returns a pointer to a valid memory or a btf_id */ RET_PTR_TO_BTF_ID ,/* returns a pointer to a btf_id */ __BPF_RET_TYPE_MAX,@@ -617,8 +617,8 @@ enum bpf_return_type {RET_PTR_TO_SOCKET_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_SOCKET ,RET_PTR_TO_TCP_SOCK_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_TCP_SOCK ,RET_PTR_TO_SOCK_COMMON_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_SOCK_COMMON , -RET_PTR_TO_ALLOC_MEM_OR_NULL =PTR_MAYBE_NULL |MEM_ALLOC |RET_PTR_TO_ALLOC_MEM , -RET_PTR_TO_DYNPTR_MEM_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_ALLOC_MEM , +RET_PTR_TO_ALLOC_MEM_OR_NULL =PTR_MAYBE_NULL |MEM_ALLOC |RET_PTR_TO_MEM , +RET_PTR_TO_DYNPTR_MEM_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_MEM ,RET_PTR_TO_BTF_ID_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_BTF_ID ,/* This must be the last entry. Its purpose is to ensure the enum is kernel/bpf/verifier.c +1
-2@@ -7621,7 +7621,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insnmark_reg_known_zero (env, regs,BPF_REG_0 ); regs[BPF_REG_0 ].type =PTR_TO_TCP_SOCK | ret_flag;break ; -case RET_PTR_TO_ALLOC_MEM : +case RET_PTR_TO_MEM :mark_reg_known_zero (env, regs,BPF_REG_0 ); regs[BPF_REG_0 ].type =PTR_TO_MEM | ret_flag; regs[BPF_REG_0 ].mem_size = meta.mem_size; --2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 5/25 bpf: Rename MEM_ALLOC to MEM_RINGBUF
In reply to Kumar Kartikeya DwivediCurrently, verifier uses MEM_ALLOC type tag to specially tag memory returned from bpf_ringbuf_reserve helper. However, this is currently only used for this purpose and there is an implicit assumption that it only refers to ringbuf memory (e.g. the check for ARG_PTR_TO_ALLOC_MEM in check_func_arg_reg_off).
Hence, rename MEM_ALLOC to MEM_RINGBUF to indicate this special relationship and instead open the use of MEM_ALLOC for more generic allocations made for user types.
Also, since ARG_PTR_TO_ALLOC_MEM_OR_NULL is unused, simply drop it.
Finally, update selftests using 'alloc_' verifier string to 'ringbuf_'.
--- include/linux/bpf.h | 11 ++++------- kernel/bpf/ringbuf.c | 6 +++--- kernel/bpf/verifier.c | 14 +++++++------- tools/testing/selftests/bpf/prog_tests/dynptr.c | 2 +- tools/testing/selftests/bpf/verifier/ringbuf.c | 2 +- tools/testing/selftests/bpf/verifier/spill_fill.c | 2 +- 6 files changed, 17 insertions(+), 20 deletions(-)
include/linux/bpf.h +4 -7
@@ -488,10 +488,8 @@ enum bpf_type_flag {*/ MEM_RDONLY =BIT (1 +BPF_BASE_TYPE_BITS ), -/* MEM was "allocated" from a different helper, and cannot be mixed -* with regular non-MEM_ALLOC'ed MEM types. -*/ -MEM_ALLOC =BIT (2 +BPF_BASE_TYPE_BITS ), +/* MEM points to BPF ring buffer reservation. */ +MEM_RINGBUF =BIT (2 +BPF_BASE_TYPE_BITS ),/* MEM is in user address space. */ MEM_USER = BIT(3 + BPF_BASE_TYPE_BITS),
@@ -565,7 +563,7 @@ enum bpf_arg_type {ARG_PTR_TO_LONG ,/* pointer to long */ ARG_PTR_TO_SOCKET ,/* pointer to bpf_sock (fullsock) */ ARG_PTR_TO_BTF_ID ,/* pointer to in-kernel struct */ -ARG_PTR_TO_ALLOC_MEM ,/* pointer to dynamically allocated memory */ +ARG_PTR_TO_RINGBUF_MEM ,/* pointer to dynamically reserved ringbuf memory */ ARG_CONST_ALLOC_SIZE_OR_ZERO ,/* number of allocated bytes requested */ ARG_PTR_TO_BTF_ID_SOCK_COMMON ,/* pointer to in-kernel sock_common or bpf-mirrored bpf_sock */ ARG_PTR_TO_PERCPU_BTF_ID, /* pointer to in-kernel percpu type */
@@ -582,7 +580,6 @@ enum bpf_arg_type {ARG_PTR_TO_MEM_OR_NULL =PTR_MAYBE_NULL |ARG_PTR_TO_MEM ,ARG_PTR_TO_CTX_OR_NULL =PTR_MAYBE_NULL |ARG_PTR_TO_CTX ,ARG_PTR_TO_SOCKET_OR_NULL =PTR_MAYBE_NULL |ARG_PTR_TO_SOCKET , -ARG_PTR_TO_ALLOC_MEM_OR_NULL =PTR_MAYBE_NULL |ARG_PTR_TO_ALLOC_MEM ,ARG_PTR_TO_STACK_OR_NULL =PTR_MAYBE_NULL |ARG_PTR_TO_STACK , ARG_PTR_TO_BTF_ID_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_BTF_ID,/* pointer to memory does not need to be initialized, helper function must fill
@@ -617,7 +614,7 @@ enum bpf_return_type {RET_PTR_TO_SOCKET_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_SOCKET ,RET_PTR_TO_TCP_SOCK_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_TCP_SOCK ,RET_PTR_TO_SOCK_COMMON_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_SOCK_COMMON , -RET_PTR_TO_ALLOC_MEM_OR_NULL =PTR_MAYBE_NULL |MEM_ALLOC |RET_PTR_TO_MEM , +RET_PTR_TO_RINGBUF_MEM_OR_NULL =PTR_MAYBE_NULL |MEM_RINGBUF |RET_PTR_TO_MEM ,RET_PTR_TO_DYNPTR_MEM_OR_NULL =PTR_MAYBE_NULL |RET_PTR_TO_MEM , RET_PTR_TO_BTF_ID_OR_NULL = PTR_MAYBE_NULL | RET_PTR_TO_BTF_ID,
kernel/bpf/ringbuf.c +3 -3
@@ -447,7 +447,7 @@ BPF_CALL_3(bpf_ringbuf_reserve, struct bpf_map *, map, u64, size, u64, flags)const struct bpf_func_proto bpf_ringbuf_reserve_proto = { .func = bpf_ringbuf_reserve, - .ret_type =RET_PTR_TO_ALLOC_MEM_OR_NULL , + .ret_type =RET_PTR_TO_RINGBUF_MEM_OR_NULL , .arg1_type =ARG_CONST_MAP_PTR , .arg2_type =ARG_CONST_ALLOC_SIZE_OR_ZERO , .arg3_type =ARG_ANYTHING ,
@@ -490,7 +490,7 @@ BPF_CALL_2(bpf_ringbuf_submit, void *, sample, u64, flags)const struct bpf_func_proto bpf_ringbuf_submit_proto = { .func = bpf_ringbuf_submit, .ret_type =RET_VOID , - .arg1_type =ARG_PTR_TO_ALLOC_MEM |OBJ_RELEASE , + .arg1_type =ARG_PTR_TO_RINGBUF_MEM |OBJ_RELEASE , .arg2_type =ARG_ANYTHING , };
@@ -503,7 +503,7 @@ BPF_CALL_2(bpf_ringbuf_discard, void *, sample, u64, flags)const struct bpf_func_proto bpf_ringbuf_discard_proto = { .func = bpf_ringbuf_discard, .ret_type =RET_VOID , - .arg1_type =ARG_PTR_TO_ALLOC_MEM |OBJ_RELEASE , + .arg1_type =ARG_PTR_TO_RINGBUF_MEM |OBJ_RELEASE , .arg2_type =ARG_ANYTHING , };
kernel/bpf/verifier.c +7 -7
@@ -577,8 +577,8 @@ static const char *reg_type_str(struct bpf_verifier_env *env,if (type &MEM_RDONLY )strncpy (prefix,"rdonly_" ,32 ); -if (type &MEM_ALLOC ) -strncpy (prefix,"alloc_" ,32 ); +if (type &MEM_RINGBUF ) +strncpy (prefix,"ringbuf_" ,32 );if (type &MEM_USER )strncpy (prefix,"user_" ,32 );if (type &MEM_PERCPU )
@@ -5780,7 +5780,7 @@ static const struct bpf_reg_types mem_types = { PTR_TO_MAP_KEY,PTR_TO_MAP_VALUE ,PTR_TO_MEM , - PTR_TO_MEM | MEM_ALLOC, + PTR_TO_MEM | MEM_RINGBUF, PTR_TO_BUF, }, };
@@ -5798,7 +5798,7 @@ static const struct bpf_reg_types int_ptr_types = {static const struct bpf_reg_types fullsock_types = { .types = {PTR_TO_SOCKET } };static const struct bpf_reg_types scalar_types = { .types = {SCALAR_VALUE } };static const struct bpf_reg_types context_types = { .types = {PTR_TO_CTX } }; -static const struct bpf_reg_types alloc_mem_types = { .types = {PTR_TO_MEM |MEM_ALLOC } }; +static const struct bpf_reg_types ringbuf_mem_types = { .types = {PTR_TO_MEM |MEM_RINGBUF } };static const struct bpf_reg_types const_map_ptr_types = { .types = {CONST_PTR_TO_MAP } };static const struct bpf_reg_types btf_ptr_types = { .types = {PTR_TO_BTF_ID } };static const struct bpf_reg_types spin_lock_types = { .types = {PTR_TO_MAP_VALUE } };
@@ -5831,7 +5831,7 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_PTR_TO_BTF_ID] = &btf_ptr_types, [ARG_PTR_TO_SPIN_LOCK] = &spin_lock_types, [ARG_PTR_TO_MEM] = &mem_types, - [ARG_PTR_TO_ALLOC_MEM] = &alloc_mem_types, + [ARG_PTR_TO_RINGBUF_MEM] = &ringbuf_mem_types, [ARG_PTR_TO_INT] = &int_ptr_types, [ARG_PTR_TO_LONG] = &int_ptr_types, [ARG_PTR_TO_PERCPU_BTF_ID] = &percpu_btf_ptr_types,
@@ -5952,14 +5952,14 @@ int check_func_arg_reg_off(struct bpf_verifier_env *env,case PTR_TO_MAP_VALUE :case PTR_TO_MEM :case PTR_TO_MEM |MEM_RDONLY : -case PTR_TO_MEM |MEM_ALLOC : +case PTR_TO_MEM |MEM_RINGBUF :case PTR_TO_BUF :case PTR_TO_BUF |MEM_RDONLY :case SCALAR_VALUE :/* Some of the argument types nevertheless require a * zero register offset. */ -if (base_type (arg_type) !=ARG_PTR_TO_ALLOC_MEM ) +if (base_type (arg_type) !=ARG_PTR_TO_RINGBUF_MEM )return 0 ;break ;/* All the rest must be rejected, except PTR_TO_BTF_ID which allows
tools/testing/selftests/bpf/prog_tests/dynptr.c +1 -1
@@ -17,7 +17,7 @@ static struct { {"ringbuf_missing_release2", "Unreleased reference id=2"}, {"ringbuf_missing_release_callback", "Unreleased reference id"}, {"use_after_invalid", "Expected an initialized dynptr as arg #3"}, - {"ringbuf_invalid_api", "type=mem expected=alloc_mem"}, + {"ringbuf_invalid_api", "type=mem expected=ringbuf_mem"}, {"add_dynptr_to_map1", "invalid indirect read from stack"}, {"add_dynptr_to_map2", "invalid indirect read from stack"}, {"data_slice_out_of_bounds_ringbuf", "value is outside of the allowed memory range"},
tools/testing/selftests/bpf/verifier/ringbuf.c +1 -1
@@ -28,7 +28,7 @@ }, .fixup_map_ringbuf = { 1 }, .result = REJECT, - .errstr ="dereference of modified alloc_mem ptr R1" , + .errstr ="dereference of modified ringbuf_mem ptr R1" , }, { "ringbuf: invalid reservation offset 2",
tools/testing/selftests/bpf/verifier/spill_fill.c +1 -1
@@ -84,7 +84,7 @@ }, .fixup_map_ringbuf = { 1 }, .result = REJECT, - .errstr ="R0 pointer arithmetic on alloc_mem_or_null prohibited" , + .errstr ="R0 pointer arithmetic on ringbuf_mem_or_null prohibited" , }, { "check corrupted spill/fill",
Signature · 1 line
2.38.1
Re: [PATCH bpf-next v5 05/25] bpf: Rename MEM_ALLOC to MEM_RINGBUF
Quoted 11 lines · Kumar Kartikeya Dwivedi
Currently, verifier uses
MEM_ALLOCtype tag to specially tag memory returned frombpf_ringbuf_reservehelper. However, this is currently only used for this purpose and there is an implicit assumption that it only refers to ringbuf memory (e.g. the check forARG_PTR_TO_ALLOC_MEMincheck_func_arg_reg_off).Hence, rename
MEM_ALLOCtoMEM_RINGBUFto indicate this special relationship and instead open the use ofMEM_ALLOCfor more generic allocations made for user types.
Quoted 6 lines
Also, since
ARG_PTR_TO_ALLOC_MEM_OR_NULLis unused, simply drop it.Finally, update selftests using '
alloc_' verifier string to 'ringbuf_'.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
Ok, so you are doing what I asked in the previous patch, nice :)
Quoted 13 lines
include/linux/bpf.h | 11 ++++------- kernel/bpf/ringbuf.c | 6 +++--- kernel/bpf/verifier.c | 14 +++++++------- tools/testing/selftests/bpf/prog_tests/dynptr.c | 2 +- tools/testing/selftests/bpf/verifier/ringbuf.c | 2 +- tools/testing/selftests/bpf/verifier/spill_fill.c | 2 +- 6 files changed, 17 insertions(+), 20 deletions(-)include/linux/bpf.h +0
-0
include/linux/bpf.h enum bpf_type_flag { · line 492
Quoted 9 lines
include/linux/bpf.h +2
-4@@ -488,10 +488,8 @@ enum bpf_type_flag {*/ MEM_RDONLY =BIT (1 +BPF_BASE_TYPE_BITS ), -/* MEM was "allocated" from a different helper, and cannot be mixed -* with regular non-MEM_ALLOC'ed MEM types. -*/ - MEM_ALLOC = BIT(2 + BPF_BASE_TYPE_BITS), +/* MEM points to BPF ring buffer reservation. */ + MEM_RINGBUF = BIT(2 + BPF_BASE_TYPE_BITS),
What do we gain by having ringbuf memory as additional modified flag instead of its own type like PTR_TO_MAP_VALUE or PTR_TO_PACKET? It feels like here separate register type is more justified and is less error prone overall.
Quoted 12 lines
include/linux/bpf.h +1
-1/* MEM is in user address space. */ MEM_USER = BIT(3 + BPF_BASE_TYPE_BITS),@@ -565,7 +563,7 @@ enum bpf_arg_type {ARG_PTR_TO_LONG ,/* pointer to long */ ARG_PTR_TO_SOCKET ,/* pointer to bpf_sock (fullsock) */ ARG_PTR_TO_BTF_ID ,/* pointer to in-kernel struct */ -ARG_PTR_TO_ALLOC_MEM ,/* pointer to dynamically allocated memory */ +ARG_PTR_TO_RINGBUF_MEM ,/* pointer to dynamically reserved ringbuf memory */ ARG_CONST_ALLOC_SIZE_OR_ZERO ,/* number of allocated bytes requested */ ARG_PTR_TO_BTF_ID_SOCK_COMMON ,/* pointer to in-kernel sock_common or bpf-mirrored bpf_sock */ ARG_PTR_TO_PERCPU_BTF_ID, /* pointer to in-kernel percpu type */
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v5 05/25] bpf: Rename MEM_ALLOC to MEM_RINGBUF
Quoted 37 lines · Andrii Nakryiko
Quoted 11 lines · Kumar Kartikeya Dwivedi
Currently, verifier uses
MEM_ALLOCtype tag to specially tag memory returned frombpf_ringbuf_reservehelper. However, this is currently only used for this purpose and there is an implicit assumption that it only refers to ringbuf memory (e.g. the check forARG_PTR_TO_ALLOC_MEMincheck_func_arg_reg_off).Hence, rename
MEM_ALLOCtoMEM_RINGBUFto indicate this special relationship and instead open the use ofMEM_ALLOCfor more generic allocations made for user types.Quoted 6 lines
Also, since
ARG_PTR_TO_ALLOC_MEM_OR_NULLis unused, simply drop it.Finally, update selftests using '
alloc_' verifier string to 'ringbuf_'.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
Ok, so you are doing what I asked in the previous patch, nice :)
Quoted 16 lines
include/linux/bpf.h | 11 ++++------- kernel/bpf/ringbuf.c | 6 +++--- kernel/bpf/verifier.c | 14 +++++++------- tools/testing/selftests/bpf/prog_tests/dynptr.c | 2 +- tools/testing/selftests/bpf/verifier/ringbuf.c | 2 +- tools/testing/selftests/bpf/verifier/spill_fill.c | 2 +- 6 files changed, 17 insertions(+), 20 deletions(-)include/linux/bpf.h +0
-0@@ -488,10 +488,8 @@ enum bpf_type_flag {*/ MEM_RDONLY = BIT(1 + BPF_BASE_TYPE_BITS),
Quoted 12 lines
include/linux/bpf.h enum bpf_type_flag { · line 492
Quoted 6 lines
include/linux/bpf.h +2
-4-/* MEM was "allocated" from a different helper, and cannot be mixed -* with regular non-MEM_ALLOC'ed MEM types. -*/ - MEM_ALLOC = BIT(2 + BPF_BASE_TYPE_BITS), +/* MEM points to BPF ring buffer reservation. */ + MEM_RINGBUF = BIT(2 + BPF_BASE_TYPE_BITS),What do we gain by having ringbuf memory as additional modified flag instead of its own type like
PTR_TO_MAP_VALUEorPTR_TO_PACKET? It feels like here separate register type is more justified and is less error prone overall.
I'm not sure it's all that different. It only matters when checking argument during release. We want to ensure it came from ringbuf_reserve. That's all, apart from that it's no different from PTR_TO_MEM. In all other places it's folded and code for PTR_TO_MEM is used. Same idea as PTR_TO_BTF_ID | MEM_ALLOC.
But I don't feel too strongly, so if you still think it's better I can make the switch.
Re: [PATCH bpf-next v5 05/25] bpf: Rename MEM_ALLOC to MEM_RINGBUF
Quoted 52 lines · Kumar Kartikeya Dwivedi
Quoted 37 lines · Andrii Nakryiko
Quoted 11 lines · Kumar Kartikeya Dwivedi
Currently, verifier uses
MEM_ALLOCtype tag to specially tag memory returned frombpf_ringbuf_reservehelper. However, this is currently only used for this purpose and there is an implicit assumption that it only refers to ringbuf memory (e.g. the check forARG_PTR_TO_ALLOC_MEMincheck_func_arg_reg_off).Hence, rename
MEM_ALLOCtoMEM_RINGBUFto indicate this special relationship and instead open the use ofMEM_ALLOCfor more generic allocations made for user types.Quoted 6 lines
Also, since
ARG_PTR_TO_ALLOC_MEM_OR_NULLis unused, simply drop it.Finally, update selftests using '
alloc_' verifier string to 'ringbuf_'.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
Ok, so you are doing what I asked in the previous patch, nice :)
Quoted 16 lines
include/linux/bpf.h | 11 ++++------- kernel/bpf/ringbuf.c | 6 +++--- kernel/bpf/verifier.c | 14 +++++++------- tools/testing/selftests/bpf/prog_tests/dynptr.c | 2 +- tools/testing/selftests/bpf/verifier/ringbuf.c | 2 +- tools/testing/selftests/bpf/verifier/spill_fill.c | 2 +- 6 files changed, 17 insertions(+), 20 deletions(-)include/linux/bpf.h +0
-0@@ -488,10 +488,8 @@ enum bpf_type_flag {*/ MEM_RDONLY = BIT(1 + BPF_BASE_TYPE_BITS),Quoted 12 lines
include/linux/bpf.h enum bpf_type_flag { · line 492
Quoted 6 lines
include/linux/bpf.h +2
-4-/* MEM was "allocated" from a different helper, and cannot be mixed -* with regular non-MEM_ALLOC'ed MEM types. -*/ - MEM_ALLOC = BIT(2 + BPF_BASE_TYPE_BITS), +/* MEM points to BPF ring buffer reservation. */ + MEM_RINGBUF = BIT(2 + BPF_BASE_TYPE_BITS),What do we gain by having ringbuf memory as additional modified flag instead of its own type like
PTR_TO_MAP_VALUEorPTR_TO_PACKET? It feels like here separate register type is more justified and is less error prone overall.
Quoted 7 lines
I'm not sure it's all that different. It only matters when checking argument during release. We want to ensure it came from
ringbuf_reserve. That's all, apart from that it's no different fromPTR_TO_MEM. In all other places it's folded and code forPTR_TO_MEMis used. Same idea asPTR_TO_BTF_ID|MEM_ALLOC.But I don't feel too strongly, so if you still think it's better I can make the switch.
Not strongly, but I think having this as a flag is more error prone. For cases where ringbuf memory should be treated just as memory, we should use the same mechanism we have for MAP_VALUE. But I haven't checked how we deal with MAP_VALUE, if that's a special case everywhere, in addition to generic PTR_TO_MEM, then fine. But if having PTR_TO_RINGBUF_MEM is converted to PTR_TO_MEM generically where needed, I'd have dedicated PTR_TO_RINGBUF_MEM.
Re: [PATCH bpf-next v5 05/25] bpf: Rename MEM_ALLOC to MEM_RINGBUF
Quoted 62 lines · Andrii Nakryiko
Quoted 52 lines · Kumar Kartikeya Dwivedi
Quoted 37 lines · Andrii Nakryiko
Quoted 11 lines · Kumar Kartikeya Dwivedi
Currently, verifier uses
MEM_ALLOCtype tag to specially tag memory returned frombpf_ringbuf_reservehelper. However, this is currently only used for this purpose and there is an implicit assumption that it only refers to ringbuf memory (e.g. the check forARG_PTR_TO_ALLOC_MEMincheck_func_arg_reg_off).Hence, rename
MEM_ALLOCtoMEM_RINGBUFto indicate this special relationship and instead open the use ofMEM_ALLOCfor more generic allocations made for user types.Quoted 6 lines
Also, since
ARG_PTR_TO_ALLOC_MEM_OR_NULLis unused, simply drop it.Finally, update selftests using '
alloc_' verifier string to 'ringbuf_'.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
Ok, so you are doing what I asked in the previous patch, nice :)
Quoted 16 lines
include/linux/bpf.h | 11 ++++------- kernel/bpf/ringbuf.c | 6 +++--- kernel/bpf/verifier.c | 14 +++++++------- tools/testing/selftests/bpf/prog_tests/dynptr.c | 2 +- tools/testing/selftests/bpf/verifier/ringbuf.c | 2 +- tools/testing/selftests/bpf/verifier/spill_fill.c | 2 +- 6 files changed, 17 insertions(+), 20 deletions(-)include/linux/bpf.h +0
-0@@ -488,10 +488,8 @@ enum bpf_type_flag {*/ MEM_RDONLY = BIT(1 + BPF_BASE_TYPE_BITS),Quoted 12 lines
include/linux/bpf.h enum bpf_type_flag { · line 492
Quoted 6 lines
include/linux/bpf.h +2
-4-/* MEM was "allocated" from a different helper, and cannot be mixed -* with regular non-MEM_ALLOC'ed MEM types. -*/ - MEM_ALLOC = BIT(2 + BPF_BASE_TYPE_BITS), +/* MEM points to BPF ring buffer reservation. */ + MEM_RINGBUF = BIT(2 + BPF_BASE_TYPE_BITS),What do we gain by having ringbuf memory as additional modified flag instead of its own type like
PTR_TO_MAP_VALUEorPTR_TO_PACKET? It feels like here separate register type is more justified and is less error prone overall.Quoted 7 lines
I'm not sure it's all that different. It only matters when checking argument during release. We want to ensure it came from
ringbuf_reserve. That's all, apart from that it's no different fromPTR_TO_MEM. In all other places it's folded and code forPTR_TO_MEMis used. Same idea asPTR_TO_BTF_ID|MEM_ALLOC.But I don't feel too strongly, so if you still think it's better I can make the switch.
Quoted 7 lines
Not strongly, but I think having this as a flag is more error prone. For cases where ringbuf memory should be treated just as memory, we should use the same mechanism we have for
MAP_VALUE. But I haven't checked how we deal withMAP_VALUE, if that's a special case everywhere, in addition to genericPTR_TO_MEM, then fine. But if havingPTR_TO_RINGBUF_MEMis converted toPTR_TO_MEMgenerically where needed, I'd have dedicatedPTR_TO_RINGBUF_MEM.
I don't think we can or at least it's not as easy to generalize ringbuf mem as map_value.
iirc MEM_ALLOC was there to make sure reserver->commit is the same mem. That's what MEM_RINGBUF will doing after this patch.
Re: [PATCH bpf-next v5 05/25] bpf: Rename MEM_ALLOC to MEM_RINGBUF
Quoted 73 lines · Alexei Starovoitov
Quoted 62 lines · Andrii Nakryiko
Quoted 52 lines · Kumar Kartikeya Dwivedi
Quoted 37 lines · Andrii Nakryiko
Quoted 11 lines · Kumar Kartikeya Dwivedi
Currently, verifier uses
MEM_ALLOCtype tag to specially tag memory returned frombpf_ringbuf_reservehelper. However, this is currently only used for this purpose and there is an implicit assumption that it only refers to ringbuf memory (e.g. the check forARG_PTR_TO_ALLOC_MEMincheck_func_arg_reg_off).Hence, rename
MEM_ALLOCtoMEM_RINGBUFto indicate this special relationship and instead open the use ofMEM_ALLOCfor more generic allocations made for user types.Quoted 6 lines
Also, since
ARG_PTR_TO_ALLOC_MEM_OR_NULLis unused, simply drop it.Finally, update selftests using '
alloc_' verifier string to 'ringbuf_'.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
Ok, so you are doing what I asked in the previous patch, nice :)
Quoted 16 lines
include/linux/bpf.h | 11 ++++------- kernel/bpf/ringbuf.c | 6 +++--- kernel/bpf/verifier.c | 14 +++++++------- tools/testing/selftests/bpf/prog_tests/dynptr.c | 2 +- tools/testing/selftests/bpf/verifier/ringbuf.c | 2 +- tools/testing/selftests/bpf/verifier/spill_fill.c | 2 +- 6 files changed, 17 insertions(+), 20 deletions(-)include/linux/bpf.h +0
-0@@ -488,10 +488,8 @@ enum bpf_type_flag {*/ MEM_RDONLY = BIT(1 + BPF_BASE_TYPE_BITS),Quoted 12 lines
include/linux/bpf.h enum bpf_type_flag { · line 492
Quoted 6 lines
include/linux/bpf.h +2
-4-/* MEM was "allocated" from a different helper, and cannot be mixed -* with regular non-MEM_ALLOC'ed MEM types. -*/ - MEM_ALLOC = BIT(2 + BPF_BASE_TYPE_BITS), +/* MEM points to BPF ring buffer reservation. */ + MEM_RINGBUF = BIT(2 + BPF_BASE_TYPE_BITS),What do we gain by having ringbuf memory as additional modified flag instead of its own type like
PTR_TO_MAP_VALUEorPTR_TO_PACKET? It feels like here separate register type is more justified and is less error prone overall.Quoted 7 lines
I'm not sure it's all that different. It only matters when checking argument during release. We want to ensure it came from
ringbuf_reserve. That's all, apart from that it's no different fromPTR_TO_MEM. In all other places it's folded and code forPTR_TO_MEMis used. Same idea asPTR_TO_BTF_ID|MEM_ALLOC.But I don't feel too strongly, so if you still think it's better I can make the switch.
Quoted 7 lines
Not strongly, but I think having this as a flag is more error prone. For cases where ringbuf memory should be treated just as memory, we should use the same mechanism we have for
MAP_VALUE. But I haven't checked how we deal withMAP_VALUE, if that's a special case everywhere, in addition to genericPTR_TO_MEM, then fine. But if havingPTR_TO_RINGBUF_MEMis converted toPTR_TO_MEMgenerically where needed, I'd have dedicatedPTR_TO_RINGBUF_MEM.
Quoted 4 lines
I don't think we can or at least it's not as easy to generalize ringbuf mem as
map_value.
iircMEM_ALLOCwas there to make surereserver->commitis the same mem. That's whatMEM_RINGBUFwill doing after this patch.
I'm not sure we are talking about the same thing. My only point was to have RINGBUF_MEM as *base type* instead of as an *add-on flag*. MAP_VALUE was an example of another special register type that is base type but behaves like PTR_TO_MEM for helpers that don't care about where specifically memory is coming from. I didn't mean to unify RINGBUF_MEM and MAP_VALUE in any way (if that's what you are proposing, I'm actually not sure exactly).
But as I said, no big deal with a flag, we can always change that in the future as well.
Kumar Kartikeya Dwivedi
maintainerPATCH v5 6/25 bpf: Introduce local kptrs
In reply to Kumar Kartikeya DwivediIntroduce local kptrs, i.e. PTR_TO_BTF_ID that point to a type in program BTF. This is indicated by the presence of MEM_ALLOC type flag in reg->type to avoid having to check btf_is_kernel when trying to match argument types in helpers.
Refactor btf_struct_access callback to just take bpf_reg_state instead of btf and btf_type paramters. Note that the call site in check_map_access now simulates access to a PTR_TO_BTF_ID by creating a dummy reg on stack. Since only the type, btf, and btf_id of the register matter for the checks, it can be done so without complicating the usual cases elsewhere in the verifier where reg->btf and reg->btf_id is used verbatim.
Whenever walking such types, any pointers being walked will always yield a SCALAR instead of pointer. In the future we might permit kptr inside local kptr (either kernel or local), and it would be permitted only in that case.
For now, these local kptrs will always be referenced in verifier context, hence ref_obj_id == 0 for them is a bug. It is allowed to write to such objects, as long fields that are special are not touched (support for which will be added in subsequent patches). Note that once such a local kptr is marked PTR_UNTRUSTED, it is no longer allowed to write to it.
No PROBE_MEM handling is therefore done for loads into this type unless PTR_UNTRUSTED is part of the register type, since they can never be in an undefined state, and their lifetime will always be valid.
--- include/linux/bpf.h | 28 ++++++++++++++++-------- include/linux/filter.h | 8 +++---- kernel/bpf/btf.c | 16 ++++++++++---- kernel/bpf/verifier.c | 37 ++++++++++++++++++++++++++------ net/bpf/bpf_dummy_struct_ops.c | 14 ++++++------ net/core/filter.c | 34 ++++++++++++----------------- net/ipv4/bpf_tcp_ca.c | 13 ++++++----- net/netfilter/nf_conntrack_bpf.c | 17 ++++++--------- 8 files changed, 99 insertions(+), 68 deletions(-)
include/linux/bpf.h +19 -9
@@ -524,6 +524,11 @@ enum bpf_type_flag {/* Size is known at compile time. */ MEM_FIXED_SIZE =BIT (10 +BPF_BASE_TYPE_BITS ), +/* MEM is of a type from program BTF, not kernel BTF. This is used to +* tag PTR_TO_BTF_ID allocated using bpf_obj_new. +*/ +MEM_ALLOC =BIT (11 +BPF_BASE_TYPE_BITS ), + __BPF_TYPE_FLAG_MAX, __BPF_TYPE_LAST_FLAG = __BPF_TYPE_FLAG_MAX - 1, };
@@ -771,6 +776,7 @@ struct bpf_prog_ops { union bpf_attr __user *uattr); }; +struct bpf_reg_state ;struct bpf_verifier_ops {/* return eBPF function prototype for verification */ const struct bpf_func_proto *
@@ -792,9 +798,8 @@ struct bpf_verifier_ops {struct bpf_insn *dst, struct bpf_prog *prog, u32 *target_size);int (*btf_struct_access)(struct bpf_verifier_log *log, -const struct btf *btf, -const struct btf_type *t,int off,int size, -enum bpf_access_type atype, +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype,u32 *next_btf_id,enum bpf_type_flag *flag); };
@@ -2080,9 +2085,9 @@ static inline bool bpf_tracing_btf_ctx_access(int off, int size,return btf_ctx_access (off, size, type, prog, info); } -int btf_struct_access (struct bpf_verifier_log *log,const struct btf *btf, -const struct btf_type *t,int off,int size, -enum bpf_access_type atype, +int btf_struct_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype,u32 *next_btf_id,enum bpf_type_flag *flag); bool btf_struct_ids_match(struct bpf_verifier_log *log, const struct btf *btf, u32 id, int off,
@@ -2333,9 +2338,8 @@ static inline struct bpf_prog *bpf_prog_by_id(u32 id) }static inline int btf_struct_access (struct bpf_verifier_log *log, -const struct btf *btf, -const struct btf_type *t,int off,int size, -enum bpf_access_type atype, +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype,u32 *next_btf_id,enum bpf_type_flag *flag) {return -EACCES ;
@@ -2792,4 +2796,10 @@ struct bpf_key {bool has_ref; };#endif /* CONFIG_KEYS */ + +static inline bool type_is_local_kptr (u32 type) +{ +return type &MEM_ALLOC ; +} +#endif /* _LINUX_BPF_H */
include/linux/filter.h +4 -4
@@ -568,10 +568,10 @@ struct sk_filter {DECLARE_STATIC_KEY_FALSE (bpf_stats_enabled_key);extern struct mutex nf_conn_btf_access_lock; -extern int (*nfct_btf_struct_access)(struct bpf_verifier_log *log,const struct btf *btf, -const struct btf_type *t,int off,int size, -enum bpf_access_type atype,u32 *next_btf_id, -enum bpf_type_flag *flag); +extern int (*nfct_btf_struct_access)(struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype, +u32 *next_btf_id,enum bpf_type_flag *flag); typedef unsigned int (*bpf_dispatcher_fn)(const void *ctx, const struct bpf_insn *insnsi,
kernel/bpf/btf.c +12 -4
@@ -6015,20 +6015,28 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,return -EINVAL ; } -int btf_struct_access (struct bpf_verifier_log *log,const struct btf *btf, -const struct btf_type *t,int off,int size, -enum bpf_access_type atype__maybe_unused , +int btf_struct_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype__maybe_unused ,u32 *next_btf_id,enum bpf_type_flag *flag) { +const struct btf *btf = reg->btf;enum bpf_type_flag tmp_flag =0 ; +const struct btf_type *t; +u32 id = reg->btf_id;int err; -u32 id; + t =btf_type_by_id (btf, id); do { err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag); switch (err) { case WALK_PTR: + /* For local types, the destination register cannot + * become a pointer again. + */ + if (type_is_local_kptr(reg->type)) + return SCALAR_VALUE; /* If we found the pointer or scalar on t+off, * we're done. */
kernel/bpf/verifier.c +30 -7
@@ -4682,17 +4682,28 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,return -EACCES ; } -if (env->ops->btf_struct_access) { - ret = env->ops->btf_struct_access (&env->log, reg->btf, t, - off, size, atype, &btf_id, &flag); +if (env->ops->btf_struct_access && !type_is_local_kptr (reg->type)) { +if (!btf_is_kernel (reg->btf)) { +verbose (env,"verifier internal error: reg->btf must be kernel btf\n" ); +return -EFAULT ; + } + ret = env->ops->btf_struct_access (&env->log, reg, off, size, atype, &btf_id, &flag); }else { -if (atype !=BPF_READ ) { +/* Writes are permitted with default btf_struct_access for local +* kptrs (which always have ref_obj_id > 0), but not for +* _untrusted_ local kptrs. +*/ +if (atype !=BPF_READ && reg->type != (PTR_TO_BTF_ID |MEM_ALLOC )) {verbose (env,"only read is supported\n" );return -EACCES ; } - ret =btf_struct_access (&env->log, reg->btf, t, off, size, - atype, &btf_id, &flag); +if (type_is_local_kptr (reg->type) && !reg->ref_obj_id) { +verbose (env,"verifier internal error: ref_obj_id for local kptr must be non-zero\n" ); +return -EFAULT ; + } + + ret =btf_struct_access (&env->log, reg, off, size, atype, &btf_id, &flag); }if (ret <0 )
@@ -4718,6 +4729,7 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env, {struct bpf_reg_state *reg = regs + regno;struct bpf_map *map = reg->map_ptr; +struct bpf_reg_state map_reg;enum bpf_type_flag flag =0 ;const struct btf_type *t;const char *tname;
@@ -4756,7 +4768,10 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env,return -EACCES ; } - ret =btf_struct_access (&env->log, btf_vmlinux, t, off, size, atype, &btf_id, &flag); +/* Simulate access to a PTR_TO_BTF_ID */ +memset (&map_reg,0 ,sizeof (map_reg)); +mark_btf_ld_reg (env, &map_reg,0 ,PTR_TO_BTF_ID , btf_vmlinux, *map->ops->map_btf_id,0 ); + ret =btf_struct_access (&env->log, &map_reg, off, size, atype, &btf_id, &flag);if (ret <0 )return ret;
@@ -5966,6 +5981,7 @@ int check_func_arg_reg_off(struct bpf_verifier_env *env,* fixed offset. */ case PTR_TO_BTF_ID : +case PTR_TO_BTF_ID |MEM_ALLOC :/* When referenced PTR_TO_BTF_ID is passed to release function, * it's fixed offset must be 0. In the other cases, fixed offset * can be non-zero.
@@ -13648,6 +13664,13 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)break ;case PTR_TO_BTF_ID :case PTR_TO_BTF_ID |PTR_UNTRUSTED : +/* PTR_TO_BTF_ID | MEM_ALLOC always has a valid lifetime, unlike +* PTR_TO_BTF_ID, and an active ref_obj_id, but the same cannot +* be said once it is marked PTR_UNTRUSTED, hence we must handle +* any faults for loads into such types. BPF_WRITE is disallowed +* for this case. +*/ +case PTR_TO_BTF_ID |MEM_ALLOC |PTR_UNTRUSTED :if (type ==BPF_READ ) { insn->code =BPF_LDX |BPF_PROBE_MEM |BPF_SIZE ((insn)->code);
net/bpf/bpf_dummy_struct_ops.c +7 -7
@@ -156,29 +156,29 @@ static bool bpf_dummy_ops_is_valid_access(int off, int size, }static int bpf_dummy_ops_btf_struct_access (struct bpf_verifier_log *log, -const struct btf *btf, -const struct btf_type *t,int off, -int size,enum bpf_access_type atype, +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype,u32 *next_btf_id,enum bpf_type_flag *flag) {const struct btf_type *state; +const struct btf_type *t;s32 type_id;int err; - type_id =btf_find_by_name_kind (btf,"bpf_dummy_ops_state" , + type_id =btf_find_by_name_kind (reg->btf,"bpf_dummy_ops_state" ,BTF_KIND_STRUCT );if (type_id <0 )return -EINVAL ; - state =btf_type_by_id (btf, type_id); + t =btf_type_by_id (reg->btf, reg->btf_id); + state =btf_type_by_id (reg->btf, type_id);if (t != state) {bpf_log (log,"only access to bpf_dummy_ops_state is supported\n" );return -EACCES ; } - err =btf_struct_access (log, btf, t, off, size, atype, next_btf_id, - flag); + err =btf_struct_access (log, reg, off, size, atype, next_btf_id, flag);if (err <0 )return err;
net/core/filter.c +14 -20
@@ -8651,28 +8651,25 @@ static bool tc_cls_act_is_valid_access(int off, int size,DEFINE_MUTEX (nf_conn_btf_access_lock);EXPORT_SYMBOL_GPL (nf_conn_btf_access_lock); -int (*nfct_btf_struct_access)(struct bpf_verifier_log *log,const struct btf *btf, -const struct btf_type *t,int off,int size, -enum bpf_access_type atype,u32 *next_btf_id, -enum bpf_type_flag *flag); +int (*nfct_btf_struct_access)(struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype, +u32 *next_btf_id,enum bpf_type_flag *flag);EXPORT_SYMBOL_GPL (nfct_btf_struct_access);static int tc_cls_act_btf_struct_access (struct bpf_verifier_log *log, -const struct btf *btf, -const struct btf_type *t,int off, -int size,enum bpf_access_type atype, -u32 *next_btf_id, -enum bpf_type_flag *flag) +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype, +u32 *next_btf_id,enum bpf_type_flag *flag) {int ret = -EACCES ;if (atype ==BPF_READ ) -return btf_struct_access (log, btf, t, off, size, atype, next_btf_id, - flag); +return btf_struct_access (log, reg, off, size, atype, next_btf_id, flag);mutex_lock (&nf_conn_btf_access_lock);if (nfct_btf_struct_access) - ret =nfct_btf_struct_access (log, btf, t, off, size, atype, next_btf_id, flag); + ret =nfct_btf_struct_access (log, reg, off, size, atype, next_btf_id, flag);mutex_unlock (&nf_conn_btf_access_lock);return ret;
@@ -8738,21 +8735,18 @@ void bpf_warn_invalid_xdp_action(struct net_device *dev, struct bpf_prog *prog,EXPORT_SYMBOL_GPL (bpf_warn_invalid_xdp_action);static int xdp_btf_struct_access (struct bpf_verifier_log *log, -const struct btf *btf, -const struct btf_type *t,int off, -int size,enum bpf_access_type atype, -u32 *next_btf_id, -enum bpf_type_flag *flag) +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype, +u32 *next_btf_id,enum bpf_type_flag *flag) {int ret = -EACCES ;if (atype ==BPF_READ ) -return btf_struct_access (log, btf, t, off, size, atype, next_btf_id, - flag); +return btf_struct_access (log, reg, off, size, atype, next_btf_id, flag);mutex_lock (&nf_conn_btf_access_lock);if (nfct_btf_struct_access) - ret =nfct_btf_struct_access (log, btf, t, off, size, atype, next_btf_id, flag); + ret =nfct_btf_struct_access (log, reg, off, size, atype, next_btf_id, flag);mutex_unlock (&nf_conn_btf_access_lock);return ret;
net/ipv4/bpf_tcp_ca.c +6 -7
@@ -69,18 +69,17 @@ static bool bpf_tcp_ca_is_valid_access(int off, int size, }static int bpf_tcp_ca_btf_struct_access (struct bpf_verifier_log *log, -const struct btf *btf, -const struct btf_type *t,int off, -int size,enum bpf_access_type atype, -u32 *next_btf_id, -enum bpf_type_flag *flag) +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype, +u32 *next_btf_id,enum bpf_type_flag *flag) { +const struct btf_type *t;size_t end;if (atype ==BPF_READ ) -return btf_struct_access (log, btf, t, off, size, atype, next_btf_id, - flag); +return btf_struct_access (log, reg, off, size, atype, next_btf_id, flag); + t =btf_type_by_id (reg->btf, reg->btf_id);if (t != tcp_sock_type) {bpf_log (log,"only read is supported\n" );return -EACCES ;
net/netfilter/nf_conntrack_bpf.c +7 -10
@@ -191,19 +191,16 @@ BTF_ID(struct, nf_conn___init)/* Check writes into `struct nf_conn` */ static int _nf_conntrack_btf_struct_access (struct bpf_verifier_log *log, -const struct btf *btf, -const struct btf_type *t,int off, -int size,enum bpf_access_type atype, -u32 *next_btf_id, -enum bpf_type_flag *flag) +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype, +u32 *next_btf_id,enum bpf_type_flag *flag) { -const struct btf_type *ncit; -const struct btf_type *nct; +const struct btf_type *ncit, *nct, *t;size_t end; - ncit =btf_type_by_id (btf, btf_nf_conn_ids[1 ]); - nct =btf_type_by_id (btf, btf_nf_conn_ids[0 ]); - + ncit =btf_type_by_id (reg->btf, btf_nf_conn_ids[1 ]); + nct =btf_type_by_id (reg->btf, btf_nf_conn_ids[0 ]); + t =btf_type_by_id (reg->btf, reg->btf_id);if (t != nct && t != ncit) {bpf_log (log,"only read is supported\n" );return -EACCES ;
Signature · 1 line
2.38.1
Re: [PATCH bpf-next v5 06/25] bpf: Introduce local kptrs
Quoted 47 lines · Kumar Kartikeya Dwivedi
Introduce local kptrs, i.e.
PTR_TO_BTF_IDthat point to a type in program BTF. This is indicated by the presence ofMEM_ALLOCtype flag inreg->typeto avoid having to checkbtf_is_kernelwhen trying to match argument types in helpers.Refactor
btf_struct_accesscallback to just takebpf_reg_stateinstead of btf andbtf_typeparamters. Note that the call site incheck_map_accessnow simulates access to aPTR_TO_BTF_IDby creating a dummy reg on stack. Since only the type, btf, andbtf_idof the register matter for the checks, it can be done so without complicating the usual cases elsewhere in the verifier wherereg->btfandreg->btf_idis used verbatim.Whenever walking such types, any pointers being walked will always yield a SCALAR instead of pointer. In the future we might permit kptr inside local kptr (either kernel or local), and it would be permitted only in that case.
For now, these local kptrs will always be referenced in verifier context, hence
ref_obj_id== 0 for them is a bug. It is allowed to write to such objects, as long fields that are special are not touched (support for which will be added in subsequent patches). Note that once such a local kptr is markedPTR_UNTRUSTED, it is no longer allowed to write to it.No
PROBE_MEMhandling is therefore done for loads into this type unlessPTR_UNTRUSTEDis part of the register type, since they can never be in an undefined state, and their lifetime will always be valid.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 28 ++++++++++++++++-------- include/linux/filter.h | 8 +++---- kernel/bpf/btf.c | 16 ++++++++++---- kernel/bpf/verifier.c | 37 ++++++++++++++++++++++++++------ net/bpf/bpf_dummy_struct_ops.c | 14 ++++++------ net/core/filter.c | 34 ++++++++++++----------------- net/ipv4/bpf_tcp_ca.c | 13 ++++++----- net/netfilter/nf_conntrack_bpf.c | 17 ++++++--------- 8 files changed, 99 insertions(+), 68 deletions(-)include/linux/bpf.h +0
-0
include/linux/bpf.h enum bpf_type_flag { · line 531
Quoted 8 lines
include/linux/bpf.h +5
-0@@ -524,6 +524,11 @@ enum bpf_type_flag {/* Size is known at compile time. */ MEM_FIXED_SIZE =BIT (10 +BPF_BASE_TYPE_BITS ), +/* MEM is of a type from program BTF, not kernel BTF. This is used to +* tag PTR_TO_BTF_ID allocated using bpf_obj_new. +*/ + MEM_ALLOC = BIT(11 + BPF_BASE_TYPE_BITS), +
you fixed one naming confusion with RINGBUF and basically are creating a new one, where "ALLOC" means "local kptr"... If we are stuck with "local kptr" (which I find very confusing as well, but that's beside the point), why not stick to the whole "local" terminology here? MEM_LOCAL?
Quoted 7 lines
include/linux/bpf.h +0
-0__BPF_TYPE_FLAG_MAX, __BPF_TYPE_LAST_FLAG = __BPF_TYPE_FLAG_MAX - 1, };@@ -771,6 +776,7 @@ struct bpf_prog_ops { union bpf_attr __user *uattr); };
[...]
Quoted 16 lines
include/linux/bpf.h +7
-4-int btf_struct_access (struct bpf_verifier_log *log,const struct btf *btf, -const struct btf_type *t,int off,int size, -enum bpf_access_type atype__maybe_unused , +int btf_struct_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype__maybe_unused ,u32 *next_btf_id,enum bpf_type_flag *flag) { +const struct btf *btf = reg->btf;enum bpf_type_flag tmp_flag =0 ; +const struct btf_type *t; +u32 id = reg->btf_id;int err; -u32 id; + t =btf_type_by_id (btf, id);
include/linux/bpf.h
Quoted 10 lines
include/linux/bpf.h +5
-0do { err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag); switch (err) { case WALK_PTR: + /* For local types, the destination register cannot + * become a pointer again. + */ + if (type_is_local_kptr(reg->type)) + return SCALAR_VALUE;
passing the entire bpf_reg_state just to differentiate between local vs kernel pointer seems like a huge overkill. bpf_reg_state is quite a complicated and extensive amount of state, and it seems cleaner to just pass it as a flag whether to allow pointer chasing or not. At least then we know we only care about that specific aspect, not about dozens of other possible fields of bpf_reg_state.
Quoted 3 lines
/* If we found the pointer or scalar on t+off,
* we're done.
*/[...]
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v5 06/25] bpf: Introduce local kptrs
Quoted 51 lines · Andrii Nakryiko
Quoted 50 lines · Kumar Kartikeya Dwivedi
Introduce local kptrs, i.e.
PTR_TO_BTF_IDthat point to a type in program BTF. This is indicated by the presence ofMEM_ALLOCtype flag inreg->typeto avoid having to checkbtf_is_kernelwhen trying to match argument types in helpers.Refactor
btf_struct_accesscallback to just takebpf_reg_stateinstead of btf andbtf_typeparamters. Note that the call site incheck_map_accessnow simulates access to aPTR_TO_BTF_IDby creating a dummy reg on stack. Since only the type, btf, andbtf_idof the register matter for the checks, it can be done so without complicating the usual cases elsewhere in the verifier wherereg->btfandreg->btf_idis used verbatim.Whenever walking such types, any pointers being walked will always yield a SCALAR instead of pointer. In the future we might permit kptr inside local kptr (either kernel or local), and it would be permitted only in that case.
For now, these local kptrs will always be referenced in verifier context, hence
ref_obj_id== 0 for them is a bug. It is allowed to write to such objects, as long fields that are special are not touched (support for which will be added in subsequent patches). Note that once such a local kptr is markedPTR_UNTRUSTED, it is no longer allowed to write to it.No
PROBE_MEMhandling is therefore done for loads into this type unlessPTR_UNTRUSTEDis part of the register type, since they can never be in an undefined state, and their lifetime will always be valid.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 28 ++++++++++++++++-------- include/linux/filter.h | 8 +++---- kernel/bpf/btf.c | 16 ++++++++++---- kernel/bpf/verifier.c | 37 ++++++++++++++++++++++++++------ net/bpf/bpf_dummy_struct_ops.c | 14 ++++++------ net/core/filter.c | 34 ++++++++++++----------------- net/ipv4/bpf_tcp_ca.c | 13 ++++++----- net/netfilter/nf_conntrack_bpf.c | 17 ++++++--------- 8 files changed, 99 insertions(+), 68 deletions(-)include/linux/bpf.h +0
-0@@ -524,6 +524,11 @@ enum bpf_type_flag {/* Size is known at compile time. */ MEM_FIXED_SIZE = BIT(10 + BPF_BASE_TYPE_BITS),
Quoted 12 lines
include/linux/bpf.h enum bpf_type_flag { · line 531
Quoted 5 lines
include/linux/bpf.h +5
-0+/* MEM is of a type from program BTF, not kernel BTF. This is used to +* tag PTR_TO_BTF_ID allocated using bpf_obj_new. +*/ + MEM_ALLOC = BIT(11 + BPF_BASE_TYPE_BITS), +you fixed one naming confusion with RINGBUF and basically are creating a new one, where "ALLOC" means "local kptr"... If we are stuck with "local kptr" (which I find very confusing as well, but that's beside the point), why not stick to the whole "local" terminology here?
MEM_LOCAL?
See the discussion about this in v4:
https://lore.kernel.org/bpf/20221104075113.5ighwdvero4mugu7@apollo
It was MEM_TYPE_LOCAL before. Also, better naming suggestions are always welcome, I asked the same in that message as well.
Quoted 29 lines
Quoted 7 lines
include/linux/bpf.h +0
-0__BPF_TYPE_FLAG_MAX, __BPF_TYPE_LAST_FLAG = __BPF_TYPE_FLAG_MAX - 1, };@@ -771,6 +776,7 @@ struct bpf_prog_ops { union bpf_attr __user *uattr); };[...]
Quoted 19 lines
include/linux/bpf.h +7
-4-int btf_struct_access (struct bpf_verifier_log *log,const struct btf *btf, -const struct btf_type *t,int off,int size, -enum bpf_access_type atype__maybe_unused , +int btf_struct_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype__maybe_unused ,u32 *next_btf_id,enum bpf_type_flag *flag) { +const struct btf *btf = reg->btf;enum bpf_type_flag tmp_flag =0 ; +const struct btf_type *t; +u32 id = reg->btf_id;int err; -u32 id; + t =btf_type_by_id (btf, id); do { err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag);
Quoted 15 lines
include/linux/bpf.h
Quoted 7 lines
switch (err) { case WALK_PTR:include/linux/bpf.h +5
-0+/* For local types, the destination register cannot +* become a pointer again. +*/ +if (type_is_local_kptr (reg->type)) +return SCALAR_VALUE ;passing the entire
bpf_reg_statejust to differentiate between local vs kernel pointer seems like a huge overkill.bpf_reg_stateis quite a complicated and extensive amount of state, and it seems cleaner to just pass it as a flag whether to allow pointer chasing or not. At least then we know we only care about that specific aspect, not about dozens of other possible fields ofbpf_reg_state.
I agree that the separation is usually better, especially because this is also a callback. I don't feel too strong about this though, we certainly do pass the whole reg to functions which only work on a specific type of pointer. Though the concern in this case is justified as it's not only an internal function but also a callback.
It was just a bool in the RFC.
But in https://lore.kernel.org/bpf/20220907233023.x3uclwlnjuhftvtb@macbook-pro-4.dhcp.thefacebook.com
Alexei suggested passing reg instead.
From the link:
Quoted 3 lines
imo it's cleaner to pass 'reg' instead of '
reg->btf', so we don't have to pass another boolean.
And checktype_is_local(reg)insidebtf_struct_access().
Re: [PATCH bpf-next v5 06/25] bpf: Introduce local kptrs
Quoted 66 lines · Kumar Kartikeya Dwivedi
Quoted 51 lines · Andrii Nakryiko
Quoted 50 lines · Kumar Kartikeya Dwivedi
Introduce local kptrs, i.e.
PTR_TO_BTF_IDthat point to a type in program BTF. This is indicated by the presence ofMEM_ALLOCtype flag inreg->typeto avoid having to checkbtf_is_kernelwhen trying to match argument types in helpers.Refactor
btf_struct_accesscallback to just takebpf_reg_stateinstead of btf andbtf_typeparamters. Note that the call site incheck_map_accessnow simulates access to aPTR_TO_BTF_IDby creating a dummy reg on stack. Since only the type, btf, andbtf_idof the register matter for the checks, it can be done so without complicating the usual cases elsewhere in the verifier wherereg->btfandreg->btf_idis used verbatim.Whenever walking such types, any pointers being walked will always yield a SCALAR instead of pointer. In the future we might permit kptr inside local kptr (either kernel or local), and it would be permitted only in that case.
For now, these local kptrs will always be referenced in verifier context, hence
ref_obj_id== 0 for them is a bug. It is allowed to write to such objects, as long fields that are special are not touched (support for which will be added in subsequent patches). Note that once such a local kptr is markedPTR_UNTRUSTED, it is no longer allowed to write to it.No
PROBE_MEMhandling is therefore done for loads into this type unlessPTR_UNTRUSTEDis part of the register type, since they can never be in an undefined state, and their lifetime will always be valid.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 28 ++++++++++++++++-------- include/linux/filter.h | 8 +++---- kernel/bpf/btf.c | 16 ++++++++++---- kernel/bpf/verifier.c | 37 ++++++++++++++++++++++++++------ net/bpf/bpf_dummy_struct_ops.c | 14 ++++++------ net/core/filter.c | 34 ++++++++++++----------------- net/ipv4/bpf_tcp_ca.c | 13 ++++++----- net/netfilter/nf_conntrack_bpf.c | 17 ++++++--------- 8 files changed, 99 insertions(+), 68 deletions(-)include/linux/bpf.h +0
-0@@ -524,6 +524,11 @@ enum bpf_type_flag {/* Size is known at compile time. */ MEM_FIXED_SIZE = BIT(10 + BPF_BASE_TYPE_BITS),Quoted 12 lines
include/linux/bpf.h enum bpf_type_flag { · line 531
Quoted 5 lines
include/linux/bpf.h +5
-0+/* MEM is of a type from program BTF, not kernel BTF. This is used to +* tag PTR_TO_BTF_ID allocated using bpf_obj_new. +*/ + MEM_ALLOC = BIT(11 + BPF_BASE_TYPE_BITS), +you fixed one naming confusion with RINGBUF and basically are creating a new one, where "ALLOC" means "local kptr"... If we are stuck with "local kptr" (which I find very confusing as well, but that's beside the point), why not stick to the whole "local" terminology here?
MEM_LOCAL?
Quoted 5 lines
See the discussion about this in v4:
https://lore.kernel.org/bpf/20221104075113.5ighwdvero4mugu7@apolloIt was
MEM_TYPE_LOCALbefore. Also, better naming suggestions are always welcome, I asked the same in that message as well.
Sorry, I haven't followed <v5. Don't have perfect name, but logically this is BPF program memory. So "prog_kptr" would be something to convert this, but I'm not super happy with such a name. "user_kptr" would be too confusing, drawing incorrect "kernel space vs user space" comparison, while both are kernel memory. It's BPF-side kptr, so "bpf_kptr", but also not great.
So that's why didn't suggest anything specific, but at least as far as MEM_xxx flag goes, MEM_LOCAL_KPTR is better than MEM_ALLOC, IMO. It's at least consistent with the official name of the concept it represents.
Quoted 46 lines
Quoted 30 lines
Quoted 8 lines
include/linux/bpf.h +0
-0__BPF_TYPE_FLAG_MAX, __BPF_TYPE_LAST_FLAG = __BPF_TYPE_FLAG_MAX - 1, };@@ -771,6 +776,7 @@ struct bpf_prog_ops { union bpf_attr __user *uattr); };[...]
Quoted 19 lines
include/linux/bpf.h +7
-4-int btf_struct_access (struct bpf_verifier_log *log,const struct btf *btf, -const struct btf_type *t,int off,int size, -enum bpf_access_type atype__maybe_unused , +int btf_struct_access (struct bpf_verifier_log *log, +const struct bpf_reg_state *reg, +int off,int size,enum bpf_access_type atype__maybe_unused ,u32 *next_btf_id,enum bpf_type_flag *flag) { +const struct btf *btf = reg->btf;enum bpf_type_flag tmp_flag =0 ; +const struct btf_type *t; +u32 id = reg->btf_id;int err; -u32 id; + t =btf_type_by_id (btf, id); do { err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag);Quoted 15 lines
include/linux/bpf.h
Quoted 7 lines
switch (err) { case WALK_PTR:include/linux/bpf.h +5
-0+/* For local types, the destination register cannot +* become a pointer again. +*/ +if (type_is_local_kptr (reg->type)) +return SCALAR_VALUE ;passing the entire
bpf_reg_statejust to differentiate between local vs kernel pointer seems like a huge overkill.bpf_reg_stateis quite a complicated and extensive amount of state, and it seems cleaner to just pass it as a flag whether to allow pointer chasing or not. At least then we know we only care about that specific aspect, not about dozens of other possible fields ofbpf_reg_state.
Quoted 3 lines
I agree that the separation is usually better, especially because this is also a callback. I don't feel too strong about this though, we certainly do pass the whole reg to functions which only work on a specific type of pointer. Though the
Yeah, and then it takes a lot of grepping and jumping around the code to verify that only one simple field out of the entire bpf_reg_state is actually used. I'd say this is actually bad that we do pass it around so willy-nilly. Verifier code is already a hot complex mess, let's not actively make it harder than necessary.
Quoted 10 lines
concern in this case is justified as it's not only an internal function but also a callback.
It was just a bool in the RFC.
But in https://lore.kernel.org/bpf/20220907233023.x3uclwlnjuhftvtb@macbook-pro-4.dhcp.thefacebook.com
Alexei suggested passing reg instead.
From the link:Quoted 3 lines
imo it's cleaner to pass 'reg' instead of '
reg->btf', so we don't have to pass another boolean.
And checktype_is_local(reg)insidebtf_struct_access().
I sympathize with "too many input args" (especially if it's a bunch of bools) argument, but see above, I find it increasingly harder to know what parts of complex internal register state is used by helper functions and which are not.
And the fact that we have to construct a fake register state in some case is a red flag to me. Pass enum bpf_reg_type type to avoid passing true/false. Or let's invent a new enum. Or extend enum bpf_access_type to have READ_NOPTR/WRITE_NOPTR or something like that. Don't know.
This isn't a major issue, I can live with this just fine, but this definitely doesn't feel like a clean approach.
Re: [PATCH bpf-next v5 06/25] bpf: Introduce local kptrs
Quoted 74 lines · Andrii Nakryiko
Quoted 66 lines · Kumar Kartikeya Dwivedi
Quoted 51 lines · Andrii Nakryiko
Quoted 50 lines · Kumar Kartikeya Dwivedi
Introduce local kptrs, i.e.
PTR_TO_BTF_IDthat point to a type in program BTF. This is indicated by the presence ofMEM_ALLOCtype flag inreg->typeto avoid having to checkbtf_is_kernelwhen trying to match argument types in helpers.Refactor
btf_struct_accesscallback to just takebpf_reg_stateinstead of btf andbtf_typeparamters. Note that the call site incheck_map_accessnow simulates access to aPTR_TO_BTF_IDby creating a dummy reg on stack. Since only the type, btf, andbtf_idof the register matter for the checks, it can be done so without complicating the usual cases elsewhere in the verifier wherereg->btfandreg->btf_idis used verbatim.Whenever walking such types, any pointers being walked will always yield a SCALAR instead of pointer. In the future we might permit kptr inside local kptr (either kernel or local), and it would be permitted only in that case.
For now, these local kptrs will always be referenced in verifier context, hence
ref_obj_id== 0 for them is a bug. It is allowed to write to such objects, as long fields that are special are not touched (support for which will be added in subsequent patches). Note that once such a local kptr is markedPTR_UNTRUSTED, it is no longer allowed to write to it.No
PROBE_MEMhandling is therefore done for loads into this type unlessPTR_UNTRUSTEDis part of the register type, since they can never be in an undefined state, and their lifetime will always be valid.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 28 ++++++++++++++++-------- include/linux/filter.h | 8 +++---- kernel/bpf/btf.c | 16 ++++++++++---- kernel/bpf/verifier.c | 37 ++++++++++++++++++++++++++------ net/bpf/bpf_dummy_struct_ops.c | 14 ++++++------ net/core/filter.c | 34 ++++++++++++----------------- net/ipv4/bpf_tcp_ca.c | 13 ++++++----- net/netfilter/nf_conntrack_bpf.c | 17 ++++++--------- 8 files changed, 99 insertions(+), 68 deletions(-)include/linux/bpf.h +0
-0@@ -524,6 +524,11 @@ enum bpf_type_flag {/* Size is known at compile time. */ MEM_FIXED_SIZE = BIT(10 + BPF_BASE_TYPE_BITS),Quoted 12 lines
include/linux/bpf.h enum bpf_type_flag { · line 531
Quoted 5 lines
include/linux/bpf.h +5
-0+/* MEM is of a type from program BTF, not kernel BTF. This is used to +* tag PTR_TO_BTF_ID allocated using bpf_obj_new. +*/ + MEM_ALLOC = BIT(11 + BPF_BASE_TYPE_BITS), +you fixed one naming confusion with RINGBUF and basically are creating a new one, where "ALLOC" means "local kptr"... If we are stuck with "local kptr" (which I find very confusing as well, but that's beside the point), why not stick to the whole "local" terminology here?
MEM_LOCAL?Quoted 5 lines
See the discussion about this in v4:
https://lore.kernel.org/bpf/20221104075113.5ighwdvero4mugu7@apolloIt was
MEM_TYPE_LOCALbefore. Also, better naming suggestions are always welcome, I asked the same in that message as well.
Quoted 6 lines
Sorry, I haven't followed <v5. Don't have perfect name, but logically this is BPF program memory. So "
prog_kptr" would be something to convert this, but I'm not super happy with such a name. "user_kptr" would be too confusing, drawing incorrect "kernel space vs user space" comparison, while both are kernel memory. It's BPF-side kptr, so "bpf_kptr", but also not great.
yep. I went through the same thinking process.
Quoted 4 lines
So that's why didn't suggest anything specific, but at least as far as
MEM_xxxflag goes,MEM_LOCAL_KPTRis better thanMEM_ALLOC, IMO. It's at least consistent with the official name of the concept it represents.
"local kptr" doesn't fit here.
In libbpf, "local" is equally badly named.
If "local" was a good name we wouldn't have had this discussion. So we need to fix it libbpf, but we should start with a proper name in the kernel.
And "local kptr" is not it.
We must avoid exhausting bikeshedding too. MEM_ALLOC is something we can use right now and
as long as "local kptr" doesn't appear in docs, comments and commit logs we're good.
We can rename MEM_ALLOC to something else later.
In commit logs we can just say that this is
a pointer to an object allocated by the bpf program. It's crystal clear definition whereas "local kptr" is nonsensical.
Going back to the kptr definition.
kptr was supposed to mean a pointer to a kernel object.
In that light "pointer to an object allocated by the bpf prog" is something else.
Maybe "bptr" ?
In some ways bpf is a layer different from kernel space and user space. Some people joked that there is ring-0 for kernel, ring-3 for user space while bpf runs in ring-B.
Two new btf_tags __bptr and __bptr_ref (or may be just one?) might be necessary as well to make it easier to distinguish kernel and bpf prog allocated objects.
Quoted 14 lines
Quoted 8 lines
It was just a bool in the RFC.
But in https://lore.kernel.org/bpf/20220907233023.x3uclwlnjuhftvtb@macbook-pro-4.dhcp.thefacebook.com
Alexei suggested passing reg instead.
From the link:Quoted 3 lines
imo it's cleaner to pass 'reg' instead of '
reg->btf', so we don't have to pass another boolean.
And checktype_is_local(reg)insidebtf_struct_access().I sympathize with "too many input args" (especially if it's a bunch of bools) argument, but see above, I find it increasingly harder to know what parts of complex internal register state is used by helper functions and which are not.
Quoted 7 lines
And the fact that we have to construct a fake register state in some case is a red flag to me. Pass enum
bpf_reg_typetype to avoid passing true/false. Or let's invent a new enum. Or extend enumbpf_access_typeto have READ_NOPTR/WRITE_NOPTR or something like that. Don't know.This isn't a major issue, I can live with this just fine, but this definitely doesn't feel like a clean approach.
I think passing bpf_reg_state is a lesser evil _right_now_ and I prefer we proceed this way, since other works are blocked on this patch set.
We did plenty of refactoring of the verifier in the past. There will be more in the future.
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v5 06/25] bpf: Introduce local kptrs
Quoted 103 lines · Alexei Starovoitov
Quoted 74 lines · Andrii Nakryiko
Quoted 66 lines · Kumar Kartikeya Dwivedi
Quoted 51 lines · Andrii Nakryiko
Quoted 50 lines · Kumar Kartikeya Dwivedi
Introduce local kptrs, i.e.
PTR_TO_BTF_IDthat point to a type in program BTF. This is indicated by the presence ofMEM_ALLOCtype flag inreg->typeto avoid having to checkbtf_is_kernelwhen trying to match argument types in helpers.Refactor
btf_struct_accesscallback to just takebpf_reg_stateinstead of btf andbtf_typeparamters. Note that the call site incheck_map_accessnow simulates access to aPTR_TO_BTF_IDby creating a dummy reg on stack. Since only the type, btf, andbtf_idof the register matter for the checks, it can be done so without complicating the usual cases elsewhere in the verifier wherereg->btfandreg->btf_idis used verbatim.Whenever walking such types, any pointers being walked will always yield a SCALAR instead of pointer. In the future we might permit kptr inside local kptr (either kernel or local), and it would be permitted only in that case.
For now, these local kptrs will always be referenced in verifier context, hence
ref_obj_id== 0 for them is a bug. It is allowed to write to such objects, as long fields that are special are not touched (support for which will be added in subsequent patches). Note that once such a local kptr is markedPTR_UNTRUSTED, it is no longer allowed to write to it.No
PROBE_MEMhandling is therefore done for loads into this type unlessPTR_UNTRUSTEDis part of the register type, since they can never be in an undefined state, and their lifetime will always be valid.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 28 ++++++++++++++++-------- include/linux/filter.h | 8 +++---- kernel/bpf/btf.c | 16 ++++++++++---- kernel/bpf/verifier.c | 37 ++++++++++++++++++++++++++------ net/bpf/bpf_dummy_struct_ops.c | 14 ++++++------ net/core/filter.c | 34 ++++++++++++----------------- net/ipv4/bpf_tcp_ca.c | 13 ++++++----- net/netfilter/nf_conntrack_bpf.c | 17 ++++++--------- 8 files changed, 99 insertions(+), 68 deletions(-)include/linux/bpf.h +0
-0@@ -524,6 +524,11 @@ enum bpf_type_flag {/* Size is known at compile time. */ MEM_FIXED_SIZE = BIT(10 + BPF_BASE_TYPE_BITS),Quoted 12 lines
include/linux/bpf.h enum bpf_type_flag { · line 531
Quoted 5 lines
include/linux/bpf.h +5
-0+/* MEM is of a type from program BTF, not kernel BTF. This is used to +* tag PTR_TO_BTF_ID allocated using bpf_obj_new. +*/ + MEM_ALLOC = BIT(11 + BPF_BASE_TYPE_BITS), +you fixed one naming confusion with RINGBUF and basically are creating a new one, where "ALLOC" means "local kptr"... If we are stuck with "local kptr" (which I find very confusing as well, but that's beside the point), why not stick to the whole "local" terminology here?
MEM_LOCAL?Quoted 5 lines
See the discussion about this in v4:
https://lore.kernel.org/bpf/20221104075113.5ighwdvero4mugu7@apolloIt was
MEM_TYPE_LOCALbefore. Also, better naming suggestions are always welcome, I asked the same in that message as well.Quoted 6 lines
Sorry, I haven't followed <v5. Don't have perfect name, but logically this is BPF program memory. So "
prog_kptr" would be something to convert this, but I'm not super happy with such a name. "user_kptr" would be too confusing, drawing incorrect "kernel space vs user space" comparison, while both are kernel memory. It's BPF-side kptr, so "bpf_kptr", but also not great.yep. I went through the same thinking process.
Quoted 4 lines
So that's why didn't suggest anything specific, but at least as far as
MEM_xxxflag goes,MEM_LOCAL_KPTRis better thanMEM_ALLOC, IMO. It's at least consistent with the official name of the concept it represents."local kptr" doesn't fit here.
In libbpf, "local" is equally badly named.
If "local" was a good name we wouldn't have had this discussion. So we need to fix it libbpf, but we should start with a proper name in the kernel.
And "local kptr" is not it.We must avoid exhausting bikeshedding too.
MEM_ALLOCis something we can use right now and
as long as "local kptr" doesn't appear in docs, comments and commit logs we're good.
We can renameMEM_ALLOCto something else later.
Quoted 4 lines
In commit logs we can just say that this is
a pointer to an object allocated by the bpf program. It's crystal clear definition whereas "local kptr" is nonsensical.
Ok, I'll drop the naming everywhere.
Quoted 12 lines
Going back to the kptr definition.
kptr was supposed to mean a pointer to a kernel object.
In that light "pointer to an object allocated by the bpf prog" is something else.
Maybe "bptr" ?
In some ways bpf is a layer different from kernel space and user space. Some people joked that there is ring-0 for kernel, ring-3 for user space while bpf runs in ring-B.
Two newbtf_tags__bptrand__bptr_ref(or may be just one?) might be necessary as well to make it easier to distinguish kernel and bpf prog allocated objects.
There's also the option of simply using __kptr and __kptr_ref for these (without __local tag in BPF maps) and doing two stage name lookup for the types. Kernel BTF takes precedence, if not found there, then it searches program BTF for a local type. It would probably the simplest for users.
struct map_value {
struct nf_conn __kptr_ref *ct; // kernel struct foo __kptr_ref *f; // local struct task_struct __kptr_ref *t; // kernel struct bar __kptr_ref *b; // local
}
We can revisit this again once the post the follow up to store them in maps.
Re: [PATCH bpf-next v5 06/25] bpf: Introduce local kptrs
Quoted 125 lines · Kumar Kartikeya Dwivedi
Quoted 103 lines · Alexei Starovoitov
Quoted 74 lines · Andrii Nakryiko
Quoted 66 lines · Kumar Kartikeya Dwivedi
Quoted 51 lines · Andrii Nakryiko
Quoted 50 lines · Kumar Kartikeya Dwivedi
Introduce local kptrs, i.e.
PTR_TO_BTF_IDthat point to a type in program BTF. This is indicated by the presence ofMEM_ALLOCtype flag inreg->typeto avoid having to checkbtf_is_kernelwhen trying to match argument types in helpers.Refactor
btf_struct_accesscallback to just takebpf_reg_stateinstead of btf andbtf_typeparamters. Note that the call site incheck_map_accessnow simulates access to aPTR_TO_BTF_IDby creating a dummy reg on stack. Since only the type, btf, andbtf_idof the register matter for the checks, it can be done so without complicating the usual cases elsewhere in the verifier wherereg->btfandreg->btf_idis used verbatim.Whenever walking such types, any pointers being walked will always yield a SCALAR instead of pointer. In the future we might permit kptr inside local kptr (either kernel or local), and it would be permitted only in that case.
For now, these local kptrs will always be referenced in verifier context, hence
ref_obj_id== 0 for them is a bug. It is allowed to write to such objects, as long fields that are special are not touched (support for which will be added in subsequent patches). Note that once such a local kptr is markedPTR_UNTRUSTED, it is no longer allowed to write to it.No
PROBE_MEMhandling is therefore done for loads into this type unlessPTR_UNTRUSTEDis part of the register type, since they can never be in an undefined state, and their lifetime will always be valid.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 28 ++++++++++++++++-------- include/linux/filter.h | 8 +++---- kernel/bpf/btf.c | 16 ++++++++++---- kernel/bpf/verifier.c | 37 ++++++++++++++++++++++++++------ net/bpf/bpf_dummy_struct_ops.c | 14 ++++++------ net/core/filter.c | 34 ++++++++++++----------------- net/ipv4/bpf_tcp_ca.c | 13 ++++++----- net/netfilter/nf_conntrack_bpf.c | 17 ++++++--------- 8 files changed, 99 insertions(+), 68 deletions(-)include/linux/bpf.h +0
-0@@ -524,6 +524,11 @@ enum bpf_type_flag {/* Size is known at compile time. */ MEM_FIXED_SIZE = BIT(10 + BPF_BASE_TYPE_BITS),Quoted 12 lines
include/linux/bpf.h enum bpf_type_flag { · line 531
Quoted 5 lines
include/linux/bpf.h +5
-0+/* MEM is of a type from program BTF, not kernel BTF. This is used to +* tag PTR_TO_BTF_ID allocated using bpf_obj_new. +*/ + MEM_ALLOC = BIT(11 + BPF_BASE_TYPE_BITS), +you fixed one naming confusion with RINGBUF and basically are creating a new one, where "ALLOC" means "local kptr"... If we are stuck with "local kptr" (which I find very confusing as well, but that's beside the point), why not stick to the whole "local" terminology here?
MEM_LOCAL?Quoted 5 lines
See the discussion about this in v4:
https://lore.kernel.org/bpf/20221104075113.5ighwdvero4mugu7@apolloIt was
MEM_TYPE_LOCALbefore. Also, better naming suggestions are always welcome, I asked the same in that message as well.Quoted 6 lines
Sorry, I haven't followed <v5. Don't have perfect name, but logically this is BPF program memory. So "
prog_kptr" would be something to convert this, but I'm not super happy with such a name. "user_kptr" would be too confusing, drawing incorrect "kernel space vs user space" comparison, while both are kernel memory. It's BPF-side kptr, so "bpf_kptr", but also not great.yep. I went through the same thinking process.
Quoted 4 lines
So that's why didn't suggest anything specific, but at least as far as
MEM_xxxflag goes,MEM_LOCAL_KPTRis better thanMEM_ALLOC, IMO. It's at least consistent with the official name of the concept it represents."local kptr" doesn't fit here.
In libbpf, "local" is equally badly named.
If "local" was a good name we wouldn't have had this discussion. So we need to fix it libbpf, but we should start with a proper name in the kernel.
And "local kptr" is not it.We must avoid exhausting bikeshedding too.
MEM_ALLOCis something we can use right now and
as long as "local kptr" doesn't appear in docs, comments and commit logs we're good.
We can renameMEM_ALLOCto something else later.Quoted 4 lines
In commit logs we can just say that this is
a pointer to an object allocated by the bpf program. It's crystal clear definition whereas "local kptr" is nonsensical.Ok, I'll drop the naming everywhere.
Quoted 12 lines
Going back to the kptr definition.
kptr was supposed to mean a pointer to a kernel object.
In that light "pointer to an object allocated by the bpf prog" is something else.
Maybe "bptr" ?
In some ways bpf is a layer different from kernel space and user space. Some people joked that there is ring-0 for kernel, ring-3 for user space while bpf runs in ring-B.
Two newbtf_tags__bptrand__bptr_ref(or may be just one?) might be necessary as well to make it easier to distinguish kernel and bpf prog allocated objects.
Quoted 4 lines
There's also the option of simply using
__kptrand__kptr_reffor these (without__localtag in BPF maps) and doing two stage name lookup for the types. Kernel BTF takes precedence, if not found there, then it searches program BTF for a local type. It would probably the simplest for users.
Disagree about the simplest for users. It's going to be quite confusing for anyone reading the code and trying to understand what's going on. Explicit tag seems better to me. But it's subjective.
Quoted 9 lines
struct
map_value{
structnf_conn__kptr_ref*ct; // kernel struct foo__kptr_ref*f; // local structtask_struct__kptr_ref*t; // kernel struct bar__kptr_ref*b; // local
}We can revisit this again once the post the follow up to store them in maps.
Re: [PATCH bpf-next v5 06/25] bpf: Introduce local kptrs
In reply to Alexei StarovoitovQuoted 98 lines · Alexei Starovoitov
Quoted 74 lines · Andrii Nakryiko
Quoted 66 lines · Kumar Kartikeya Dwivedi
Quoted 51 lines · Andrii Nakryiko
Quoted 50 lines · Kumar Kartikeya Dwivedi
Introduce local kptrs, i.e.
PTR_TO_BTF_IDthat point to a type in program BTF. This is indicated by the presence ofMEM_ALLOCtype flag inreg->typeto avoid having to checkbtf_is_kernelwhen trying to match argument types in helpers.Refactor
btf_struct_accesscallback to just takebpf_reg_stateinstead of btf andbtf_typeparamters. Note that the call site incheck_map_accessnow simulates access to aPTR_TO_BTF_IDby creating a dummy reg on stack. Since only the type, btf, andbtf_idof the register matter for the checks, it can be done so without complicating the usual cases elsewhere in the verifier wherereg->btfandreg->btf_idis used verbatim.Whenever walking such types, any pointers being walked will always yield a SCALAR instead of pointer. In the future we might permit kptr inside local kptr (either kernel or local), and it would be permitted only in that case.
For now, these local kptrs will always be referenced in verifier context, hence
ref_obj_id== 0 for them is a bug. It is allowed to write to such objects, as long fields that are special are not touched (support for which will be added in subsequent patches). Note that once such a local kptr is markedPTR_UNTRUSTED, it is no longer allowed to write to it.No
PROBE_MEMhandling is therefore done for loads into this type unlessPTR_UNTRUSTEDis part of the register type, since they can never be in an undefined state, and their lifetime will always be valid.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 28 ++++++++++++++++-------- include/linux/filter.h | 8 +++---- kernel/bpf/btf.c | 16 ++++++++++---- kernel/bpf/verifier.c | 37 ++++++++++++++++++++++++++------ net/bpf/bpf_dummy_struct_ops.c | 14 ++++++------ net/core/filter.c | 34 ++++++++++++----------------- net/ipv4/bpf_tcp_ca.c | 13 ++++++----- net/netfilter/nf_conntrack_bpf.c | 17 ++++++--------- 8 files changed, 99 insertions(+), 68 deletions(-)include/linux/bpf.h +0
-0@@ -524,6 +524,11 @@ enum bpf_type_flag {/* Size is known at compile time. */ MEM_FIXED_SIZE = BIT(10 + BPF_BASE_TYPE_BITS),Quoted 12 lines
include/linux/bpf.h enum bpf_type_flag { · line 531
Quoted 5 lines
include/linux/bpf.h +5
-0+/* MEM is of a type from program BTF, not kernel BTF. This is used to +* tag PTR_TO_BTF_ID allocated using bpf_obj_new. +*/ + MEM_ALLOC = BIT(11 + BPF_BASE_TYPE_BITS), +you fixed one naming confusion with RINGBUF and basically are creating a new one, where "ALLOC" means "local kptr"... If we are stuck with "local kptr" (which I find very confusing as well, but that's beside the point), why not stick to the whole "local" terminology here?
MEM_LOCAL?Quoted 5 lines
See the discussion about this in v4:
https://lore.kernel.org/bpf/20221104075113.5ighwdvero4mugu7@apolloIt was
MEM_TYPE_LOCALbefore. Also, better naming suggestions are always welcome, I asked the same in that message as well.Quoted 6 lines
Sorry, I haven't followed <v5. Don't have perfect name, but logically this is BPF program memory. So "
prog_kptr" would be something to convert this, but I'm not super happy with such a name. "user_kptr" would be too confusing, drawing incorrect "kernel space vs user space" comparison, while both are kernel memory. It's BPF-side kptr, so "bpf_kptr", but also not great.yep. I went through the same thinking process.
Quoted 4 lines
So that's why didn't suggest anything specific, but at least as far as
MEM_xxxflag goes,MEM_LOCAL_KPTRis better thanMEM_ALLOC, IMO. It's at least consistent with the official name of the concept it represents."local kptr" doesn't fit here.
In libbpf, "local" is equally badly named.
If "local" was a good name we wouldn't have had this discussion. So we need to fix it libbpf, but we should start with a proper name in the kernel.
And "local kptr" is not it.
Quoted 5 lines
We must avoid exhausting bikeshedding too.
MEM_ALLOCis something we can use right now and
as long as "local kptr" doesn't appear in docs, comments and commit logs we're good.
We can renameMEM_ALLOCto something else later.
agreed
Quoted 5 lines
In commit logs we can just say that this is
a pointer to an object allocated by the bpf program. It's crystal clear definition whereas "local kptr" is nonsensical.
Quoted 11 lines
Going back to the kptr definition.
kptr was supposed to mean a pointer to a kernel object.
In that light "pointer to an object allocated by the bpf prog" is something else.
Maybe "bptr" ?
In some ways bpf is a layer different from kernel space and user space. Some people joked that there is ring-0 for kernel, ring-3 for user space while bpf runs in ring-B.
Two newbtf_tags__bptrand__bptr_ref(or may be just one?) might be necessary as well to make it easier to distinguish kernel and bpf prog allocated objects.
bptr isn't terrific, but I don't have any good suggestions. bptr, bpfptr, tptr (for "typed pointer" as opposed to untyped dynptr), all kind of meh. I'm fine with whatever.
Quoted 20 lines
Quoted 20 lines
Quoted 9 lines
It was just a bool in the RFC.
But in https://lore.kernel.org/bpf/20220907233023.x3uclwlnjuhftvtb@macbook-pro-4.dhcp.thefacebook.com
Alexei suggested passing reg instead.
From the link:Quoted 3 lines
imo it's cleaner to pass 'reg' instead of '
reg->btf', so we don't have to pass another boolean.
And checktype_is_local(reg)insidebtf_struct_access().I sympathize with "too many input args" (especially if it's a bunch of bools) argument, but see above, I find it increasingly harder to know what parts of complex internal register state is used by helper functions and which are not.
And the fact that we have to construct a fake register state in some case is a red flag to me. Pass enum
bpf_reg_typetype to avoid passing true/false. Or let's invent a new enum. Or extend enumbpf_access_typeto have READ_NOPTR/WRITE_NOPTR or something like that. Don't know.
Quoted 8 lines
Quoted 2 lines
This isn't a major issue, I can live with this just fine, but this definitely doesn't feel like a clean approach.
I think passing
bpf_reg_stateis a lesser evil _right_now_ and I prefer we proceed this way, since other works are blocked on this patch set.
We did plenty of refactoring of the verifier in the past. There will be more in the future.
sure, not crucial
Kumar Kartikeya Dwivedi
maintainerPATCH v5 7/25 bpf: Recognize bpf_{spin_lock,list_head,list_node} in local kptrs
In reply to Kumar Kartikeya DwivediAllow specifying bpf_spin_lock, bpf_list_head, bpf_list_node fields in a local kptr.
Update btf_struct_access callback to reject direct access to these special fields in local kptrs.
A bpf_list_head allows implementing map-in-map style use cases, where local kptr with bpf_list_head is linked into a list in a map value. This would require embedding a bpf_list_node, support for which is also included.
Lastly, while we strictly don't require to hold a bpf_spin_lock while manipulating the bpf_list_head of a local kptr, as when have access to it, we have complete ownership of the object, the locking constraint is still kept and may be conditionally lifted in the future.
Note that the specification of such types can be done just like map values, e.g.:
struct bar {
struct bpf_list_node node;
};struct foo {
struct bpf_spin_lock lock;
struct bpf_list_head head __contains(bar, node);
struct bpf_list_node node;
};struct map_value {
struct bpf_spin_lock lock;
struct bpf_list_head head __contains(foo, node);
};To recognize such types in user BTF, we build a btf_struct_metas array of metadata items corresponding to each BTF ID. This is done once during the btf_parse stage to avoid having to do it each time during the verification process's requirement to inspect the metadata.
Moreover, the computed metadata needs to be passed to some helpers in future patches which requires allocating them and storing them in the BTF that is pinned by the program itself, so that valid access can be assumed to such data during program runtime.
A key thing to note is that once a btf_struct_meta is available for a type, both the btf_record and btf_field_offs should be available. It is critical that btf_field_offs is available in case special fields are present, as we extensively rely on special fields being zeroed out in map values and local kptrs in later patches. The code ensures that by bailing out in case of errors and ensuring both are available together. If the record is not available, the special fields won't be recognized, so not having both is also fine (in terms of being a verification error and not a runtime bug).
--- include/linux/bpf.h | 7 ++ include/linux/btf.h | 35 ++++++++ kernel/bpf/btf.c | 196 +++++++++++++++++++++++++++++++++++++++---- kernel/bpf/syscall.c | 4 + 4 files changed, 224 insertions(+), 18 deletions(-)
include/linux/bpf.h +7 -0
@@ -176,6 +176,7 @@ enum btf_field_type {BPF_KPTR_REF = (1 <<3 ),BPF_KPTR =BPF_KPTR_UNREF |BPF_KPTR_REF ,BPF_LIST_HEAD = (1 <<4 ), + BPF_LIST_NODE = (1 << 5), }; struct btf_field_kptr {
@@ -276,6 +277,8 @@ static inline const char *btf_field_type_name(enum btf_field_type type)return "kptr" ;case BPF_LIST_HEAD :return "bpf_list_head" ; +case BPF_LIST_NODE : +return "bpf_list_node" ;default :WARN_ON_ONCE (1 );return "unknown" ;
@@ -294,6 +297,8 @@ static inline u32 btf_field_type_size(enum btf_field_type type)return sizeof (u64);case BPF_LIST_HEAD :return sizeof (struct bpf_list_head ); +case BPF_LIST_NODE : +return sizeof (struct bpf_list_node );default :WARN_ON_ONCE (1 );return 0 ;
@@ -312,6 +317,8 @@ static inline u32 btf_field_type_align(enum btf_field_type type)return __alignof__(u64 );case BPF_LIST_HEAD :return __alignof__(struct bpf_list_head ); +case BPF_LIST_NODE : +return __alignof__(struct bpf_list_node );default :WARN_ON_ONCE (1 );return 0 ;
include/linux/btf.h +35 -0
@@ -6,6 +6,8 @@#include <linux/types.h> #include <linux/bpfptr.h> +#include <linux/bsearch.h> +#include <linux/btf_ids.h> #include <uapi/linux/btf.h> #include <uapi/linux/bpf.h>
@@ -78,6 +80,17 @@ struct btf_id_dtor_kfunc {u32 kfunc_btf_id; }; +struct btf_struct_meta { +u32 btf_id; +struct btf_record *record; +struct btf_field_offs *field_offs; +}; + +struct btf_struct_metas { +u32 cnt; +struct btf_struct_meta types[]; +}; +typedef void (*btf_dtor_kfunc_t )(void *);extern const struct file_operations btf_fops;
@@ -408,6 +421,23 @@ static inline struct btf_param *btf_params(const struct btf_type *t)return (struct btf_param *)(t +1 ); } +static inline int btf_id_cmp_func (const void *a,const void *b) +{ +const int *pa = a, *pb = b; + +return *pa - *pb; +} + +static inline bool btf_id_set_contains (const struct btf_id_set *set,u32 id) +{ +return bsearch (&id, set->ids, set->cnt,sizeof (u32), btf_id_cmp_func) !=NULL ; +} + +static inline void *btf_id_set8_contains (const struct btf_id_set8 *set,u32 id) +{ +return bsearch (&id, set->pairs, set->cnt,sizeof (set->pairs[0 ]), btf_id_cmp_func); +} +#ifdef CONFIG_BPF_SYSCALL struct bpf_prog ;
@@ -423,6 +453,7 @@ int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,s32 btf_find_dtor_kfunc (struct btf *btf,u32 btf_id);int register_btf_id_dtor_kfuncs (const struct btf_id_dtor_kfunc *dtors,u32 add_cnt,struct module *owner); +struct btf_struct_meta *btf_find_struct_meta (const struct btf *btf,u32 btf_id);#else static inline const struct btf_type *btf_type_by_id (const struct btf *btf,u32 type_id)
@@ -454,6 +485,10 @@ static inline int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dt {return 0 ; } +static inline struct btf_struct_meta *btf_find_struct_meta (const struct btf *btf,u32 btf_id) +{ +return NULL ; +}#endif static inline bool btf_type_is_struct_ptr (struct btf *btf,const struct btf_type *t)
kernel/bpf/btf.c +178 -18
272 lines not on this page · raw message
kernel/bpf/syscall.c +4 -0
@@ -537,6 +537,7 @@ void btf_record_free(struct btf_record *rec)btf_put (rec->fields[i].kptr.btf);break ;case BPF_LIST_HEAD : +case BPF_LIST_NODE :/* Nothing to release for bpf_list_head */ break ;default :
@@ -582,6 +583,7 @@ struct btf_record *btf_record_dup(const struct btf_record *rec) }break ;case BPF_LIST_HEAD : +case BPF_LIST_NODE :/* Nothing to acquire for bpf_list_head */ break ;default :
@@ -648,6 +650,8 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj)continue ;bpf_list_head_free (field, field_ptr, obj + rec->spin_lock_off);break ; +case BPF_LIST_NODE : +break ;default :WARN_ON_ONCE (1 );continue ;
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 8/25 bpf: Verify ownership relationships for user BTF types
In reply to Kumar Kartikeya DwivediEnsure that there can be no ownership cycles among different types by way of having owning objects that can hold some other type as their element. For instance, a map value can only hold local kptrs, but these are allowed to have another bpf_list_head. To prevent unbounded recursion while freeing resources, elements of bpf_list_head in local kptrs can never have a bpf_list_head which are part of list in a map value. Later patches will verify this by having dedicated BTF selftests.
Also, to make runtime destruction easier, once btf_struct_metas is fully populated, we can stash the metadata of the value type directly in the metadata of the list_head fields, as that allows easier access to the value type's layout to destruct it at runtime from the btf_field entry of the list head itself.
--- include/linux/bpf.h | 1 + include/linux/btf.h | 1 + kernel/bpf/btf.c | 71 ++++++++++++++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 4 +++ 4 files changed, 77 insertions(+)
include/linux/bpf.h +1 -0
@@ -190,6 +190,7 @@ struct btf_field_list_head {struct btf *btf;u32 value_btf_id;u32 node_offset; +struct btf_record *value_rec; };struct btf_field {
include/linux/btf.h +1 -0
@@ -178,6 +178,7 @@ int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t);int btf_find_timer (const struct btf *btf,const struct btf_type *t);struct btf_record *btf_parse_fields (const struct btf *btf,const struct btf_type *t,u32 field_mask,u32 value_size); +int btf_check_and_fixup_fields (const struct btf *btf,struct btf_record *rec);struct btf_field_offs *btf_parse_field_offs (struct btf_record *rec);bool btf_type_is_void (const struct btf_type *t);s32 btf_find_by_name_kind (const struct btf *btf,const char *name,u8 kind);
kernel/bpf/btf.c +71 -0
83 lines not on this page · raw message
kernel/bpf/syscall.c +4 -0
@@ -1044,6 +1044,10 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, } } + ret =btf_check_and_fixup_fields (btf, map->record); +if (ret <0 ) + goto free_map_tab; +if (map->ops->map_check_btf) { ret = map->ops->map_check_btf (map, btf, key_type, value_type);if (ret <0 )
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 9/25 bpf: Allow locking bpf_spin_lock in local kptr
In reply to Kumar Kartikeya DwivediAllow locking a bpf_spin_lock embedded in local kptr, in addition to already support map value pointers. The handling is similar to that of map values, by just preserving the reg->id of local kptrs as well, and adjusting process_spin_lock to work with non-PTR_TO_MAP_VALUE and remember the id in verifier state.
Refactor the existing process_spin_lock to work with PTR_TO_BTF_ID | MEM_ALLOC in addition to PTR_TO_MAP_VALUE. We need to update the reg_may_point_to_spin_lock which is used in mark_ptr_or_null_reg to preserve reg->id, that will be used in env->cur_state->active_spin_lock to remember the currently held spin lock.
--- kernel/bpf/helpers.c | 2 ++ kernel/bpf/verifier.c | 70 ++++++++++++++++++++++++++++++++----------- 2 files changed, 55 insertions(+), 17 deletions(-)
kernel/bpf/helpers.c +2 -0
@@ -336,6 +336,7 @@ const struct bpf_func_proto bpf_spin_lock_proto = { .gpl_only = false, .ret_type = RET_VOID, .arg1_type = ARG_PTR_TO_SPIN_LOCK, + .arg1_btf_id = BPF_PTR_POISON, };static inline void __bpf_spin_unlock_irqrestore (struct bpf_spin_lock *lock)
@@ -358,6 +359,7 @@ const struct bpf_func_proto bpf_spin_unlock_proto = { .gpl_only = false, .ret_type = RET_VOID, .arg1_type = ARG_PTR_TO_SPIN_LOCK, + .arg1_btf_id = BPF_PTR_POISON, }; void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
kernel/bpf/verifier.c +53 -17
123 lines not on this page · raw message
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 10/25 bpf: Allow locking bpf_spin_lock global variables
In reply to Kumar Kartikeya DwivediGlobal variables reside in maps accessible using direct_value_addr callbacks, so giving each load instruction's rewrite a unique reg->id disallows us from holding locks which are global.
The reason for preserving reg->id as a unique value for registers that may point to spin lock is that two separate lookups are treated as two separate memory regions, and any possible aliasing is ignored for the purposes of spin lock correctness.
This is not great especially for the global variable case, which are served from maps that have max_entries == 1, i.e. they always lead to map values pointing into the same map value.
So refactor the active_spin_lock into a 'active_lock' structure which represents the lock identity, and instead of the reg->id, remember two fields, a pointer and the reg->id. The pointer will store reg->map_ptr or reg->btf. It's only necessary to distinguish for the id == 0 case of global variables, but always setting the pointer to a non-NULL value and using the pointer to check whether the lock is held simplifies code in the verifier.
This is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.
Note that while whether a lock is held can be answered by just comparing active_lock.ptr to NULL, to determine whether the register is pointing to the same held lock requires comparing _both_ ptr and id.
Finally, as a result of this refactoring, pseudo load instructions are not given a unique reg->id, as they are doing lookup for the same map value (max_entries is never greater than 1).
Essentially, we consider that the tuple of (ptr, id) will always be unique for any kind of argument to bpf_spin_{lock,unlock}.
Note that this can be extended in the future to also remember offset used for locking, so that we can introduce multiple bpf_spin_lock fields in the same allocation.
--- include/linux/bpf_verifier.h | 5 ++++- kernel/bpf/verifier.c | 41 ++++++++++++++++++++++++------------ 2 files changed, 32 insertions(+), 14 deletions(-)
include/linux/bpf_verifier.h +4 -1
@@ -323,7 +323,10 @@ struct bpf_verifier_state {u32 branches;u32 insn_idx;u32 curframe; -u32 active_spin_lock; +struct { +void *ptr; +u32 id; + } active_lock;bool speculative;/* first and last insn idx of this verifier state */
kernel/bpf/verifier.c +28 -13
100 lines not on this page · raw message
Signature · 1 line
2.38.1
Re: [PATCH bpf-next v5 10/25] bpf: Allow locking bpf_spin_lock global variables
Quoted 51 lines · Kumar Kartikeya Dwivedi
Global variables reside in maps accessible using
direct_value_addrcallbacks, so giving each load instruction's rewrite a uniquereg->iddisallows us from holding locks which are global.The reason for preserving
reg->idas a unique value for registers that may point to spin lock is that two separate lookups are treated as two separate memory regions, and any possible aliasing is ignored for the purposes of spin lock correctness.This is not great especially for the global variable case, which are served from maps that have
max_entries== 1, i.e. they always lead to map values pointing into the same map value.So refactor the
active_spin_lockinto a 'active_lock' structure which represents the lock identity, and instead of thereg->id, remember two fields, a pointer and thereg->id. The pointer will storereg->map_ptrorreg->btf. It's only necessary to distinguish for the id == 0 case of global variables, but always setting the pointer to a non-NULL value and using the pointer to check whether the lock is held simplifies code in the verifier.This is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.
Note that while whether a lock is held can be answered by just comparing active_lock.ptr to NULL, to determine whether the register is pointing to the same held lock requires comparing _both_ ptr and id.
Finally, as a result of this refactoring, pseudo load instructions are not given a unique
reg->id, as they are doing lookup for the same map value (max_entriesis never greater than 1).Essentially, we consider that the tuple of (ptr, id) will always be unique for any kind of argument to bpf_spin_{lock,unlock}.
Note that this can be extended in the future to also remember offset used for locking, so that we can introduce multiple
bpf_spin_lockfields in the same allocation.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf_verifier.h | 5 ++++- kernel/bpf/verifier.c | 41 ++++++++++++++++++++++++------------ 2 files changed, 32 insertions(+), 14 deletions(-)include/linux/bpf_verifier.h +0
-0
include/linux/bpf_verifier.h struct bpf_verifier_state { · line 327
Quoted 6 lines
include/linux/bpf_verifier.h +2
-1@@ -323,7 +323,10 @@ struct bpf_verifier_state {u32 branches;u32 insn_idx;u32 curframe; -u32 active_spin_lock; +struct { +void *ptr;
document that this could be either struct bpf_map or struct btf pointer, at least?
Quoted 5 lines
include/linux/bpf_verifier.h +2
-0+u32 id; + } active_lock;bool speculative;/* first and last insn idx of this verifier state */
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v5 10/25] bpf: Allow locking bpf_spin_lock global variables
Quoted 47 lines · Andrii Nakryiko
Quoted 46 lines · Kumar Kartikeya Dwivedi
Global variables reside in maps accessible using
direct_value_addrcallbacks, so giving each load instruction's rewrite a uniquereg->iddisallows us from holding locks which are global.The reason for preserving
reg->idas a unique value for registers that may point to spin lock is that two separate lookups are treated as two separate memory regions, and any possible aliasing is ignored for the purposes of spin lock correctness.This is not great especially for the global variable case, which are served from maps that have
max_entries== 1, i.e. they always lead to map values pointing into the same map value.So refactor the
active_spin_lockinto a 'active_lock' structure which represents the lock identity, and instead of thereg->id, remember two fields, a pointer and thereg->id. The pointer will storereg->map_ptrorreg->btf. It's only necessary to distinguish for the id == 0 case of global variables, but always setting the pointer to a non-NULL value and using the pointer to check whether the lock is held simplifies code in the verifier.This is generic enough to allow it for global variables, map lookups, and local kptr registers at the same time.
Note that while whether a lock is held can be answered by just comparing active_lock.ptr to NULL, to determine whether the register is pointing to the same held lock requires comparing _both_ ptr and id.
Finally, as a result of this refactoring, pseudo load instructions are not given a unique
reg->id, as they are doing lookup for the same map value (max_entriesis never greater than 1).Essentially, we consider that the tuple of (ptr, id) will always be unique for any kind of argument to bpf_spin_{lock,unlock}.
Note that this can be extended in the future to also remember offset used for locking, so that we can introduce multiple
bpf_spin_lockfields in the same allocation.Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf_verifier.h | 5 ++++- kernel/bpf/verifier.c | 41 ++++++++++++++++++++++++------------ 2 files changed, 32 insertions(+), 14 deletions(-)
Quoted 15 lines
include/linux/bpf_verifier.h struct bpf_verifier_state { · line 327
Quoted 11 lines
include/linux/bpf_verifier.h +2
-1@@ -323,7 +323,10 @@ struct bpf_verifier_state {u32 branches;u32 insn_idx;u32 curframe; -u32 active_spin_lock; +struct { +void *ptr;document that this could be either struct
bpf_mapor struct btf pointer, at least?
Ack, I'll add a comment.
Though it's not really meant to be used (i.e. turned back into a pointer to them), it's just an 'identity' pointer.
Kumar Kartikeya Dwivedi
maintainerPATCH v5 11/25 bpf: Allow locking bpf_spin_lock in inner map values
In reply to Kumar Kartikeya DwivediThere is no need to restrict users from locking bpf_spin_lock in map values of inner maps. Each inner map lookup gets a unique reg->id assigned to the returned PTR_TO_MAP_VALUE which will be preserved after the NULL check. Distinct lookups into different inner map get unique IDs, and distinct lookups into same inner map also get unique IDs.
Hence, lift the restriction by removing the check return -ENOTSUPP in map_in_map.c. Later commits will add comprehensive test cases to ensure that invalid cases are rejected.
--- kernel/bpf/map_in_map.c | 5 ----- 1 file changed, 5 deletions(-)
kernel/bpf/map_in_map.c +0 -5
@@ -29,11 +29,6 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)return ERR_PTR (-ENOTSUPP ); } -if (btf_record_has_field (inner_map->record,BPF_SPIN_LOCK )) { -fdput (f); -return ERR_PTR (-ENOTSUPP ); - } - inner_map_meta_size =sizeof (*inner_map_meta);/* In some cases verifier needs to access beyond just base map. */ if (inner_map->ops == &array_map_ops)
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 12/25 bpf: Rewrite kfunc argument handling
In reply to Kumar Kartikeya DwivediAs we continue to add more features, argument types, kfunc flags, and different extensions to kfuncs, the code to verify the correctness of the kfunc prototype wrt the passed in registers has become ad-hoc and ugly to read. To make life easier, and make a very clear split between different stages of argument processing, move all the code into verifier.c and refactor into easier to read helpers and functions.
This also makes sharing code within the verifier easier with kfunc argument processing. This will be more and more useful in later patches as we are now moving to implement very core BPF helpers as kfuncs, to keep them experimental before baking into UAPI.
--- include/linux/btf.h | 31 +- kernel/bpf/btf.c | 16 +- kernel/bpf/verifier.c | 547 +++++++++++++++++- .../bpf/prog_tests/kfunc_dynptr_param.c | 2 +- tools/testing/selftests/bpf/verifier/calls.c | 2 +- .../selftests/bpf/verifier/ref_tracking.c | 4 +- 6 files changed, 569 insertions(+), 33 deletions(-)
include/linux/btf.h +30 -1
@@ -338,6 +338,16 @@ static inline bool btf_type_is_struct(const struct btf_type *t)return kind ==BTF_KIND_STRUCT || kind ==BTF_KIND_UNION ; } +static inline bool __btf_type_is_struct (const struct btf_type *t) +{ +return BTF_INFO_KIND (t->info) ==BTF_KIND_STRUCT ; +} + +static inline bool btf_type_is_array (const struct btf_type *t) +{ +return BTF_INFO_KIND (t->info) ==BTF_KIND_ARRAY ; +} +static inline u16 btf_type_vlen (const struct btf_type *t) {return BTF_INFO_VLEN (t->info);
@@ -439,9 +449,10 @@ static inline void *btf_id_set8_contains(const struct btf_id_set8 *set, u32 id)return bsearch (&id, set->pairs, set->cnt,sizeof (set->pairs[0 ]), btf_id_cmp_func); } -#ifdef CONFIG_BPF_SYSCALL struct bpf_prog ; +struct bpf_verifier_log ; +#ifdef CONFIG_BPF_SYSCALL const struct btf_type *btf_type_by_id (const struct btf *btf,u32 type_id);const char *btf_name_by_offset (const struct btf *btf,u32 offset);struct btf *btf_parse_vmlinux (void );
@@ -455,6 +466,12 @@ s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id);int register_btf_id_dtor_kfuncs (const struct btf_id_dtor_kfunc *dtors,u32 add_cnt,struct module *owner);struct btf_struct_meta *btf_find_struct_meta (const struct btf *btf,u32 btf_id); +const struct btf_member * +btf_get_prog_ctx_type (struct bpf_verifier_log *log,const struct btf *btf, +const struct btf_type *t,enum bpf_prog_type prog_type, +int arg); +bool btf_types_are_same (const struct btf *btf1,u32 id1, +const struct btf *btf2,u32 id2);#else static inline const struct btf_type *btf_type_by_id (const struct btf *btf,u32 type_id)
@@ -490,6 +507,18 @@ static inline struct btf_struct_meta *btf_find_struct_meta(const struct btf *btf {return NULL ; } +static inline const struct btf_member * +btf_get_prog_ctx_type (struct bpf_verifier_log *log,const struct btf *btf, +const struct btf_type *t,enum bpf_prog_type prog_type, +int arg) +{ +return NULL ; +} +static inline bool btf_types_are_same (const struct btf *btf1,u32 id1, +const struct btf *btf2,u32 id2) +{ +return false; +}#endif static inline bool btf_type_is_struct_ptr (struct btf *btf,const struct btf_type *t)
kernel/bpf/btf.c +3 -13
@@ -478,16 +478,6 @@ static bool btf_type_nosize_or_null(const struct btf_type *t)return !t ||btf_type_nosize (t); } -static bool __btf_type_is_struct (const struct btf_type *t) -{ -return BTF_INFO_KIND (t->info) ==BTF_KIND_STRUCT ; -} - -static bool btf_type_is_array (const struct btf_type *t) -{ -return BTF_INFO_KIND (t->info) ==BTF_KIND_ARRAY ; -} -static bool btf_type_is_datasec (const struct btf_type *t) {return BTF_INFO_KIND (t->info) ==BTF_KIND_DATASEC ;
@@ -5533,7 +5523,7 @@ static u8 bpf_ctx_convert_map[] = {#undef BPF_MAP_TYPE#undef BPF_LINK_TYPE -static const struct btf_member * +const struct btf_member *btf_get_prog_ctx_type (struct bpf_verifier_log *log,const struct btf *btf,const struct btf_type *t,enum bpf_prog_type prog_type,int arg)
@@ -6319,8 +6309,8 @@ int btf_struct_access(struct bpf_verifier_log *log,* end up with two different module BTFs, but IDs point to the common type in * vmlinux BTF. */ -static bool btf_types_are_same (const struct btf *btf1,u32 id1, -const struct btf *btf2,u32 id2) +bool btf_types_are_same (const struct btf *btf1,u32 id1, +const struct btf *btf2,u32 id2) {if (id1 != id2)return false;
kernel/bpf/verifier.c +532 -15
602 lines not on this page · raw message
tools/testing/selftests/bpf/prog_tests/kfunc_dynptr_param.c +1 -1
@@ -22,7 +22,7 @@ static struct { "arg#0 pointer type STRUCT bpf_dynptr_kern points to unsupported dynamic pointer type", 0}, {"not_valid_dynptr" , "arg#0 pointer type STRUCT bpf_dynptr_kern must be valid and initialized", 0}, - {"not_ptr_to_stack", "arg#0 pointer type STRUCT bpf_dynptr_kern not to stack", 0}, + {"not_ptr_to_stack", "arg#0 expected pointer to stack", 0}, {"dynptr_data_null", NULL, -EBADMSG}, };
tools/testing/selftests/bpf/verifier/calls.c +1 -1
@@ -109,7 +109,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "arg#0 pointer type STRUCT prog_test_ref_kfunc must point", + .errstr = "arg#0 expected pointer to btf or socket", .fixup_kfunc_btf_id = { { "bpf_kfunc_call_test_acquire", 3 }, { "bpf_kfunc_call_test_release", 5 },
tools/testing/selftests/bpf/verifier/ref_tracking.c +2 -2
@@ -142,7 +142,7 @@ .kfunc = "bpf", .expected_attach_type = BPF_LSM_MAC, .flags = BPF_F_SLEEPABLE, - .errstr = "arg#0 pointer type STRUCT bpf_key must point to scalar, or struct with scalar", + .errstr = "arg#0 expected pointer to btf or socket", .fixup_kfunc_btf_id = { { "bpf_lookup_user_key", 2 }, { "bpf_key_put", 4 },
@@ -163,7 +163,7 @@ .kfunc = "bpf", .expected_attach_type = BPF_LSM_MAC, .flags = BPF_F_SLEEPABLE, - .errstr = "arg#0 pointer type STRUCT bpf_key must point to scalar, or struct with scalar", + .errstr = "arg#0 expected pointer to btf or socket", .fixup_kfunc_btf_id = { { "bpf_lookup_system_key", 1 }, { "bpf_key_put", 3 },
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 13/25 bpf: Drop kfunc bits from btf_check_func_arg_match
In reply to Kumar Kartikeya DwivediRemove all kfunc related bits now from btf_check_func_arg_match, as users have been converted away to refactored kfunc argument handling.
This is split into a separate commit to aid review, in order to compare what has been preserved from the removed bits easily instead of mixing removed hunks with previous patch.
--- include/linux/bpf.h | 11 -- include/linux/bpf_verifier.h | 2 - kernel/bpf/btf.c | 364 +---------------------------------- kernel/bpf/verifier.c | 4 +- 4 files changed, 10 insertions(+), 371 deletions(-)
include/linux/bpf.h +0 -11
@@ -2108,22 +2108,11 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,const char *func_name, struct btf_func_model *m); -struct bpf_kfunc_arg_meta { -u64 r0_size; -bool r0_rdonly; -int ref_obj_id; -u32 flags; -}; -struct bpf_reg_state ;int btf_check_subprog_arg_match (struct bpf_verifier_env *env,int subprog,struct bpf_reg_state *regs);int btf_check_subprog_call (struct bpf_verifier_env *env,int subprog,struct bpf_reg_state *regs); -int btf_check_kfunc_arg_match (struct bpf_verifier_env *env, -const struct btf *btf,u32 func_id, -struct bpf_reg_state *regs, -struct bpf_kfunc_arg_meta *meta);int btf_prepare_func_args (struct bpf_verifier_env *env,int subprog,struct bpf_reg_state *reg); int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *prog,
include/linux/bpf_verifier.h +0 -2
@@ -592,8 +592,6 @@ int check_ptr_off_reg(struct bpf_verifier_env *env,int check_func_arg_reg_off (struct bpf_verifier_env *env,const struct bpf_reg_state *reg,int regno,enum bpf_arg_type arg_type); -int check_kfunc_mem_size_reg (struct bpf_verifier_env *env,struct bpf_reg_state *reg, -u32 regno);int check_mem_reg (struct bpf_verifier_env *env,struct bpf_reg_state *reg,u32 regno,u32 mem_size); bool is_dynptr_reg_valid_init(struct bpf_verifier_env *env,
kernel/bpf/btf.c +8 -356
447 lines not on this page · raw message
kernel/bpf/verifier.c +2 -2
@@ -5539,8 +5539,8 @@ int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,return err; } -int check_kfunc_mem_size_reg (struct bpf_verifier_env *env,struct bpf_reg_state *reg, -u32 regno) +static int check_kfunc_mem_size_reg (struct bpf_verifier_env *env,struct bpf_reg_state *reg, +u32 regno) {struct bpf_reg_state *mem_reg = &cur_regs (env)[regno -1 ];bool may_be_null =type_may_be_null (mem_reg->type);
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 14/25 bpf: Support constant scalar arguments for kfuncs
In reply to Kumar Kartikeya DwivediAllow passing known constant scalars as arguments to kfuncs that do not represent a size parameter. This makes the search pruning optimization of verifier more conservative for such kfunc calls, and each non-distinct argument is considered unequivalent.
We will use this support to then expose a global bpf_kptr_alloc function where it takes the local type ID in program BTF, and returns a PTR_TO_BTF_ID to the local type. These will be called local kptrs, and allows programs to allocate their own objects.
However, this is still not completely safe, as mark_chain_precision logic is buggy without more work when the constant argument is not a size, but still needs precise marker propagation for pruning checks. Next patch will fix this problem.
--- Documentation/bpf/kfuncs.rst | 30 ++++++++++++++++++ kernel/bpf/verifier.c | 59 +++++++++++++++++++++++++++--------- 2 files changed, 75 insertions(+), 14 deletions(-)
Documentation/bpf/kfuncs.rst +30 -0
@@ -72,6 +72,36 @@ argument as its size. By default, without __sz annotation, the size of the type of the pointer is used. Without __sz annotation, a kfunc cannot accept a void pointer. +2.2.1 __k Annotation +-------------------- + +This annotation is only understood for scalar arguments, where it indicates that +the verifier must check the scalar argument to be a known constant, which does +not indicate a size parameter. This distinction is important, as when the scalar +argument does not represent a size parameter, verifier is more conservative in +state search pruning and does not consider two arguments equivalent for safety +purposes if the already verified value was within range of the new one. + +This assumption holds well for sizes (as memory accessed within smaller bounds +in old verified state will also work for bigger bounds in current to be explored +state), but not for other constant arguments where each carries a distinct +semantic effect. + +An example is given below:: + + void *bpf_obj_new(u32 local_type_id__k, ...) + { + ... + } + +Here, bpf_obj_new uses local_type_id argument to find out the size of that +type ID in program's BTF and return a sized pointer to it. Each type ID will +have a distinct size, hence it is crucial to treat each such call as distinct +when values don't match. + +Hence, whenever a constant scalar argument is accepted by a kfunc which is not a +size parameter, __k suffix must be used. + .. _BPF_kfunc_nodef: 2.3 Using an existing kernel function
kernel/bpf/verifier.c +45 -14
88 lines not on this page · raw message
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 15/25 bpf: Teach verifier about non-size constant arguments
In reply to Kumar Kartikeya DwivediCurrently, the verifier has support for various arguments that either describe the size of the memory being passed in to a helper, or describe the size of the memory being returned. When a constant is passed in like this, it is assumed for the purposes of precision tracking that if the value in the already explored safe state is within the value in current state, it would fine to prune the search.
While this holds well for size arguments, arguments where each value may denote a distinct meaning and needs to be verified separately needs more work. Search can only be pruned if both are equivalent values. In all other cases, it would be incorrect to treat those two precise registers as equivalent if the new value satisfies the old one (i.e. old <= cur).
Hence, make the register precision marker tri-state. There are now three values that reg->precise takes: NOT_PRECISE, PRECISE, EXACT.
Both PRECISE and EXACT are 'true' values. EXACT affects how regsafe decides whether both registers are equivalent for the purposes of verifier state equivalence. When it sees that old state register has reg->precise == EXACT, unless both are equivalent, it will return false. Otherwise, for PRECISE case it falls back to the default check that is present now (i.e. thinking that we're talking about sizes).
This is required as a future patch introduces a BPF memory allocator interface, where we take the program BTF's type ID as an argument. Each distinct type ID may result in the returned pointer obtaining a different size, hence precision tracking is needed, and pruning cannot just happen when the old value is within the current value. It must only happen when the type ID is equal. The type ID will always correspond to prog->aux->btf hence actual type match is not required.
Finally, change mark_chain_precision precision argument to EXACT for kfuncs constant non-size scalar arguments (tagged with __k suffix).
--- include/linux/bpf_verifier.h | 10 +++- kernel/bpf/verifier.c | 111 ++++++++++++++++++++++------------- 2 files changed, 76 insertions(+), 45 deletions(-)
include/linux/bpf_verifier.h +7 -3
@@ -43,6 +43,12 @@ enum bpf_reg_liveness { REG_LIVE_DONE = 0x8, /* liveness won't be updating this register anymore */ }; +enum bpf_reg_precise { +NOT_PRECISE , +PRECISE , +EXACT , +}; +struct bpf_reg_state {/* Ordering of fields matters. See states_equal() */ enum bpf_reg_type type;
@@ -180,7 +186,7 @@ struct bpf_reg_state {s32 subreg_def;enum bpf_reg_liveness live;/* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */ -bool precise; +enum bpf_reg_precise precise; };enum bpf_stack_slot_type {
@@ -626,8 +632,6 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, struct bpf_attach_target_info *tgt_info);void bpf_free_kfunc_btf_tab (struct bpf_kfunc_btf_tab *tab); -int mark_chain_precision (struct bpf_verifier_env *env,int regno); -#define BPF_BASE_TYPE_MASK GENMASK(BPF_BASE_TYPE_BITS - 1, 0)/* extract base type from bpf_{arg, return, reg}_type. */
kernel/bpf/verifier.c +69 -42
318 lines not on this page · raw message
Signature · 1 line
2.38.1
Re: [PATCH bpf-next v5 15/25] bpf: Teach verifier about non-size constant arguments
Quoted 32 lines · Kumar Kartikeya Dwivedi
Currently, the verifier has support for various arguments that either describe the size of the memory being passed in to a helper, or describe the size of the memory being returned. When a constant is passed in like this, it is assumed for the purposes of precision tracking that if the value in the already explored safe state is within the value in current state, it would fine to prune the search.
While this holds well for size arguments, arguments where each value may denote a distinct meaning and needs to be verified separately needs more work. Search can only be pruned if both are equivalent values. In all other cases, it would be incorrect to treat those two precise registers as equivalent if the new value satisfies the old one (i.e. old <= cur).
Hence, make the register precision marker tri-state. There are now three values that
reg->precisetakes:NOT_PRECISE, PRECISE, EXACT.Both PRECISE and EXACT are 'true' values. EXACT affects how regsafe decides whether both registers are equivalent for the purposes of verifier state equivalence. When it sees that old state register has
reg->precise== EXACT, unless both are equivalent, it will return false. Otherwise, for PRECISE case it falls back to the default check that is present now (i.e. thinking that we're talking about sizes).This is required as a future patch introduces a BPF memory allocator interface, where we take the program BTF's type ID as an argument. Each distinct type ID may result in the returned pointer obtaining a different size, hence precision tracking is needed, and pruning cannot just happen when the old value is within the current value. It must only happen when the type ID is equal. The type ID will always correspond to
prog->aux->btfhence actual type match is not required.
Quoted 5 lines
Finally, change
mark_chain_precisionprecision argument to EXACT for kfuncs constant non-size scalar arguments (tagged with__ksuffix).Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
I think this needs a bit more thinking, tbh. First, with my recent changes we don't even set precision marks in current state, everything stays imprecise. And then under CAP_BPF we also aggressively reset precision. This might work for EXACT, but needs careful consideration.
But, taking a step back. I'm trying to understand why this whole EXACT mode is necessary. SCALAR has min/max values which regsafe() does check. So for those special ___k arguments, if we correctly set min/max values to be *exactly* the btf_id passed in, why would regsafe()'s logic not work?
Quoted 9 lines
include/linux/bpf_verifier.h | 10 +++- kernel/bpf/verifier.c | 111 ++++++++++++++++++++++------------- 2 files changed, 76 insertions(+), 45 deletions(-)include/linux/bpf_verifier.h +0
-0
include/linux/bpf_verifier.h enum bpf_reg_liveness { · line 47
Quoted 5 lines
include/linux/bpf_verifier.h +2
-0@@ -43,6 +43,12 @@ enum bpf_reg_liveness { REG_LIVE_DONE = 0x8, /* liveness won't be updating this register anymore */ }; +enum bpf_reg_precise { +NOT_PRECISE ,
IMPRECISE?
Quoted 48 lines
include/linux/bpf_verifier.h +5
-3+ PRECISE, + EXACT, +}; +struct bpf_reg_state {/* Ordering of fields matters. See states_equal() */ enum bpf_reg_type type;@@ -180,7 +186,7 @@ struct bpf_reg_state {s32 subreg_def;enum bpf_reg_liveness live;/* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */ -bool precise; +enum bpf_reg_precise precise; };enum bpf_stack_slot_type {@@ -626,8 +632,6 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, struct bpf_attach_target_info *tgt_info);void bpf_free_kfunc_btf_tab (struct bpf_kfunc_btf_tab *tab); -int mark_chain_precision (struct bpf_verifier_env *env,int regno); -#define BPF_BASE_TYPE_MASK GENMASK(BPF_BASE_TYPE_BITS - 1, 0)/* extract base type from bpf_{arg, return, reg}_type. */ kernel/bpf/verifier.c +2
-2@@ -864,7 +864,7 @@ static void print_verifier_state(struct bpf_verifier_env *env,print_liveness (env, reg->live);verbose (env,"=" );if (t ==SCALAR_VALUE && reg->precise) -verbose (env,"P" ); +verbose (env, reg->precise ==EXACT ?"E" :"P" );if ((t ==SCALAR_VALUE || t ==PTR_TO_STACK ) &&tnum_is_const (reg->var_off)) {/* reg->off should be 0 for SCALAR_VALUE */ @@ -961,7 +961,7 @@ static void print_verifier_state(struct bpf_verifier_env *env, t = reg->type;verbose (env,"=%s" , t ==SCALAR_VALUE ?"" :reg_type_str (env, t));if (t ==SCALAR_VALUE && reg->precise) -verbose (env,"P" ); +verbose (env, reg->precise ==EXACT ?"E" :"P" );if (t ==SCALAR_VALUE &&tnum_is_const (reg->var_off))verbose (env,"%lld" , reg->var_off.value + reg->off); }else {
kernel/bpf/verifier.c static void __mark_reg_unknown(const struct bpf_verifier_env *env, · line 1702
Quoted 9 lines
kernel/bpf/verifier.c +5
-1@@ -1695,7 +1695,16 @@ static void __mark_reg_unknown(const struct bpf_verifier_env *env, reg->type =SCALAR_VALUE ; reg->var_off = tnum_unknown; reg->frameno =0 ; - reg->precise = !env->bpf_capable; +/* Helpers requiring EXACT for constant arguments cannot be called from +* programs without CAP_BPF. This is because we don't propagate +* precision markers when CAP_BPF is missing. If we allowed calling such +* heleprs in those programs, the default would have to be EXACT for +* them, which would be too aggresive, or we'd have to propagate it.
typos: helpers, aggressive
Quoted 8 lines
kernel/bpf/verifier.c +5
-0+* +* Currently, only bpf_obj_new kfunc requires EXACT precision for its +* scalar argument, which is a kfunc (and thus behind CAP_BPF). +*/ + reg->precise = !env->bpf_capable ?PRECISE :NOT_PRECISE ;__mark_reg_unbounded (reg); }
[...]
Quoted 12 lines
/* Do sanity checks against current state of register and/or stack
* slot, but don't set precise flag in current state, as precisionkernel/bpf/verifier.c +1
-1@@ -2969,7 +2982,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r reg_mask &= ~(1u << i);continue ; } - reg->precise = true; + reg->precise = precise; }return 0 ; }
kernel/bpf/verifier.c static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r · line 3004
Quoted 5 lines
kernel/bpf/verifier.c +1
-1@@ -2988,7 +3001,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r err =backtrack_insn (env, i, ®_mask, &stack_mask); }if (err == -ENOTSUPP ) { -mark_all_scalars_precise (env, st); +mark_all_scalars_precise (env, st, precise);
well... do you really intend to remark everything as EXACT, even registers that have no business of being EXACT? Seems a bit too blunt.
Quoted 12 lines
kernel/bpf/verifier.c +1
-1return 0 ; }else if (err) {return err;@@ -3029,7 +3042,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, int frame, int r }if (!reg->precise) new_marks = true; - reg->precise = true; + reg->precise = precise; }bitmap_from_u64 (mask, stack_mask);
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v5 15/25] bpf: Teach verifier about non-size constant arguments
Quoted 36 lines · Andrii Nakryiko
Quoted 35 lines · Kumar Kartikeya Dwivedi
Currently, the verifier has support for various arguments that either describe the size of the memory being passed in to a helper, or describe the size of the memory being returned. When a constant is passed in like this, it is assumed for the purposes of precision tracking that if the value in the already explored safe state is within the value in current state, it would fine to prune the search.
While this holds well for size arguments, arguments where each value may denote a distinct meaning and needs to be verified separately needs more work. Search can only be pruned if both are equivalent values. In all other cases, it would be incorrect to treat those two precise registers as equivalent if the new value satisfies the old one (i.e. old <= cur).
Hence, make the register precision marker tri-state. There are now three values that
reg->precisetakes:NOT_PRECISE, PRECISE, EXACT.Both PRECISE and EXACT are 'true' values. EXACT affects how regsafe decides whether both registers are equivalent for the purposes of verifier state equivalence. When it sees that old state register has
reg->precise== EXACT, unless both are equivalent, it will return false. Otherwise, for PRECISE case it falls back to the default check that is present now (i.e. thinking that we're talking about sizes).This is required as a future patch introduces a BPF memory allocator interface, where we take the program BTF's type ID as an argument. Each distinct type ID may result in the returned pointer obtaining a different size, hence precision tracking is needed, and pruning cannot just happen when the old value is within the current value. It must only happen when the type ID is equal. The type ID will always correspond to
prog->aux->btfhence actual type match is not required.Finally, change
mark_chain_precisionprecision argument to EXACT for kfuncs constant non-size scalar arguments (tagged with__ksuffix).
Quoted 8 lines
Quoted 2 lines
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
I think this needs a bit more thinking, tbh. First, with my recent changes we don't even set precision marks in current state, everything stays imprecise. And then under
CAP_BPFwe also aggressively reset precision. This might work for EXACT, but needs careful consideration.
I'm sorry, I didn't have the time to look at the series, but I spent the whole day going over it today, and it makes a lot of sense to me. I think I also misunderstood some things and going through it brought some clarity.
I think resetting precision is fine here too. As you've stated in that series, the verifier while simulating execution in current state already checks everything.
Quoted 6 lines
But, taking a step back. I'm trying to understand why this whole EXACT mode is necessary. SCALAR has min/max values which
regsafe()does check. So for those special___karguments, if we correctly set min/max values to be *exactly* thebtf_idpassed in, why would regsafe()'s logic not work?
Yes, when you have tnum_is_const var_off, regsafe will return false (when reg->precise is true). So EXACT is unnecessary in that case.
I think you're probably right. The range_within will fail for k1 != k2, but I was more concerned for cases where it's not a constant made a concrete value later.
rX = ...; [X1, Y1];
if (cond) // unknown
rX = ...; [X2, Y2];
p:
...
p is a pruning point, so states will be compared there. I spent a fair amount of time today trying to break this in different ways but failed so far. Also, thinking about if it still lies within that range, things should still be ok since it's in the same admissible set of values for the scalar that was later refined and used as a constant for bpf_obj_new. So it's probably unncessary to make things more pessimistic. The verifier will ensure the current value is always a subset of the old value.
I guess I'll hold on for this patch and work towards getting the rest in first (since Alexei mentioned others are waiting on this), and circle back to this if I am able to construct a real test case that breaks. bpf_obj_new is already behind CAP_BPF.
Kumar Kartikeya Dwivedi
maintainerPATCH v5 16/25 bpf: Introduce bpf_obj_new
In reply to Kumar Kartikeya DwivediIntroduce type safe memory allocator bpf_obj_new for BPF programs. The kernel side kfunc is named bpf_obj_new_impl, as passing hidden arguments to kfuncs still requires having them in prototype, unlike BPF helpers which always take 5 arguments and have them checked using bpf_func_proto in verifier, ignoring unset argument types.
Introduce __ign suffix to ignore a specific kfunc argument during type checks, then use this to introduce support for passing type metadata to the bpf_obj_new_impl kfunc.
The user passes BTF ID of the type it wants to allocates in program BTF, the verifier then rewrites the first argument as the size of this type, after performing some sanity checks (to ensure it exists and it is a struct type).
The second argument is also fixed up and passed by the verifier. This is the btf_struct_meta for the type being allocated. It would be needed mostly for the offset array which is required for zero initializing special fields while leaving the rest of storage in unitialized state.
It would also be needed in the next patch to perform proper destruction of the object's special fields.
Under the hood, bpf_obj_new will call bpf_mem_alloc and bpf_mem_free, using the any context BPF memory allocator introduced recently. To this end, a global instance of the BPF memory allocator is initialized on boot to be used for this purpose. This 'bpf_global_ma' serves all allocations for bpf_obj_new. In the future, bpf_obj_new variants will allow specifying a custom allocator.
Note that now that bpf_obj_new can be used to allocate local kptrs that can be linked to BPF linked list (when future linked list helpers are available), we need to also free the elements using bpf_mem_free. However, since the draining of elements is done outside the bpf_spin_lock, we need to do migrate_disable around the call since bpf_list_head_free can be called from map free path where migration is enabled. Otherwise, when called from BPF programs migration is already disabled.
A convenience macro is included in the bpf_experimental.h header to hide over the ugly details of the implementation, leading to user code looking similar to a language level extension which allocates and constructs fields of a user type.
struct bar {
struct bpf_list_node node;
};struct foo {
struct bpf_spin_lock lock;
struct bpf_list_head head __contains(bar, node);
};void prog(void) {
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return;
...
}A key piece of this story is still missing, i.e. the free function, which will come in the next patch.
--- include/linux/bpf.h | 21 ++-- include/linux/bpf_verifier.h | 2 + kernel/bpf/core.c | 16 +++ kernel/bpf/helpers.c | 47 ++++++-- kernel/bpf/verifier.c | 107 ++++++++++++++++-- .../testing/selftests/bpf/bpf_experimental.h | 25 ++++ 6 files changed, 195 insertions(+), 23 deletions(-) create mode 100644 tools/testing/selftests/bpf/bpf_experimental.h
include/linux/bpf.h +13 -8
@@ -54,6 +54,8 @@ struct cgroup;extern struct idr btf_idr;extern spinlock_t btf_idr_lock;extern struct kobject *btf_kobj; +extern struct bpf_mem_alloc bpf_global_ma; +extern bool bpf_global_ma_set;typedef u64 (*bpf_callback_t )(u64 ,u64 ,u64 ,u64 ,u64 ); typedef int (*bpf_iter_init_seq_priv_t)(void *private_data,
@@ -333,16 +335,19 @@ static inline bool btf_record_has_field(const struct btf_record *rec, enum btf_freturn rec->field_mask & type; } -static inline void check_and_init_map_value (struct bpf_map *map,void *dst) +static inline void bpf_obj_init (const struct btf_field_offs *foffs,void *obj) { -if (!IS_ERR_OR_NULL (map->record)) { -struct btf_field *fields = map->record->fields; -u32 cnt = map->record->cnt; -int i; +int i; -for (i =0 ; i < cnt; i++) -memset (dst + fields[i].offset,0 ,btf_field_type_size (fields[i].type)); - } +if (!foffs) +return ; +for (i =0 ; i < foffs->cnt; i++) +memset (obj + foffs->field_off[i],0 , foffs->field_sz[i]); +} + +static inline void check_and_init_map_value (struct bpf_map *map,void *dst) +{ +bpf_obj_init (map->field_offs, dst); }/* memcpy that is used with 8-byte aligned pointers, power-of-8 size and
include/linux/bpf_verifier.h +2 -0
@@ -428,6 +428,8 @@ struct bpf_insn_aux_data {*/ struct bpf_loop_inline_state loop_inline_state; }; +u64 obj_new_size;/* remember the size of type passed to bpf_obj_new to rewrite R1 */ +struct btf_struct_meta *kptr_struct_meta;u64 map_key_state;/* constant (32 bit) key tracking for maps */ int ctx_field_size;/* the ctx field size for load insn, maybe 0 */ u32 seen;/* this insn was processed by the verifier at env->pass_cnt */
kernel/bpf/core.c +16 -0
@@ -34,6 +34,7 @@#include <linux/log2.h> #include <linux/bpf_verifier.h> #include <linux/nodemask.h> +#include <linux/bpf_mem_alloc.h> #include <asm/barrier.h> #include <asm/unaligned.h>
@@ -60,6 +61,9 @@#define CTX regs[BPF_REG_CTX]#define IMM insn->imm +struct bpf_mem_alloc bpf_global_ma; +bool bpf_global_ma_set; +/* No hurry in this branch * * Exported for the bpf jit load helper.
@@ -2746,6 +2750,18 @@ int __weak bpf_arch_text_invalidate(void *dst, size_t len)return -ENOTSUPP ; } +#ifdef CONFIG_BPF_SYSCALL +static int __init bpf_global_ma_init (void ) +{ +int ret; + + ret =bpf_mem_alloc_init (&bpf_global_ma,0 , false); + bpf_global_ma_set = !ret; +return ret; +} +late_initcall (bpf_global_ma_init); +#endif +DEFINE_STATIC_KEY_FALSE (bpf_stats_enabled_key);EXPORT_SYMBOL (bpf_stats_enabled_key);
kernel/bpf/helpers.c +40 -7
71 lines not on this page · raw message
kernel/bpf/verifier.c +99 -8
165 lines not on this page · raw message
tools/testing/selftests/bpf/bpf_experimental.h new file +25 -0
@@ -0,0 +1,25 @@ +#ifndef __BPF_EXPERIMENTAL__ +#define __BPF_EXPERIMENTAL__ + +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_core_read.h> + +/* Description +* Allocates a local kptr of type represented by 'local_type_id' in program +* BTF. User may use the bpf_core_type_id_local macro to pass the type ID +* of a struct in program BTF. +* +* The 'local_type_id' parameter must be a known constant. +* The 'meta' parameter is a hidden argument that is ignored. +* Returns +* A local kptr corresponding to passed in 'local_type_id', or NULL on +* failure. +*/ +extern void *bpf_obj_new_impl (__u64 local_type_id,void *meta) __ksym; + +/* Convenience macro to wrap over bpf_obj_new_impl */ +#define bpf_obj_new (type) ((type *)bpf_obj_new_impl(bpf_core_type_id_local(type), NULL)) + +#endif
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 17/25 bpf: Introduce bpf_obj_drop
In reply to Kumar Kartikeya DwivediIntroduce bpf_obj_drop, which is the kfunc used to free local kptrs allocated using bpf_obj_new. Similar to bpf_obj_new, it implicitly destructs the fields part of the local kptr automatically without user intervention.
Just like the previous patch, btf_struct_meta that is needed to free up the special fields is passed as a hidden argument to the kfunc.
For the user, a convenience macro hides over the kernel side kfunc which is named bpf_obj_drop_impl.
Continuing the previous example:
void prog(void) {
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return;
bpf_obj_drop(f);
}--- kernel/bpf/helpers.c | 11 ++++ kernel/bpf/verifier.c | 66 +++++++++++++++---- .../testing/selftests/bpf/bpf_experimental.h | 13 ++++ 3 files changed, 79 insertions(+), 11 deletions(-)
kernel/bpf/helpers.c +11 -0
@@ -1770,6 +1770,16 @@ void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign)return p; } +void bpf_obj_drop_impl (void *p__lkptr,void *meta__ign) +{ +struct btf_struct_meta *meta = meta__ign; +void *p = p__lkptr; + +if (meta) +bpf_obj_free_fields (meta->record, p); +bpf_mem_free (&bpf_global_ma, p); +} +__diag_pop ();BTF_SET8_START (generic_btf_ids)
@@ -1777,6 +1787,7 @@ BTF_SET8_START(generic_btf_ids) BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)#endif BTF_ID_FLAGS(func, bpf_obj_new_impl, KF_ACQUIRE | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE)BTF_SET8_END (generic_btf_ids) static const struct btf_kfunc_id_set generic_kfunc_set = {
kernel/bpf/verifier.c +55 -11
126 lines not on this page · raw message
tools/testing/selftests/bpf/bpf_experimental.h +13 -0
@@ -22,4 +22,17 @@ extern void *bpf_obj_new_impl(__u64 local_type_id, void *meta) __ksym;/* Convenience macro to wrap over bpf_obj_new_impl */ #define bpf_obj_new (type) ((type *)bpf_obj_new_impl(bpf_core_type_id_local(type), NULL)) +/* Description +* Free a local kptr. All fields of local kptr that require destruction +* will be destructed before the storage is freed. +* +* The 'meta' parameter is a hidden argument that is ignored. +* Returns +* Void. +*/ +extern void bpf_obj_drop_impl(void *kptr, void *meta) __ksym; + +/* Convenience macro to wrap over bpf_obj_drop_impl */ +#define bpf_obj_drop (kptr) bpf_obj_drop_impl(kptr, NULL) +#endif
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 18/25 bpf: Permit NULL checking pointer with non-zero fixed offset
In reply to Kumar Kartikeya DwivediPointer increment on seeing PTR_MAYBE_NULL is already protected against, hence make an exception for local kptrs while still keeping the warning for other unintended cases that might creep in.
bpf_list_pop_{front,_back} helpers planned to be introduced in next commit will return a local kptr with incremented offset pointing to bpf_list_node field. The user is supposed to then obtain the pointer to the entry using container_of after NULL checking it. The current restrictions trigger a warning when doing the NULL checking. Revisiting the reason, it is meant as an assertion which seems to actually work and catch the bad case.
Hence, under no other circumstances can reg->off be non-zero for a register that has the PTR_MAYBE_NULL type flag set.
--- kernel/bpf/verifier.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/bpf/verifier.c +9 -4
@@ -10809,15 +10809,20 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state, {if (type_may_be_null (reg->type) && reg->id == id && !WARN_ON_ONCE (!reg->id)) { -if (WARN_ON_ONCE (reg->smin_value || reg->smax_value || - !tnum_equals_const (reg->var_off,0 ) || - reg->off)) { +if (reg->smin_value || reg->smax_value || !tnum_equals_const (reg->var_off,0 ) || reg->off) {/* Old offset (both fixed and variable parts) should * have been known-zero, because we don't allow pointer * arithmetic on pointers that might be NULL. If we * see this happening, don't convert the register. +* +* But in some cases, some helpers that return local +* kptrs advance offset for the returned pointer. +* In those cases, it is fine to expect to see reg->off. */ -return ; +if (WARN_ON_ONCE (reg->type != (PTR_TO_BTF_ID |MEM_ALLOC |PTR_MAYBE_NULL ))) +return ; +if (WARN_ON_ONCE (reg->smin_value || reg->smax_value || !tnum_equals_const (reg->var_off,0 ))) +return ; }if (is_null) { reg->type =SCALAR_VALUE ;
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 19/25 bpf: Introduce single ownership BPF linked list API
In reply to Kumar Kartikeya DwivediAdd a linked list API for use in BPF programs, where it expects protection from the bpf_spin_lock in the same allocation as the bpf_list_head. For now, only one bpf_spin_lock can be present hence that is assumed to be the one protecting the bpf_list_head.
The following functions are added to kick things off:
// Add node to beginning of list
void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node);
// Add node to end of list
void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node);
// Remove node at beginning of list and return it
struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head);
// Remove node at end of list and return it
struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head);
The lock protecting the bpf_list_head needs to be taken for all operations. The verifier ensures that the lock that needs to be taken is always held, and only the correct lock is taken for these operations. These checks are made statically by relying on the reg->id preserved for registers pointing into regions having both bpf_spin_lock and the objects protected by it. The comment over check_reg_allocation_locked in this change describes the logic in detail.
Note that bpf_list_push_front and bpf_list_push_back are meant to consume the object containing the node in the 1st argument, however that specific mechanism is intended to not release the ref_obj_id directly until the bpf_spin_unlock is called. In this commit, nothing is done, but the next commit will be introducing logic to handle this case, so it has been left as is for now.
bpf_list_pop_front and bpf_list_pop_back delete the first or last item of the list respectively, and return pointer to the element at the list_node offset. The user can then use container_of style macro to get the actual entry type. The verifier however statically knows the actual type, so the safety properties are still preserved.
With these additions, programs can now manage their own linked lists and store their objects in them.
--- kernel/bpf/helpers.c | 55 +++- kernel/bpf/verifier.c | 292 +++++++++++++++++- .../testing/selftests/bpf/bpf_experimental.h | 28 ++ 3 files changed, 361 insertions(+), 14 deletions(-)
kernel/bpf/helpers.c +54 -1
@@ -1780,6 +1780,50 @@ void bpf_obj_drop_impl(void *p__lkptr, void *meta__ign)bpf_mem_free (&bpf_global_ma, p); } +static void __bpf_list_add (struct bpf_list_node *node,struct bpf_list_head *head,bool tail) +{ +struct list_head *n = (void *)node, *h = (void *)head; + +if (unlikely (!h->next)) +INIT_LIST_HEAD (h); +if (unlikely (!n->next)) +INIT_LIST_HEAD (n); + tail ?list_add_tail (n, h) :list_add (n, h); +} + +void bpf_list_push_front (struct bpf_list_head *head,struct bpf_list_node *node) +{ +return __bpf_list_add (node, head, false); +} + +void bpf_list_push_back (struct bpf_list_head *head,struct bpf_list_node *node) +{ +return __bpf_list_add (node, head, true); +} + +static struct bpf_list_node *__bpf_list_del (struct bpf_list_head *head,bool tail) +{ +struct list_head *n, *h = (void *)head; + +if (unlikely (!h->next)) +INIT_LIST_HEAD (h); +if (list_empty (h)) +return NULL ; + n = tail ? h->prev : h->next; +list_del_init (n); +return (struct bpf_list_node *)n; +} + +struct bpf_list_node *bpf_list_pop_front (struct bpf_list_head *head) +{ +return __bpf_list_del (head, false); +} + +struct bpf_list_node *bpf_list_pop_back (struct bpf_list_head *head) +{ +return __bpf_list_del (head, true); +} +__diag_pop ();BTF_SET8_START (generic_btf_ids)
@@ -1788,6 +1832,10 @@ BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)#endif BTF_ID_FLAGS(func, bpf_obj_new_impl, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_list_push_front) +BTF_ID_FLAGS(func, bpf_list_push_back) +BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)BTF_SET8_END (generic_btf_ids) static const struct btf_kfunc_id_set generic_kfunc_set = {
@@ -1797,7 +1845,12 @@ static const struct btf_kfunc_id_set generic_kfunc_set = {static int __init kfunc_init (void ) { -return register_btf_kfunc_id_set (BPF_PROG_TYPE_TRACING , &generic_kfunc_set); +int ret; + + ret =register_btf_kfunc_id_set (BPF_PROG_TYPE_TRACING , &generic_kfunc_set); +if (ret) +return ret; +return register_btf_kfunc_id_set (BPF_PROG_TYPE_SCHED_CLS , &generic_kfunc_set); }late_initcall (kfunc_init);
kernel/bpf/verifier.c +279 -13
379 lines not on this page · raw message
tools/testing/selftests/bpf/bpf_experimental.h +28 -0
@@ -35,4 +35,32 @@ extern void bpf_obj_drop_impl(void *kptr, void *meta) __ksym;/* Convenience macro to wrap over bpf_obj_drop_impl */ #define bpf_obj_drop (kptr) bpf_obj_drop_impl(kptr, NULL) +/* Description +* Add a new entry to the beginning of the BPF linked list. +* Returns +* Void. +*/ +extern void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node) __ksym; + +/* Description +* Add a new entry to the end of the BPF linked list. +* Returns +* Void. +*/ +extern void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node) __ksym; + +/* Description +* Remove the entry at the beginning of the BPF linked list. +* Returns +* Pointer to bpf_list_node of deleted entry, or NULL if list is empty. +*/ +extern struct bpf_list_node *bpf_list_pop_front (struct bpf_list_head *head) __ksym; + +/* Description +* Remove the entry at the end of the BPF linked list. +* Returns +* Pointer to bpf_list_node of deleted entry, or NULL if list is empty. +*/ +extern struct bpf_list_node *bpf_list_pop_back (struct bpf_list_head *head) __ksym; +#endif
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 20/25 bpf: Add 'release on unlock' logic for bpf_list_push_{front,back}
In reply to Kumar Kartikeya DwivediThis commit implements the delayed release logic for bpf_list_push_front and bpf_list_push_back.
Once a node has been added to the list, it's pointer changes to PTR_UNTRUSTED. However, it is only released once the lock protecting the list is unlocked. For such local kptrs with PTR_UNTRUSTED set but an active ref_obj_id, it is still permitted to read them as long as the lock is held. Writing to them is not allowed.
This allows having read access to push items we no longer own until we release the lock guarding the list, allowing a little more flexibility when working with these APIs.
Note that enabling write support has fairly tricky interactions with what happens inside the critical section. Just as an example, currently, bpf_obj_drop is not permitted, but if it were, being able to write to the PTR_UNTRUSTED pointer while the object gets released back to the memory allocator would violate safety properties we wish to guarantee (i.e. not crashing the kernel). The memory could be reused for a different type in the BPF program or even in the kernel as it gets eventually kfree'd.
Not enabling bpf_obj_drop inside the critical section would appear to prevent all of the above, but that is more of an artifical limitation right now. Since the write support is tangled with how we handle potential aliasing of nodes inside the critical section that may or may not be part of the list anymore, it has been deferred to a future patch.
--- include/linux/bpf_verifier.h | 5 ++++ kernel/bpf/verifier.c | 48 +++++++++++++++++++++++++++++++++++- 2 files changed, 52 insertions(+), 1 deletion(-)
include/linux/bpf_verifier.h +5 -0
@@ -229,6 +229,11 @@ struct bpf_reference_state {* exiting a callback function. */ int callback_ref; +/* Mark the reference state to release the registers sharing the same id +* on bpf_spin_unlock (for nodes that we will lose ownership to but are +* safe to access inside the critical section). +*/ +bool release_on_unlock; };/* state of the program:
kernel/bpf/verifier.c +47 -1
73 lines not on this page · raw message
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 21/25 selftests/bpf: Add __contains macro to bpf_experimental.h
In reply to Kumar Kartikeya DwivediAdd user facing __contains macro which provides a convenient wrapper over the verbose kernel specific BTF declaration tag required to annotate BPF list head structs in user types.
--- tools/testing/selftests/bpf/bpf_experimental.h | 2 ++ 1 file changed, 2 insertions(+)
tools/testing/selftests/bpf/bpf_experimental.h +2 -0
@@ -6,6 +6,8 @@#include <bpf/bpf_helpers.h> #include <bpf/bpf_core_read.h> +#define __contains (name, node) __attribute__((btf_decl_tag("contains:" #name ":" #node))) +/* Description * Allocates a local kptr of type represented by 'local_type_id' in program * BTF. User may use the bpf_core_type_id_local macro to pass the type ID
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 22/25 selftests/bpf: Update spinlock selftest
In reply to Kumar Kartikeya DwivediMake updates in preparation for adding more test cases to this selftest:
- Convert from CHECK_ to ASSERT macros.
- Use BPF skeleton
- Fix typo sping -> spin
- Rename spinlock.c -> spin_lock.c
--- .../selftests/bpf/prog_tests/spin_lock.c | 46 +++++++++++++++++++ .../selftests/bpf/prog_tests/spinlock.c | 45 ------------------ .../selftests/bpf/progs/test_spin_lock.c | 4 +- 3 files changed, 48 insertions(+), 47 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/spin_lock.c delete mode 100644 tools/testing/selftests/bpf/prog_tests/spinlock.c
tools/testing/selftests/bpf/prog_tests/spin_lock.c new file +46 -0
@@ -0,0 +1,46 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <test_progs.h> +#include <network_helpers.h> + +#include "test_spin_lock.skel.h" + +static void *spin_lock_thread (void *arg) +{ +int err, prog_fd = *(u32 *) arg; +LIBBPF_OPTS (bpf_test_run_opts, topts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 10000, + ); + + err =bpf_prog_test_run_opts (prog_fd, &topts); +ASSERT_OK (err,"test_run" ); +ASSERT_OK (topts.retval,"test_run retval" ); +pthread_exit (arg); +} + +void test_spinlock (void ) +{ +struct test_spin_lock *skel; +pthread_t thread_id[4 ]; +int prog_fd, i; +void *ret; + + skel =test_spin_lock__open_and_load (); +if (!ASSERT_OK_PTR (skel,"test_spin_lock__open_and_load" )) +return ; + prog_fd =bpf_program__fd (skel->progs.bpf_spin_lock_test); +for (i =0 ; i <4 ; i++) +if (!ASSERT_OK (pthread_create (&thread_id[i],NULL , + &spin_lock_thread, &prog_fd),"pthread_create" )) + goto end; + +for (i =0 ; i <4 ; i++) { +if (!ASSERT_OK (pthread_join (thread_id[i], &ret),"pthread_join" )) + goto end; +if (!ASSERT_EQ (ret, &prog_fd,"ret == prog_fd" )) + goto end; + } +end: +test_spin_lock__destroy (skel); +}
tools/testing/selftests/bpf/prog_tests/spinlock.c deleted +0 -45
45 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_spin_lock.c +2 -2
@@ -45,8 +45,8 @@ struct {#define CREDIT_PER_NS (delta, rate) (((delta) * rate) >> 20) -SEC("tc") -int bpf_sping_lock_test (struct __sk_buff *skb) +SEC("cgroup_skb/ingress") +int bpf_spin_lock_test (struct __sk_buff *skb) {volatile int credit =0 , max_credit =100 , pkt_len =64 ;struct hmap_elem zero = {}, *val;
Signature · 1 line
2.38.1
Re: [PATCH bpf-next v5 22/25] selftests/bpf: Update spinlock selftest
Quoted 16 lines · Kumar Kartikeya Dwivedi
Make updates in preparation for adding more test cases to this selftest:
- Convert fromCHECK_to ASSERT macros.
- Use BPF skeleton
- Fix typo sping -> spin
- Renamespinlock.c->spin_lock.cSigned-off-by Kumar Kartikeya Dwivedi <[email protected]>--- .../selftests/bpf/prog_tests/spin_lock.c | 46 +++++++++++++++++++ .../selftests/bpf/prog_tests/spinlock.c | 45 ------------------ .../selftests/bpf/progs/test_spin_lock.c | 4 +- 3 files changed, 48 insertions(+), 47 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/spin_lock.c delete mode 100644 tools/testing/selftests/bpf/prog_tests/spinlock.c
[...]
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/spin_lock.c new file +4
-0+void test_spinlock (void ) +{ +struct test_spin_lock *skel; +pthread_t thread_id[4 ];
tools/testing/selftests/bpf/prog_tests/spin_lock.c line 35
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/spin_lock.c new file +10
-0+int prog_fd, i; +void *ret; + + skel =test_spin_lock__open_and_load (); +if (!ASSERT_OK_PTR (skel,"test_spin_lock__open_and_load" )) +return ; + prog_fd =bpf_program__fd (skel->progs.bpf_spin_lock_test); +for (i =0 ; i <4 ; i++) +if (!ASSERT_OK (pthread_create (&thread_id[i],NULL , + &spin_lock_thread, &prog_fd),"pthread_create" ))
I mean... does that pthread_create() call have to happen inside ASSERT_OK?
err = pthread_create(...)
if (!ASSERT_OK(err, "pthread_create"))
goto end;
Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/spin_lock.c new file +11
-0+ goto end; + +for (i =0 ; i <4 ; i++) { +if (!ASSERT_OK (pthread_join (thread_id[i], &ret),"pthread_join" )) + goto end; +if (!ASSERT_EQ (ret, &prog_fd,"ret == prog_fd" )) + goto end; + } +end: +test_spin_lock__destroy (skel); +}
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v5 22/25] selftests/bpf: Update spinlock selftest
Quoted 37 lines · Andrii Nakryiko
Quoted 16 lines · Kumar Kartikeya Dwivedi
Make updates in preparation for adding more test cases to this selftest:
- Convert fromCHECK_to ASSERT macros.
- Use BPF skeleton
- Fix typo sping -> spin
- Renamespinlock.c->spin_lock.cSigned-off-by Kumar Kartikeya Dwivedi <[email protected]>--- .../selftests/bpf/prog_tests/spin_lock.c | 46 +++++++++++++++++++ .../selftests/bpf/prog_tests/spinlock.c | 45 ------------------ .../selftests/bpf/progs/test_spin_lock.c | 4 +- 3 files changed, 48 insertions(+), 47 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/spin_lock.c delete mode 100644 tools/testing/selftests/bpf/prog_tests/spinlock.c[...]
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/spin_lock.c new file +4
-0+void test_spinlock (void ) +{ +struct test_spin_lock *skel; +pthread_t thread_id[4 ];tools/testing/selftests/bpf/prog_tests/spin_lock.c line 35
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/spin_lock.c new file +10
-0+int prog_fd, i; +void *ret; + + skel =test_spin_lock__open_and_load (); +if (!ASSERT_OK_PTR (skel,"test_spin_lock__open_and_load" )) +return ; + prog_fd =bpf_program__fd (skel->progs.bpf_spin_lock_test); +for (i =0 ; i <4 ; i++) +if (!ASSERT_OK (pthread_create (&thread_id[i],NULL , + &spin_lock_thread, &prog_fd),"pthread_create" ))I mean... does that
pthread_create()call have to happen insideASSERT_OK?
Quoted 4 lines
err =
pthread_create(...)
if (!ASSERT_OK(err, "pthread_create"))goto end;
Ack, I'll rewrite it like this.
Kumar Kartikeya Dwivedi
maintainerPATCH v5 23/25 selftests/bpf: Add failure test cases for spin lock pairing
In reply to Kumar Kartikeya DwivediFirst, ensure that whenever a bpf_spin_lock is present in an allocation, the reg->id is preserved. This won't be true for global variables however, since they have a single map value per map, hence the verifier harcodes it to 0 (so that multiple pseudo ldimm64 insns can yield the same lock object per map at a given offset).
Next, add test cases for all possible combinations (kptr, global, map value, inner map value). Since we lifted restriction on locking in inner maps, also add test cases for them. Currently, each lookup into an inner map gets a fresh reg->id, so even if the reg->map_ptr is same, they will be treated as separate allocations and the incorrect unlock pairing will be rejected.
--- .../selftests/bpf/prog_tests/spin_lock.c | 89 +++++++- .../selftests/bpf/progs/test_spin_lock_fail.c | 204 ++++++++++++++++++ 2 files changed, 292 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/test_spin_lock_fail.c
tools/testing/selftests/bpf/prog_tests/spin_lock.c +88 -1
@@ -3,6 +3,79 @@#include <network_helpers.h> #include "test_spin_lock.skel.h" +#include "test_spin_lock_fail.skel.h" + +static char log_buf[1024 *1024 ]; + +static struct { +const char *prog_name; +const char *err_msg; +} spin_lock_fail_tests[] = { + {"lock_id_kptr_preserve" , +"5: (bf) r1 = r0 ; R0_w=ptr_foo(id=2,ref_obj_id=2,off=0,imm=0) " +"R1_w=ptr_foo(id=2,ref_obj_id=2,off=0,imm=0) refs=2\n6: (85) call bpf_this_cpu_ptr#154\n" +"R1 type=ptr_ expected=percpu_ptr_" }, + {"lock_id_global_zero" , +"; R1_w=map_value(off=0,ks=4,vs=4,imm=0)\n2: (85) call bpf_this_cpu_ptr#154\n" +"R1 type=map_value expected=percpu_ptr_" }, + {"lock_id_mapval_preserve" , +"8: (bf) r1 = r0 ; R0_w=map_value(id=1,off=0,ks=4,vs=8,imm=0) " +"R1_w=map_value(id=1,off=0,ks=4,vs=8,imm=0)\n9: (85) call bpf_this_cpu_ptr#154\n" +"R1 type=map_value expected=percpu_ptr_" }, + {"lock_id_innermapval_preserve" , +"13: (bf) r1 = r0 ; R0=map_value(id=2,off=0,ks=4,vs=8,imm=0) " +"R1_w=map_value(id=2,off=0,ks=4,vs=8,imm=0)\n14: (85) call bpf_this_cpu_ptr#154\n" +"R1 type=map_value expected=percpu_ptr_" }, + {"lock_id_mismatch_kptr_kptr" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_kptr_global" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_kptr_mapval" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_kptr_innermapval" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_global_global" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_global_kptr" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_global_mapval" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_global_innermapval" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_mapval_mapval" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_mapval_kptr" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_mapval_global" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_mapval_innermapval" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_innermapval_innermapval1" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_innermapval_innermapval2" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_innermapval_kptr" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_innermapval_global" ,"bpf_spin_unlock of different lock" }, + {"lock_id_mismatch_innermapval_mapval" ,"bpf_spin_unlock of different lock" }, +}; + +static void test_spin_lock_fail_prog (const char *prog_name,const char *err_msg) +{ + LIBBPF_OPTS(bpf_object_open_opts, opts, .kernel_log_buf = log_buf, + .kernel_log_size = sizeof(log_buf), + .kernel_log_level = 1); +struct test_spin_lock_fail *skel; +struct bpf_program *prog; +int ret; + + skel =test_spin_lock_fail__open_opts (&opts); +if (!ASSERT_OK_PTR (skel,"test_spin_lock_fail__open_opts" )) +return ; + + prog =bpf_object__find_program_by_name (skel->obj, prog_name); +if (!ASSERT_OK_PTR (prog,"bpf_object__find_program_by_name" )) + goto end; + +bpf_program__set_autoload (prog, true); + + ret =test_spin_lock_fail__load (skel); +if (!ASSERT_ERR (ret,"test_spin_lock_fail__load must fail" )) + goto end; + +if (!ASSERT_OK_PTR (strstr (log_buf, err_msg),"expected error message" )) { +fprintf (stderr,"Expected: %s\n" , err_msg); +fprintf (stderr,"Verifier: %s\n" , log_buf); + } + +end: +test_spin_lock_fail__destroy (skel); +}static void *spin_lock_thread (void *arg) {
@@ -19,7 +92,7 @@ static void *spin_lock_thread(void *arg)pthread_exit (arg); } -void test_spinlock (void ) +void test_spin_lock_success (void ) {struct test_spin_lock *skel;pthread_t thread_id[4 ];
@@ -44,3 +117,17 @@ void test_spinlock(void) end:test_spin_lock__destroy (skel); } + +void test_spin_lock (void ) +{ +int i; + +test_spin_lock_success (); + +for (i =0 ; i <ARRAY_SIZE (spin_lock_fail_tests); i++) { +if (!test__start_subtest (spin_lock_fail_tests[i].prog_name)) +continue ; +test_spin_lock_fail_prog (spin_lock_fail_tests[i].prog_name, + spin_lock_fail_tests[i].err_msg); + } +}
tools/testing/selftests/bpf/progs/test_spin_lock_fail.c new file +204 -0
204 lines not on this page · raw message
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 24/25 selftests/bpf: Add BPF linked list API tests
In reply to Kumar Kartikeya DwivediInclude various tests covering the success and failure cases. Also, run the success cases at runtime to verify correctness of linked list manipulation routines, in addition to ensuring successful verification.
--- tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../selftests/bpf/prog_tests/linked_list.c | 253 ++++++++ .../testing/selftests/bpf/progs/linked_list.c | 370 +++++++++++ .../testing/selftests/bpf/progs/linked_list.h | 56 ++ .../selftests/bpf/progs/linked_list_fail.c | 581 ++++++++++++++++++ 5 files changed, 1261 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/linked_list.c create mode 100644 tools/testing/selftests/bpf/progs/linked_list.c create mode 100644 tools/testing/selftests/bpf/progs/linked_list.h create mode 100644 tools/testing/selftests/bpf/progs/linked_list_fail.c
tools/testing/selftests/bpf/DENYLIST.s390x +1 -0
@@ -33,6 +33,7 @@ ksyms_module # test_ksyms_module__open_and_load unex ksyms_module_libbpf # JIT does not support calling kernel function (kfunc) ksyms_module_lskel # test_ksyms_module_lskel__open_and_load unexpected error: -9 (?) libbpf_get_fd_by_id_opts # failed to attach: ERROR: strerror_r(-524)=22 (trampoline) +linked_list # JIT does not support calling kernel function (kfunc) lookup_key # JIT does not support calling kernel function (kfunc) lru_bug # prog 'printk': failed to auto-attach: -524 map_kptr # failed to open_and_load program: -524 (trampoline)
tools/testing/selftests/bpf/prog_tests/linked_list.c new file +253 -0
253 lines not on this page · raw message
tools/testing/selftests/bpf/progs/linked_list.c new file +370 -0
370 lines not on this page · raw message
tools/testing/selftests/bpf/progs/linked_list.h new file +56 -0
56 lines not on this page · raw message
tools/testing/selftests/bpf/progs/linked_list_fail.c new file +581 -0
581 lines not on this page · raw message
Signature · 1 line
2.38.1
Kumar Kartikeya Dwivedi
maintainerPATCH v5 25/25 selftests/bpf: Add BTF sanity tests
In reply to Kumar Kartikeya DwivediPreparing the metadata for bpf_list_head involves a complicated parsing step and type resolution for the contained value. Ensure that corner cases are tested against and invalid specifications in source are duly rejected. Also include tests for incorrect ownership relationships in the BTF.
--- .../selftests/bpf/prog_tests/linked_list.c | 271 ++++++++++++++++++ 1 file changed, 271 insertions(+)
tools/testing/selftests/bpf/prog_tests/linked_list.c +271 -0
@@ -1,4 +1,7 @@// SPDX-License-Identifier: GPL-2.0 +#include <bpf/btf.h> +#include <test_btf.h> +#include <linux/btf.h> #include <test_progs.h> #include <network_helpers.h>
@@ -233,6 +236,273 @@ static void test_linked_list_success(int mode, bool leave_in_map)linked_list__destroy (skel); } +/* struct bpf_spin_lock { +* int foo; +* }; +* struct bpf_list_head { +* __u64 :64; +* __u64 :64; +* } __attribute__((aligned(8))); +* struct bpf_list_node { +* __u64 :64; +* __u64 :64; +* } __attribute__((aligned(8))); +*/ +static const char btf_str_sec[] ="\0bpf_spin_lock\0bpf_list_head\0bpf_list_node\0foo\0bar\0baz" +"\0contains:foo:foo\0contains:bar:bar\0contains:baz:baz\0bam" +"\0contains:bam:bam" ; + +#define INIT_BTF_TILL_4 \ +/* int */ \ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4),/* [1] */ \ +/* struct bpf_spin_lock */ /* [2] */ \ +BTF_TYPE_ENC (1 ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,1 ),4 ), \ +BTF_MEMBER_ENC (43 ,1 ,0 ), \ +/* struct bpf_list_head */ /* [3] */ \ +BTF_TYPE_ENC (15 ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,0 ),16 ), \ +/* struct bpf_list_node */ /* [4] */ \ +BTF_TYPE_ENC (29 ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,0 ),16 ), + +static void check_btf(u32 *types, u32 types_len, int error) +{ +LIBBPF_OPTS (bpf_btf_load_opts, opts, + .log_buf = log_buf, + .log_size = sizeof(log_buf), + ); +struct btf_header hdr = { + .magic =BTF_MAGIC , + .version =BTF_VERSION , + .hdr_len =sizeof (struct btf_header ), + .type_len = types_len, + .str_off = types_len, + .str_len =sizeof (btf_str_sec), + }; +void *ptr, *raw_btf; +int fd, ret; + + raw_btf =malloc (sizeof (hdr) + hdr.type_len + hdr.str_len); +if (!ASSERT_OK_PTR (raw_btf,"malloc(raw_btf)" )) +return ; + + ptr = raw_btf; +memcpy (ptr, &hdr,sizeof (hdr)); + ptr +=sizeof (hdr); +memcpy (ptr, types, hdr.type_len); + ptr += hdr.type_len; +memcpy (ptr, btf_str_sec, hdr.str_len); + ptr += hdr.str_len; + + fd =bpf_btf_load (raw_btf, ptr - raw_btf, &opts); + ret = fd <0 ? -errno :0 ; +if (fd >=0 ) +close (fd); +if (error) +ASSERT_LT (fd,0 ,"bpf_btf_load" ); +else +ASSERT_GE (fd,0 ,"bpf_btf_load" ); +if (!ASSERT_EQ (ret, error,"-errno == error" )) +printf ("BTF Log:\n%s\n" , log_buf); +free (raw_btf); +return ; +} + +#define SPIN_LOCK 2 +#define LIST_HEAD 3 +#define LIST_NODE 4 +#define FOO 43 +#define BAR 47 +#define BAZ 51 +#define BAM 106 +#define CONT_FOO_FOO 55 +#define CONT_BAR_BAR 72 +#define CONT_BAZ_BAZ 89 +#define CONT_BAM_BAM 110 + +static void test_btf (void ) +{ +if (test__start_subtest ("btf: too many locks" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,3 ),24 ),/* [5] */ +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,0 ), +BTF_MEMBER_ENC (FOO ,SPIN_LOCK ,32 ), +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,64 ), + }; +check_btf (types,sizeof (types), -E2BIG ); + } +if (test__start_subtest ("btf: missing lock" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,1 ),16 ),/* [5] */ +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,0 ), +BTF_DECL_TAG_ENC (CONT_BAZ_BAZ ,5 ,0 ), +BTF_TYPE_ENC (BAZ ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,1 ),16 ), +BTF_MEMBER_ENC (BAZ ,LIST_NODE ,0 ), + }; +check_btf (types,sizeof (types), -EINVAL ); + } +if (test__start_subtest ("btf: bad offset" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,3 ),36 ),/* [5] */ +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (FOO ,LIST_NODE ,0 ), +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,0 ), +BTF_DECL_TAG_ENC (CONT_FOO_FOO ,5 ,0 ), + }; +check_btf (types,sizeof (types), -EFAULT ); + } +if (test__start_subtest ("btf: missing contains:" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,2 ),24 ),/* [5] */ +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,0 ), +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,64 ), + }; +check_btf (types,sizeof (types), -EINVAL ); + } +if (test__start_subtest ("btf: missing struct" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,2 ),24 ),/* [5] */ +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,0 ), +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,64 ), +BTF_DECL_TAG_ENC (CONT_BAR_BAR ,5 ,1 ), + }; +check_btf (types,sizeof (types), -ENOENT ); + } +if (test__start_subtest ("btf: missing node" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,2 ),24 ),/* [5] */ +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,0 ), +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,64 ), +BTF_DECL_TAG_ENC (CONT_FOO_FOO ,5 ,1 ), + }; +check_btf (types,sizeof (types), -ENOENT ); + } +if (test__start_subtest ("btf: node incorrect type" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,2 ),20 ),/* [5] */ +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (BAR ,SPIN_LOCK ,128 ), +BTF_DECL_TAG_ENC (CONT_BAZ_BAZ ,5 ,0 ), +BTF_TYPE_ENC (BAZ ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,1 ),4 ), +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,0 ), + }; +check_btf (types,sizeof (types), -EINVAL ); + } +if (test__start_subtest ("btf: multiple bpf_list_node with name foo" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,3 ),52 ),/* [5] */ +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (FOO ,LIST_NODE ,128 ), +BTF_MEMBER_ENC (FOO ,LIST_NODE ,256 ), +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,384 ), +BTF_DECL_TAG_ENC (CONT_FOO_FOO ,5 ,0 ), + }; +check_btf (types,sizeof (types), -EINVAL ); + } +if (test__start_subtest ("btf: owning | owned AA cycle" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,3 ),36 ),/* [5] */ +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (FOO ,LIST_NODE ,128 ), +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,256 ), +BTF_DECL_TAG_ENC (CONT_FOO_FOO ,5 ,0 ), + }; +check_btf (types,sizeof (types), -ELOOP ); + } +if (test__start_subtest ("btf: owning | owned ABA cycle" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,3 ),36 ),/* [5] */ +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (FOO ,LIST_NODE ,128 ), +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,256 ), +BTF_DECL_TAG_ENC (CONT_BAR_BAR ,5 ,0 ),/* [6] */ +BTF_TYPE_ENC (BAR ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,3 ),36 ),/* [7] */ +BTF_MEMBER_ENC (FOO ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (BAR ,LIST_NODE ,128 ), +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,256 ), +BTF_DECL_TAG_ENC (CONT_FOO_FOO ,7 ,0 ), + }; +check_btf (types,sizeof (types), -ELOOP ); + } +if (test__start_subtest ("btf: owning -> owned" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,2 ),20 ),/* [5] */ +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (BAR ,SPIN_LOCK ,128 ), +BTF_DECL_TAG_ENC (CONT_BAZ_BAZ ,5 ,0 ), +BTF_TYPE_ENC (BAZ ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,1 ),16 ), +BTF_MEMBER_ENC (BAZ ,LIST_NODE ,0 ), + }; +check_btf (types,sizeof (types),0 ); + } +if (test__start_subtest ("btf: owning -> owning | owned -> owned" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,2 ),20 ),/* [5] */ +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,128 ), +BTF_DECL_TAG_ENC (CONT_BAR_BAR ,5 ,0 ),/* [6] */ +BTF_TYPE_ENC (BAR ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,3 ),36 ),/* [7] */ +BTF_MEMBER_ENC (FOO ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (BAR ,LIST_NODE ,128 ), +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,256 ), +BTF_DECL_TAG_ENC (CONT_BAZ_BAZ ,7 ,0 ), +BTF_TYPE_ENC (BAZ ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,1 ),16 ), +BTF_MEMBER_ENC (BAZ ,LIST_NODE ,0 ), + }; +check_btf (types,sizeof (types),0 ); + } +if (test__start_subtest ("btf: owning | owned -> owning | owned -> owned" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,3 ),36 ),/* [5] */ +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (FOO ,LIST_NODE ,128 ), +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,256 ), +BTF_DECL_TAG_ENC (CONT_BAR_BAR ,5 ,0 ),/* [6] */ +BTF_TYPE_ENC (BAR ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,3 ),36 ),/* [7] */ +BTF_MEMBER_ENC (FOO ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (BAR ,LIST_NODE ,128 ), +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,256 ), +BTF_DECL_TAG_ENC (CONT_BAZ_BAZ ,7 ,0 ), +BTF_TYPE_ENC (BAZ ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,1 ),16 ), +BTF_MEMBER_ENC (BAZ ,LIST_NODE ,0 ), + }; +check_btf (types,sizeof (types), -ELOOP ); + } +if (test__start_subtest ("btf: owning -> owning | owned -> owning | owned -> owned" )) { +u32 types[] = { + INIT_BTF_TILL_4 +BTF_TYPE_ENC (FOO ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,2 ),20 ),/* [5] */ +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,128 ), +BTF_DECL_TAG_ENC (CONT_BAR_BAR ,5 ,0 ),/* [6] */ +BTF_TYPE_ENC (BAR ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,3 ),36 ),/* [7] */ +BTF_MEMBER_ENC (FOO ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (BAR ,LIST_NODE ,128 ), +BTF_MEMBER_ENC (BAZ ,SPIN_LOCK ,256 ), +BTF_DECL_TAG_ENC (CONT_BAZ_BAZ ,7 ,0 ), +BTF_TYPE_ENC (BAZ ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,3 ),36 ),/* [9] */ +BTF_MEMBER_ENC (BAR ,LIST_HEAD ,0 ), +BTF_MEMBER_ENC (BAZ ,LIST_NODE ,128 ), +BTF_MEMBER_ENC (FOO ,SPIN_LOCK ,256 ), +BTF_DECL_TAG_ENC (CONT_BAM_BAM ,9 ,0 ), +BTF_TYPE_ENC (BAM ,BTF_INFO_ENC (BTF_KIND_STRUCT ,0 ,1 ),16 ), +BTF_MEMBER_ENC (BAM ,LIST_NODE ,0 ), + }; +check_btf (types,sizeof (types), -ELOOP ); + } +} +void test_linked_list (void ) {int i;
@@ -243,6 +513,7 @@ void test_linked_list(void)test_linked_list_fail_prog (linked_list_fail_tests[i].prog_name, linked_list_fail_tests[i].err_msg); } +test_btf ();test_linked_list_success (PUSH_POP , false);test_linked_list_success (PUSH_POP , true);test_linked_list_success (PUSH_POP_MULT , false);
Signature · 1 line
2.38.1
Re: [PATCH bpf-next v5 25/25] selftests/bpf: Add BTF sanity tests
Quoted 7 lines · Kumar Kartikeya Dwivedi
Preparing the metadata for
bpf_list_headinvolves a complicated parsing step and type resolution for the contained value. Ensure that corner cases are tested against and invalid specifications in source are duly rejected. Also include tests for incorrect ownership relationships in the BTF.
Quoted 5 lines
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- .../selftests/bpf/prog_tests/linked_list.c | 271 ++++++++++++++++++ 1 file changed, 271 insertions(+)
Have you considered using BTW write API to construct BTFs? btf__new_empty() + btf__add_xxx()?
Quoted 12 lines
tools/testing/selftests/bpf/prog_tests/linked_list.c +3
-0@@ -1,4 +1,7 @@// SPDX-License-Identifier: GPL-2.0 +#include <bpf/btf.h> +#include <test_btf.h> +#include <linux/btf.h> #include <test_progs.h> #include <network_helpers.h>
[...]
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v5 25/25] selftests/bpf: Add BTF sanity tests
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Kumar Kartikeya Dwivedi
Preparing the metadata for
bpf_list_headinvolves a complicated parsing step and type resolution for the contained value. Ensure that corner cases are tested against and invalid specifications in source are duly rejected. Also include tests for incorrect ownership relationships in the BTF.
Quoted 9 lines
Quoted 5 lines
Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- .../selftests/bpf/prog_tests/linked_list.c | 271 ++++++++++++++++++ 1 file changed, 271 insertions(+)Have you considered using BTW write API to construct BTFs?
btf__new_empty()+btf__add_xxx()?
I didn't know about these. Let me give them a shot!